WIP: Adding Kite to list of resources.

[landizz]

Added a miscellaneous resource called "Kite"

[lemonsaurus]

An article brought up in #meta details a long history of controversy between this company and the open source community. See https://qz.com/1043614/this-startup-learned-the-hard-way-that-you-do-not-piss-off-open-source-programmers/.

This includes acquisitions of popular open source plugins for Atom and Sublime Text in order to inject them with Kite integrations nobody wanted, and collecting anonymous data from users without their knowledge.

That said, this appears to all be part of some advertising campaign gone horribly wrong, rather than wrongdoing on the part of the app itself, which may be a useful tool.

As was also brought up in #meta, the privacy policy leaves a lot of questions unanswered and can at best be described as "potentially shady". It reserves the right to collect your entire source code and use it for a number of things, including "marketing purposes".

All of this taken into account, I'm rejecting this PR.

[landizz]

I wanted to add this just because.

After some digging I found that they don't save the data to a cloud and let's users see what files have been uploaded and gives them the possibility to delete the files. Even tho of their shady past I personally believe that people and companies should be given a second chance, especially when the application seems to be really useful for all kinds of python users.

I will in addition of this, email their feedback address and tell them to be more clearer on their privacy policies.

[landizz]

"Thanks for taking the time to write in! We really appreciate the information and will be updating our privacy policy in the coming days.

To address your immediate personal concerns, you can rest assured that we no longer sync your code for any reason. If you were using the cloud version of Kite previously, you can visit your account settings to delete your code.

Best,
Daniel"

This is the response I got by emailing the company addressing the issues that was brought up in the discord discussion.

[GhostofGoes]

This does seem really useful. Further discussion?

[jchristgit]

I'm not a fan of this at all for the reasons @heavysaturn listed above. I'm also not quite sure what editor plugins have to do with resources. If we were to include editor integrations for Python in the resources, I'd much rather see something open-source-first like jedi.

[scragly]

I also agree with @heavysaturn and @jchristgit.

There are way too many plugins, so adding one will start a cascade of people wanting to add more that they use for their own personal setups, but a developers environment is highly individual and will not apply to large numbers of people who have their own preferred editors, environments and tastes.

With that just alone, I would agree with the closing of this ticket.

I also respect the extent you've gone to contact them and verify if they sync code, it doesn't retract the fact they are keeping currently still have their current terms and privacy policy which says people agree to allowing it in the first place which may not be immediately apparent to those using the tool.

[gdude2002]

I just figured I'd weigh in on this one since I was the person to suggest this be added to the resources in the first place.

• Kite is not a plugin. Kite is an external tool for displaying Python documentation in a separate window from your editor, and operates independently from your editor. It offers plugins - installed automatically by their application on request - to integrate the tool into your IDE (providing automatic documentation for the code at your cursor, for example).
• Previously, Kite was a cloud-based service. As a result, code was uploaded there for indexing and requests were also made to it to provide the services as shown on their site. This is no longer the case - Kite works entirely offline. The "Here's all your files and a remove button" interface exists for legacy users that used the cloud service.
  • Of course, they could have just removed them automatically, but that's another question really
• Kite allows you to completely opt-out of telemetry - I suspect the connections it makes after that are for automatic updates.
• I don't think previous advertising catastrophes are a great way to gauge a project like this, years after it happened. Every company makes mistakes and these mistakes are addressed - for example, GitHub's terrible terms of service changes over the years, Apple's previous inability to screen certain types of apps from their app stores, Sony's completely broken encryption on their PS3 systems... But of course, we still use these companies' projects and products.

Jedi looks neat, but its reference implementation is a Vim plugin and the documentation states that there are IDE plugins available while exclusively linking to plugins for text editors, and an IDE that I don't think any of us have ever heard of.

It also doesn't really compare to Kite at all, from what I can see.

[scragly]

Thanks @gdude2002 for the clarifications. It's appreciated, as I got a heavy impression from the description in the PR diff (seeing as there's zero detail in the initial PR message itself) for this resource that the tool is heavily reliant on editor integrations, making it primarily a plugin tool.

Kite's powerful editor integrations allow you to work uninterrupted on the same screen. Need to see examples and documentation? All the information from Kite's Python documentation is available right in the editor copilot — no web browser needed.

Since gdude's explanation of the tool seems a lot better, it would be good if the resource description could be rewritten to better reflect it, and to ensure the plugins are an optional extra that provides the extra feature of following your cursor, and is not at all necessary if wanting to try the tool.

In the support response given to @landizz, it's indicated there may be a privacy policy update coming in the short term that may address the issue of them forcing users to agree to read, share and use any of the user's code.

If the privacy policy issue is addressed, and the description is edited to better describe the tool, then I'd say I have no further issues with the addition.

A small recommendation for the future, please ensure the initial PR message contains as much detail as reasonable regarding the additions or changes you're implementing, along with your thoughts and justifications for it. In this case, the tool wasn't well represented, and there was no detail at all present.

[gdude2002]

I forgot to mention this before, but the tool does require an account. I'm not sure why.

[scragly]

Marketing maybe 🤔
That fact should also be included in the resource description to be clear.

[landizz]

First time I made a contribution, pull request and many other things so I am sorry for not making it clear enough. The description is copy and pasted from their own website because I thought they would best be able to describe their own tool, rather then a outsider (me) making up a description of the tool which would most likely be somewhat inaccurate in the description of it's functionality and technique. @scragly

In hindsight I should have let @gdude2002 make the PR as he is more qualified in many ways. I was however encouraged to do the PR and I appreciate that, but it would most likely have been presented in a way that would have had a better chance of acceptance. But on the other hand, you need to experience things to get better so thank you @gdude2002 and @heavysaturn for the encouragement.

I appreciate all your thoughts on the matter and I sincerely hope the Kite team will update their policy to be more clear. I put heavy emphasis in the email on the fact that they have a great opportunity here to get free publicity if done correctly, so there really isn't any reason for them to not do it. Unless they truly are shady in some way.

[scragly]

Nah, it's good you decided to do it yourself. You've had a chance to get acquainted with the system, and you've been given feedback to consider for the future. Don't let some corrections get you hesitant to contribute when you feel something is worth contributing.

I hope to see the ticket updated with the good news sometime about the policy updates and at that time we can collectively decide on a better description for the tool if it's deemed appropriate to add as a resource to the site.

Regarding the wording coming from their website, it's pretty common that marketing blurbs don't fit perfectly in objectively describing their tool, because they have something to gain by adding assumptions or focusing on certain features. No major issue, just means a little extra thought for us all when we need to decide on what suits best.

[gdude2002]

Kite sent me an email last night.

Heres the link from the email: https://www.kite.com/blog/launching-line-of-code-completions-going-cloudless-and-17-million-in-funding/

[lemonsaurus]

That sounds promising. Can someone actually use the tool for a while and then we can consider adding it to our resources when someone has first-hand experience with it (and don't hate it) rather than just "hey that looks cool" ?

[lemonsaurus]

This description will not do, if we decide we want to merge this.

[landizz]

I have come up with a new description, please give some feedback on it.

"A plugin tool for several popular editors that also runs alongside the editor as a application. The plugin provides extensive code completion and the application will give you detailed documentation on-the-fly based on what parts of your code you have selected, whether that be standard library modules or non-standard library modules."

Edit: Made it more clear that Kite consist of a plugin and a application.

[jchristgit]

I thought the plugin was just an addition to it? There's plenty of other editor plugins which do this.

[landizz]

It is a plugin, but it is also a application that runs alongside the editor to provide the documentation. I should improve the description to make that clearer.

[scragly]

The following are important points as of the moment then:

• The privacy policy is now reasonable.
• Usage metrics are able to be disabled.
• The product no longer requires an account.
• It's already available on Windows and MacOS, but not Linux. However, they said it's coming "very soon". But they've been saying that since March, 2017.

Due to the lack of Linux support, I'm personally unable to test it out, otherwise I would happily do so.

[landizz]

I have used it since the day I found it (11 days ago), I am not a consistent or a daily coder (I code in small bursts when I get the motivation for it). From my experience with it, it provides quite good documentation (main part why I use it). I've noticed sometimes that it's not that quick or even responsive when I click certain parts of my code and the application alongside is getting the documentation. I have fixed this issue by a simple restart of both the editor and kite application when it happens. This usually happens after having both the editor and Kite application running for several days and when I put my computer in Sleep mode.

When it comes to the content of the documentation it seems good. From what I can tell it only pulls up documentation and reacts to functions, when I select syntax like "for, not, else, if" it will not give any information but when I select "print(), re.compile() etc..) it will give me documentation.

When it comes to functions contained in modules or the module itself like "re.compile()" or "re" it will give me a list of functions within that module and makes that list into links which I can press to get more information about that specific function.

It works on some, but not on all non-standard modules. In my current project I use 2 modules outside of standard lib, those modules are "Tika" and "openpyxl". The application can give me documentation on "openpyxl" but not on "Tika", I'm guessing this is because "Tika" is not as well known or used compared to "openpyxl".

Oh yes, it also reacts to and provides documentation on exceptions.

I will attach 2 screenshots giving examples on the documentation from the exception "OSError" and the module "openpyxl".

[MrHemlock]

As of now, there hasn't been enough comment or support from staff to really push this through. For now, while we're working on doing a Feature Freeze, we're going to go ahead and close this. It can be readdressed after the freeze is over.

[gdude2002]

Just an additional note for this particular topic. Now that Kite supports Linux, I figured I could take a more in-depth dive.

First and foremost, you no longer need an account. The copilot still prompts you to login the first time you install it, but there's a "continue without email" button, and it'll work just fine without it.

Once of the nicer PyCharm features:

Also, you can still disable metrics.

I thought that, being on Linux, I could easily analyse the traffic this thing generates.

I thought wrong, as it turns out. I was able to capture a few things with mitmproxy, but even setting up an entire VM and shoving the whole thing down mitmproxy via SOCKS didn't really get me that far. I was able to glean a few things, though:

• Kite has two parts - kited (the daemon that handles all the heavy lifting) and kite (the Electron-based copilot app)
• The install guide recommends downloading their bash script and running it. Here's what it does:
  • It makes an HTTP request to an update-check endpoint at linux.kite.com with some params that don't really seem to mean anything
    
  • It gets redirected to a URL on Azure, which is basically just a JSON with version info and a couple signatures
    
  • It requests a kite-updater.sh from the linux.kite.com domain, redirected to Azure again
    
  • As you can see, this is a whopping 260 MiB! In the interests of science, I downloaded it to see what exactly it was - and boy, this one's a doozy. It does indeed contain a Bash script, but that's not all..
    
    That's right - there is an entire binary executable in this script! This is in fact a makeself self-extracting archive, for Unix.
  • Upon running this thing myself, it turns out that this archive contains an entire copy of Kite, which itself bundles an updater. Good times, I guess.

This was where I got stuck. I was able to run the kited daemon via proxychains, but it was unclear if it did anything useful - I certainly wasn't getting anything via mitmproxy, aside from attempts to connect to api.segment.io (which was being blocked by my pi-hole). This is, of course, Segment - a customer analytics platform. If you, too, have Segment blocked, then there is some info on Crunchbase.

When running the Kite copilot via proxychains, I was unable to do anything - simply presented with a purple screen, and completely unresponsive to input. After some random clicking, though, I did notice a few entries in mitmproxy:

These were all POSTS containing a single JSON object:

So clearly what's going on here is that - at least for metrics - the Kite copilot is sending the metrics to the local kited daemon, and it's forwarding them to Segment.

Unfortunately, that's about as far as I could get - I attempted to unblock Segment on my network, but I couldn't get the change to stick.

---

I'm not convinced Kite is doing much shady work here. Looking at the type of help it gives you in your editor, it appears to show common usages for whatever methods you're working with. It does raise a question, though..

Obviously, it's getting these common usages from somewhere. Considering it includes a string constant here, I would not recommend using it without disabling the usage metrics. Thankfully, there is an easy toggle for that in the copilot settings - enabling that is a must, at least until we have more information on what's collected. While the features are legitimately useful, and are even a big improvement on the way PyCharm handles these things, caution is advised.

If anyone has any other bright ideas on how I could do more network analysis, please let me know!

[scragly]

After another discussion at a meeting, Kite has been voted against being added to the resources page.

Aside from all the previously mentioned concerns regarding the company's controversial history, it's something that's just not really, in and of itself, a resource for learning, instead being more suitably described as a tool. We don't want to open up entries for tools at this time aside from the existing curated list of editors, lest we bombarded by more tools requests in the future.

In the near future, we'll be likely revisiting the resources page to clean up any other things that may possibly fall under the same category of being primarily a tool in order to keep the page as simple and pure to its purpose as possible.

[itsdhung]

Hi guys, just chiming in here to provide some perspective from a Kite employee...

First, we totally respect your decision to not include Kite in the resources page and sincerely appreciate the effort that was taken to perform the necessary due diligence. We're going to keep improving the product regardless, so that it provides value to members of this community should they choose to use it on their own accord.

Second, I'm personally very impressed with the work @gdude2002 put in to figure out what Kite's doing under the hood! I understand how there can be concerns around data and privacy, so I want to both:

1. Give as good of an explanation as I can about the data Kite uses and
2. Make myself available to answer more questions if there are any.

This is going to be somewhat of a long comment, so I'll try to organize it as clearly as possible.

Where do function usages come from?

To address this concern:

Obviously, it's getting these common usages from somewhere. Considering it includes a string constant here, I would not recommend using it without disabling the usage metrics. Thankfully, there is an easy toggle for that in the copilot settings - enabling that is a must, at least until we have more information on what's collected. While the features are legitimately useful, and are even a big improvement on the way PyCharm handles these things, caution is advised.

Internally we call usages like these popular patterns. For functions in publicly available packages such as dj_database_url, we crawl all of the open source code on GitHub to learn how these functions are used. From that, we can infer calling patterns and also rank the patterns by the frequencies at which they occur.

So in the screenshot posted above, Kite is saying that based on publicly available code in GitHub, config() is the most popular way to call the function, then config(default=<str>), and then config(conn_max_age=<int>). We also include argument literal values seen on GitHub to illustrate actual examples of the pattern. That is why "'postgres://localhost"' shows up in the second pattern — This means that a lot of people use the config function to hook up a Postgres DB running locally.

Unfortunately, if an open source developer doesn't adhere to best practices and stores a secret in a public file on GitHub, Kite may expose the secret depending on the popularity/ranking of the pattern the secret is used in. We're aware that this is an issue, and are working on a change to this feature to make it more user friendly and also to prevent the display of secrets.

The methodology to find popular patterns is exactly the same for local functions defined in your codebase. It may not be clear, but we don't transmit your function patterns over the network, and all the analysis done on your code happens on your own system.

What data does Kite collect from your system?

Our privacy policy contains a pretty comprehensive description of what we collect, but there are a couple of points worth reiterating.

• We don't upload your code in any form to the cloud: This means we don't upload your raw source code or any other processed forms of your code (e.g. indices) whether or not you've enabled usage metrics. We also don't upload any data structures synthesized from your code that are used in features displayed in the UI. Therefore, we don't upload your function patterns to the cloud, which was a concern raised previously.
• Your usage metrics are never shared: We use your usage metrics for the sole purpose of understanding how you use our product and what we can do to improve it. We do not share your usage metrics with other 3rd parties or other Kite users.

Finally, for those curious about what the /clientapi/metrics/counter endpoint does, it essentially tells kited that a feature was used. In the example above, we observed:

POST /clientapi/metrics/counter
{
  "name": "sidebar_focused",
  "value": 1
}

This call tells kited to increment an internal counter that keeps track of how many times the Copilot (previously called the Sidebar) was brought into focus. These internal counters are then periodically sent to Segment.

It's important to note that the /clientapi/metrics/counter endpoint is used even if you've disabled usage metrics. What is actually disabled is the transmission to Segment. This was done so that only kited needs to keep track of whether or not metrics need to be tracked.

---

I hope this helps shed some light on what Kite is doing with respect to sensitive data. Please feel free to email feedback@kite.com if you'd like more info on the internals of Kite. I could write an encyclopedia on it, but I'm not sure how many people would be interested in reading it. 😅

[gdude2002]

Hey @its-dhung, thanks for responding - appreciate the explanation, it covers a lot of stuff I wasn't able to dig up on my own. I get quite a lot of use out of Kite personally, and it's definitely interesting to see Kite team members leaving comments like this!

I did have a couple questions, if you don't mind me asking:

• What purpose do the Kite accounts serve at this point, from a user perspective?
• Is there any way to provide eg a SOCKS proxy to the Kite copilot and daemon so I can analyse their traffic a bit more accurately?
• Are there any plans to outright parse documentation for installed modules or just from Sphinx sites online, for modules that aren't explicitly supported by Kite?
• How do you guys plan to monetize? Totally understand if you don't want to answer that one. Also, any chance of open sourcing any components for users to improve or submit their own features and module documentation?

[itsdhung]

What purpose do the Kite accounts serve at this point, from a user perspective?

As you noticed, Kite works exactly the same whether or not you have an account. If you do create an account, the benefit for us is that we can reach out to you via email to get feedback. We're constantly talking to our users to learn how to improve.

Is there any way to provide eg a SOCKS proxy to the Kite copilot and daemon so I can analyse their traffic a bit more accurately?

We have a pretty small engineering team, so we have to be careful with how we allocate our resources. This won't be possible in the near future, given our constraints. I can however try to answer all your questions to best of my ability. I know this isn't quite the same as allowing you to peek under the hood, so we appreciate any level of trust you put in us.

Are there any plans to outright parse documentation for installed modules or just from Sphinx sites online, for modules that aren't explicitly supported by Kite?

This is a great question. We're currently mulling over ideas on how to improve the documentation experience. If you want to talk more about this, please drop us a line at feedback@kite.com. We take user feedback very seriously and a lot of times it guides how we change the product.

How do you guys plan to monetize? Totally understand if you don't want to answer that one.

The current thinking is that the consumer "single player" version of Kite will always be free to use. While I can't share concrete details yet, our plan is to monetize Kite by having a version that's useful in a team setting, thereby having engineering teams at companies pay for Kite. So the end goal is to have a B2B business model.

One thing I can definitively say is that the consumer product will not be monetized by ads, or by us selling your data to 3rd parties.

Also, any chance of open sourcing any components for users to improve or submit their own features and module documentation?

The open source issue is a debated topic within Kite. Suffice to say, there are differing opinions about if/when we should do this. Again, given our limited resources, this is not something we'll be undertaking in the near future.

As far as requesting features, as mentioned previously, we're always willing to listen. Just give us a shout!