[Feature Request] jupyter-style autocomplete in the editor (provide column names for dataframes for instance)

[stevenlis]

1. When typing in the main editor, there are no completions for column names but there are completions when using the input windows in interactive windows:

2. The method completions (e.g., .dropna()) in the interactive windows only show up for DataFrame not Series. Meanwhile, the editor does not seem to have completions for those common methods at all.

Version: 1.34.0 (user setup)
Python Extension: 2019.5.17517
Commit: a622c65b2c713c890fcf4fbf07cf34049d5fe758
Date: 2019-05-15T21:59:37.030Z
Electron: 3.1.8
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 10.0.17763

[stevenlis]

not sure if this is directly related the python extension issue but I couldn't find the IntelliSense repo on GitHub.

[rchiodo]

The column names do not show in the editor at the moment because the intellisense for the editor doesn't have a connection to the running jupyter instance. Not sure if we should add it or not. Maybe?

For the series problem, what's missing? It seems to work for me? Or are you referring to like double completion? (Where you type a . at the end of a return value)

Something like:

df.iloc[0].

Unfortunately, that isn't currently supported.

The intellisense repo is here:
https://github.com/Microsoft/python-language-server

[stevenlis]

Not sure if we should add it or not. Maybe?

Would be great if I don't have to type them every time. lol...

Or are you referring to like double completion? (Where you type a . at the end of a return value)

When I use dot notion or brackets to select a column and then apply a method on it.

[rchiodo]

When I use dot notion or brackets to select a column and then apply a method on it.

That would be a feature request on the language server. Feel free to add issues there too.

[rchiodo]

I should have explained more why I was hesitant to include column names in the editor intellisense.

The question is do they actually apply to the active file or not. I think we can figure that out based on if the active file ever sent a cell, but what if two files sent the same variable name? Okay then we have to parse variable creation and figure out which one was the last one. At that point we're writing another language server (right now the column names come from Jupyter's auto completion).

So it's possible, it's just a lot of work. Hence the maybe.

[stevenlis]

@rchiodo thanks for the explanation. btw, this little input window at the bottom really grows on me!
btw, is there any way to edit those read-only:

If so, it would make it feel like writing a jupyter notebook XD... and also I found that sometimes I had a typo and need to copy and paste the code to input windows again. Would be easier if I can just correct them right away.

[rchiodo]

Our intent with the interactive window is that it's a history, not a notebook. It's like a terminal (or what Spyder has). Meaning once you type something in, it can't be changed or you'd be traveling through time :)

Many people actually do generate notebooks though (instead of just python code), and have been asking for a way for the window to behave more like a notebook.

With that in mind, what would you want?

1. An interactive window that keeps in sync with the code (maybe something like this: microsoft/vscode-python#5880)
2. Inline output in the python editor, but keep the interactive window the same. This would be like hydrogen.
3. A hybrid approach where the interactive window is still used, but you can edit the last cell
4. A hybrid approach where the interactive window is in sync with the code, but removes old outputs and reorders them (basically what @gramster asked for in microsoft/vscode-python#5880)
5. Something else?

[stevenlis]

@rchiodo those are pretty much what I want. and also some better markdown support I guess?, It's something I have logged before but maybe was closed for some reasons like change font size of the markdown text.

[rchiodo]

Sorry I should have asked for a priority or which one you liked best.

Your markdown bug is still active. We just haven't gotten to it yet. #1251

[stevenlis]

@rchiodo thanks... I was confused I guess.

I think I really gotta like this on:

removes old outputs and reorders them

But I remember that we could actually close cells in the interactive windows right? Why can I no longer see that button anymore?

I used to miss inline output since I was using Atom + Hydrogen before. But right now I think the current vs code model is better since it keeps my code clear.

Btw, I really wish you guys can toggle run below and run above in CodeLens. I can't remember how many times I accidentally clicked them.

[greazer]

May not be technically feasible to support the jupyter autocomplete in the editor, but we'll take a look. This seems to be a reasonable expectation.

@StevenLi-DS, can you open separate issues for your other suggestions? This will help us make sure we don't miss any of the feedback you've been giving us. Thanks!

[stevenlis]

But I remember that we could actually close cells in the interactive windows right? Why can I no longer see that button anymore?

@rchiodo I was trying to reproduce the issue and then realized that I make a mistake. It is because I had a wide dataframe:

[rchiodo]

Yeah I've always felt like those buttons should be on the left. You shouldn't have to scroll to get to them.

[stevenlis]

@greazer for:

Btw, I really wish you guys can toggle run below and run above in CodeLens. I can't remember how many times I accidentally clicked them.

There is a closed issue: https://github.com/microsoft/vscode-python/issues/5128 and I also mentioned it here: https://github.com/microsoft/vscode-python/issues/4259 . Both of them were closed and maybe reopen them? This really bothers me.

I saw that @rchiodo has logged some of the ideas above. Should I open one for adding availability to reordering the cell or something?

[rchiodo]

Those other two issues were about turning all codelens off. We don't have a setting for specific ones.

Were also talking about changing them again to support debugging. I think we need a way for the user to pick what code lens they see. I'm going to enter an issue for that.

[rchiodo]

The new issue is #1519. Please feel free to comment there if I missed anything.

[Stanpol]

The column names do not show in the editor at the moment because the intellisense for the editor doesn't have a connection to the running jupyter instance. Not sure if we should add it or not. Maybe?

Yes, you definitely should add this. One of the main purposes of Jupyter notebooks is to do exploratory data analysis, so for our team it is crucial to have autocomplete for pandas columns.

I should have explained more why I was hesitant to include column names in the editor intellisense. The question is do they actually apply to the active file or not.

Our intent with the interactive window is that it's a history, not a notebook. It's like a terminal (or what Spyder has). Meaning once you type something in, it can't be changed or you'd be traveling through time :)

I like the idea that interactive window is a history and it shouldn't be changed, but it should be connected to only one file. If you run a cell from another file, it should be a separate interactive window (like you create a new jupyter notebook). This way there won't be mix between variable names of different files and autocomplete will apply to only one file variables.

[stevenlis]

Btw, the column name completion in jupyter works with both dot notion and brackets.

[rchiodo]

@StevenLi-DS what's break? CTRL+C?

[rchiodo]

@Stanpol I'm going to add another issue to see how many people use multiple files per interactive window now. Not sure if people would want to have 1 per file or not.

https://github.com/microsoft/vscode-python/issues/6373

[stevenlis]

@rchiodo sorry I mean brackets...
df.col_name or df['col_name']

[janosh]

Not sure if people would want to have 1 per file or not.

I think one interactive window per file would be a good start! Later on, one might look at merging two interactive windows or running cells from another file in an existing window.

[greazer]

There are at least two people in this thread that are complaining about the inability to autocomplete columns. Jupyter itself supports this. The debug console of VSCode supports it. I ran across this same issue mentioned in another bug by a different customer, and when I tried it myself I got confused as to how the debug console could "do the right thing" but not in the editor. Finally, now that we have debug cell, there's yet another entry point for more people noticing.

Therefore, I think we need to seriously consider doing something to address this. Moving to re-triage.

[greazer]

Let's do this. We will assume that most people are using one file with the interactive window. Try to make it work when debugging a cell too.

[rchiodo]

Moving back to triage, had another request for this.

[damienrj]

I was looking forward to just using Jupyter in vs code and not the web browser now that notebooks are supported but this is definitely a unexpected experience to find autocomplete not working when running the notebook in vs code.

[rchiodo]

@damienrj they should be working in notebooks. (This bug is about files, not notebooks).

You do have to connect to the server first though (by running a cell).

[damienrj]

@rchiodo It turns out that it was a combination of the problem https://github.com/microsoft/vscode-python/issues/8171 and https://github.com/microsoft/vscode-python/issues/12655 that made me thing that it wasn't working. So never mind.

[mrslausen]

@rchiodo and @greazer - I see this was on the Q1 Release and then removed. Do you have any idea when this will make it back onto the roadmap? This seems like a very standard workflow for DS teams that are working interactively in python files, and prefer that workflow vs. using the full Jupyter notebook. Thanks!

[rchiodo]

Sorry but there are no plans for this at the moment. @greazer, we really should publish our roadmap.

[janosh]

...we really should publish our roadmap.

That'd be great!

[mrslausen]

Sorry but there are no plans for this at the moment. @greazer, we really should publish our roadmap.

Ok, thanks for the update. Appreciate it!

[v-asu]

Is there any way to achieve such autocomplete (for the dataframe column names) ? It is really helpful (and much needed) while cleaning the data. Thanks

[rchiodo]

Dataframe column name auto complete is only available within the notebook editor or within the interactive window (for now as this bug is to implement it in files)

[Sutyke]

Is there any update on dataframe column name auto complete bug? It doesn't work even in the interactive window.

[rchiodo]

Interactive auto complete works for me. It might be you have to wait for a bit.

In that picture, there's a 'Miles_per_gallon' column on the dataframe.

We ask Jupyter for the autocomplete information. If jupyter is using Jedi this might take a while.

[eved1018]

I'm still a little confused if this can be implemented into the editor as well ( unrelated to jupyer) as a standard python thing.

[rchiodo]

@eved1018 this requires jupyter to work in the editor. It needs a runtime reference to the dataframe because the column names aren't available without the data being loaded.

So no this would not be available in normal python editing unless you were using the interactive window and the file you're looking at was participating in the interactive session (well once we fix this bug).

[eved1018]

Amazing thank you!

[rchiodo]

Closing for lack of up votes. We will eventually support pylance understanding kernels. At that point I think this would just work (well if we told pylance about the kernel used for the interactive window).