cli: add basic context insensitive tab completion for the CLI

[rohany]

This PR adds basic context insensitive tab completion for
the CLI. It maintains a store of keywords, variables
and identifiers and uses these to match against unfinished
tokens in the input stream.

Work with @Azhng

Release note (cli change): Add basic tab completion for the SQL CLI.

[cockroach-teamcity]

This change is 

[rohany]

Alright, I'm having some trouble adding the dependency I want to vendor... Any tips?

I've added the package to gopkg.toml/gopkg.lock, and I've made a commit in vendored that has the depedency. How do I point cockroach at that commit?

[knz]

I've added the package to gopkg.toml/gopkg.lock, and I've made a commit in vendored that has the depedency. How do I point cockroach at that commit?

1. give your commit in vendored a branch name and push it upstream
2. git add vendor in the crdb repo to link it

[rohany]

Awesome, thanks!

[rohany]

I was able to get tab suggestions to work when there were multiple completions, but only on linux (not in this PR). Does it make sense to merge something that would only add the feature on linux not mac?

[knz]

If you don't mind I'd prefer that hackathon PRs do not get merged until they go the full cycle of review (and after we iron out the most glaring UX issues).

[rohany]

Yes of course -- I've been polishing up for review. I just wanted to know what the philosophy was on differing support on different OS's.

(also, not saying this is ready to be merged -- theres most likely lots of room for improvement).

[knz]

It's OK if a feature doesn't work in environment X while it works fine on environment Y; however consider:

• usually X = windows and Y = everything else
• we like Y to be "the things that most Developers use"

Considering that macOS is a pretty popular dev platform, that means you might want to either work on the user messaging (a copious release note) or iterate on the code a bit more.

[rohany]

That makes sense.

Considering that macOS is a pretty popular dev platform, that means you might want to either work on the user messaging (a copious release note) or iterate on the code a bit more.

We can spend some more time, but attempts to sidestep the libedit bug on mac haven't led to much success.

[rohany]

cc @knz can you take a look at this and estimate what it would need added if we wanted to get this into 20.1? (if it should be included at all)

[knz]

Sorry for dropping the ball on this.
I think it is too late in the 20.1 cycle to add this, also we'd probably want to increase the test coverage.

Let's pick it back up in a month from now.

[awoods187]

@knz out of curiosity, what makes it too late in the cycle? is this risky?

[knz]

Yes there is risk.

• The feature does not work consistently across supported platforms. This will raise eyebrows, and more importantly, questions in support because we don't have a good way to explain/document this difference yet.

• The feature fails to detect new table names after CREATE TABLE or new column names after ALTER TABLE ADD COLUMN; nor does it properly recognize names that disappear after DROP. This will raise more eyebrows (= support burden).

• It works by querying then loading all the SQL schema of the current db client-side in RAM. This would cause hard-to-troubleshoot slowness and possibly OOM situations when an operator unsuspectedly connects the SQL shell to a large db.

In an ideal world we'd operate as follows:

• iterate on this a bit more to work harder at feature parity across supported platforms.

• make the algorithm work properly with a subset of the data:

  1. only populate the data in RAM the first time it's needed, instead of unconditionally.
  2. make the thing detect DDL properly and refresh names as necessary. This could leverage the newly added support for the notice protocol, using dedicated messages sent by the server to the client upon request by our CLI shell.
  3. when entering select <tab> with no table known, auto-complete a table name from the current db first, make the user choose a table, before engaging with column name completion (so that the code does not need to load all column names in RAM upfront, and only the table names from the current db)
  4. optionally, look up tables from other db if the tab key is pressed after a "db." prefix and "db" is a valid database name.
  5. if the query is already syntactically well-formed with an hypothetical identifier at the current position, ask the query planner for a list of valid identifiers in that context.

Of these latter steps, steps (1) - (4) are the required polish for the feature as-presented. Only step (5) can be seen as an architectural step further towards a more feature-complete tab completion and thus could be pushed to a v2.

[awoods187]

Thanks @knz! Let's push this to early 20.2 then!

[rohany]

I don't have the bandwidth right now to fix all of these comments, but @RichardJCai had an idea: as a starting step, would it be useful to just have tab completion of our keywords?

[knz]

would it be useful to just have tab completion of our keywords?

That certainly would be a step forward. Maybe a PM would have an educated opinion?

[awoods187]

That certainly would be a step forward. Maybe a PM would have an educated opinion?

Yes, I think that would be valuable as an intermediary step. It would be great to build toward full tab completion.

[fantapop]

Any movement on this would be great but my humble opinion is that keywords are the thing I need tab completion on the least because as a developer using the sql console I already know them all and they're usually a lot shorter than table names.

[RichardJCai]

Any movement on this would be great but my humble opinion is that keywords are the thing I need tab completion on the least because as a developer using the sql console I already know them all and they're usually a lot shorter than table names.

Might be reviving this or maybe reviving it using https://microsoft.github.io/language-server-protocol/ for breather week.