Improve code intelligence quality and reduce false positives

[chrismwendt]

Code intelligence has broken (I mean P0 broken) a few times in the past couple of weeks in ways that could have been prevented by E2E and unit tests. Also, basic-code-intel is rather slow on large repositories and often times reports false positives in definitions and references, which hampers adoption.

Plan

• Symbol search is too slow to provide jump-to-definition on torvalds/linux https://github.com/sourcegraph/sourcegraph/issues/2257
  • PR https://github.com/sourcegraph/sourcegraph/pull/2334
• Set up E2E tests for code intelligence
  • Web app https://github.com/sourcegraph/sourcegraph/pull/2533)
  • Browser extension https://github.com/sourcegraph/sourcegraph/pull/2566
  • Code intel https://github.com/sourcegraph/sourcegraph/pull/2662
• Fix and write more unit tests for basic-code-intel
  • Add dev,ci scripts, parameterize sourcegraph dep for testing code-intel-extensions#36
  • Fix unit tests code-intel-extensions#38
  • Add a bunch of tests code-intel-extensions#39
• Publish all sourcegraph/{go,python,...} extensions in CI
  • Go Deploy extension on merges to master sourcegraph-go#34
  • Python sourcegraph/sourcegraph-python@4d858ae
• Add docs for HTTP basic auth for language servers, and document this in extension READMEs https://github.com/sourcegraph/sourcegraph/issues/2595
  • Go Add HTTP basic auth docs sourcegraph-go#37
  • TypeScript Add docs for HTTP basic auth sourcegraph-typescript#129
  • Python Add docs for HTTP basic auth sourcegraph-python#10
• Indicate that basic code intel hovers/defs/refs are fuzzy in the UI (could be a "Fuzzy" label in the tooltip, or other UI treatment in the defs/refs panel) https://github.com/sourcegraph/sourcegraph/issues/1174
  • basic-code-intel Add an "Imprecise results" label to references code-intel-extensions#41
  • Go Add an "Imprecise results" label to references sourcegraph-go#39
  • TypeScript Add "Imprecise results" label to references sourcegraph-typescript#127
  • Python Add "Imprecise results" label to references sourcegraph-python#4
• Visual testing with Percy https://github.com/sourcegraph/sourcegraph/pull/2606
• Index file-local definitions in the symbols service
  • Adding fileLocal to the Symbols DB https://github.com/sourcegraph/sourcegraph/pull/2788
  • Filtering by fileLocal Filter by fileLocal code-intel-extensions#63
• Index Go packages https://github.com/sourcegraph/sourcegraph/pull/2802
• Reduce false positives in basic-code-intel https://github.com/sourcegraph/sourcegraph/issues/2298
• Perform symbol search on other repositories to find cross-repo definitions
  • Replace definition patterns with cross-repository symbol search code-intel-extensions#53
• Fall back to basic code intel when hovers take too long https://github.com/sourcegraph/sourcegraph/issues/2571
• Use @sourcegraph/lsp-client in sourcegraph-go Use @sourcegraph/lsp-client sourcegraph-go#28
  • WIP PR Use @sourcegraph/lsp-client sourcegraph-go#28
  • Blocked on issues in Use @sourcegraph/lsp-client sourcegraph-go#28 (comment)
• Incorporate basic-code-intel into @sourcegraph/lsp-client
• Eliminate dependency on Sourcegraph.com's basic-code-intel in e2e tests https://github.com/sourcegraph/sourcegraph/issues/2681
• Escape the file: filter in basic code intel Fix unit tests code-intel-extensions#38 (comment)

Test/review plan

• Symbols perf improvements have tests + benchmarks
• E2E/unit tests will be run in CI
• Reducing false positives will have unit tests

[chrismwendt]

• Indicate that basic code intel hovers/defs/refs are fuzzy in the UI (could be a "Fuzzy" label in the tooltip, or other UI treatment in the defs/refs panel)

@sqs Do you have a link for this? added link

[nicksnyder]

Please close this tracking issue for 3.2 and create a new tracking issue for 3.3 that includes the unfinished work (if still appropriate).

[chrismwendt]

3.3 tracking issue https://github.com/sourcegraph/sourcegraph/issues/2856