Code Intelligence 3.22 Tracking issue

[efritz]

Plan

Goal: Provide precise code intel for the primary programming languages that combined cover > 50% of usage across customer projects. Garo is forging ahead with lsif-clang bugs, and will aim to index 10 more open source repositories as preparation to ship to a wider audience. Noah is forging ahead with lsif-java features, and will aim to have Gradle support by the end of the milestone.

Goal: LSIF indexers can operate in monorepos architecture. Garo will be merging their work on incremental indexing into the main branch of the relevant repositories. After this is complete, we should be able incrementally index source code in customer environments (for at least Go) and accept partial indexes in the backend.

Goal: Provide best effort auto-indexing for supported languages. Eric will continue building out the auto-indexing infrastructure so we can provide a wide coverage of code intelligence in the cloud environment. By the end of the milestone we should have updated metrics and UI, and will deep-dive into errors we see currently in production to harden the system and make it more reliable.

Uncategorized. Eric and Maria will also be spending time updating metrics, pings, and documentation across the code intelligence feature space.

Availability

If you have planned unavailability this iteration (e.g., vacation), you can note that here.

@efritz 21d
@macraig 21d
@gbrik 14d
@Strum355 20d (26/10/2020)

Tracked issues

@Strum355

• lsif-java: Tracking issue (#14564)
  • Potential for numerous files to be missed on indexing (#35)
  • hovers for stdlib and cross repo symbols (#54)
  • Maven support (#56)
  • Javadoc links support (#57)
  • Java record hover text (#66)
  • Improve gradle project meta inference (#70; PRs: #74)
  • Transform Javadoc hovers to Markdown hovers (#76)
  • WIP: Intercepting of javac calls with gradle (#78)
  • WIP: JDK 8 (+?) Standard Library Hovers (#80)
  • (🏁 21 days ago) Gradle support (#42)
  • (🏁 32 days ago) reference before definition emitted causes miss (#53)
  • (🏁 27 days ago) JRE <15 compatibility (#55; PRs: #65)
  • (🏁 24 days ago) lsif-java: cross/same-file references (prev. same-file only) (#59)
  • (🏁 35 days ago) lsif-java: cross-file references (#60)
  • (🏁 15 days ago) Unexpected hover data for Enum value (#62; PRs: #75)
  • (🏁 31 days ago) First iteration of gradle: fetching of source directories and classpath (#63)
  • (🏁 22 days ago) Decoupled filetree visitor from DocumentIndexer type and tracking javac error count (#68)
  • (🏁 8 days ago) First pass external dependency hovers (#77)
• WIP: Pass one of backward-cpp based backtraces on failure (#32)

Completed

• (🏁 35 days ago) lsif-java: cross-file references (#60)
• (🏁 32 days ago) reference before definition emitted causes miss (#53)
• (🏁 31 days ago) First iteration of gradle: fetching of source directories and classpath (#63)
• (🏁 27 days ago) JRE <15 compatibility (#55; PRs: #65)
• (🏁 24 days ago) lsif-java: cross/same-file references (prev. same-file only) (#59)
• (🏁 22 days ago) Decoupled filetree visitor from DocumentIndexer type and tracking javac error count (#68)
• (🏁 21 days ago) Gradle support (#42)
• (🏁 15 days ago) Unexpected hover data for Enum value (#62; PRs: #75)
• (🏁 14 days ago) Build meta from gradle init script (#74)
• (🏁 8 days ago) First pass external dependency hovers (#77)
• (🏁 1 day ago) Added check for invalid utf8 doc string and cleaning (#31)

@efritz: 13.00d

• 🚚 LSIF-Go Delivery (#13015)
• Update RFC 236 (#14007) 1.00d
• LSIF quickstart is not quick (#14422) 1.00d
• codeintel: Topologically sort git log in gitserver (#14618) 0.50d 🧶
• Write POC LSIF indexer testing utility (#14821) 1.00d
• codeintel: git diffing fails graphql requests related to force-pushed commits (#12588) 0.50d 🐛

Completed: 9.00d

• (🏁 32 days ago) codenotify: Update db package (#14842)
• (🏁 25 days ago) codeintel: Update index UI pages (#14550)
• (🏁 25 days ago) db: Prevent frontend from starting if codeintel-db == frontend db (#14881)
• (🏁 22 days ago) codeintel: Fix bad error reference (#15117)
• (🏁 21 days ago) codeintel: Store commits as bytea in lsif_nearest_uploads (#14679)
• (🏁 20 days ago) doc: Update codeintel sequence diagrams (#15200)
• (🏁 18 days ago) codeintel: Stop writing to duplicate commit column (#15190)
• (🏁 17 days ago) codeintel: Consolidate periodic jobs into frontend (#15136)
• (🏁 17 days ago) debugserver: Explicit shutdown (#15282)
• (🏁 17 days ago) Tracking issue: Generic user code executor (#14822) 2.50d
• (🏁 16 days ago) dead code: Remove unused codeintel store methods (#15356)
• (🏁 16 days ago) dead code: Remove Update function from commit updater (#15358)
• (🏁 16 days ago) dead code: Remove tar package (#15359)
• (🏁 16 days ago) dead code: Remove unused gitserver methods (#15357)
• (🏁 16 days ago) dead code: Remove unused bundle manager client methods (#15364)
• (🏁 16 days ago) dead code: Remove unused bundle manager client methods (#15366)
• (🏁 16 days ago) dead code: Remove unused retry mechanism (#15368)
• (🏁 16 days ago) codeintel: Remove call to bundle manager in query path (#15369)
• (🏁 16 days ago) precise-code-intel-tester: Fix regex pattern (#15372)
• (🏁 15 days ago) codeintel: Inline calls to the upload store in the upload handler (#15375)
• (🏁 15 days ago) codeintel: Remove calls to bundle manager from worker (#15374)
• (🏁 15 days ago) dead code: Remove unused bundle manager client methods (#15405)
• (🏁 15 days ago) dead code: Remove unused lsifstore method (#15404)
• (🏁 15 days ago) dead code: Remove unused code around bundle manager ctor (#15407)
• (🏁 15 days ago) codeintel: Flatten bundle manager client (#15408)
• (🏁 15 days ago) codeintel: Clean up dead persistence files (#15411)
• (🏁 15 days ago) codeintel: Refactor bundle interfaces (#15371)
• (🏁 15 days ago) dead code: Remove sqlite batch inserter (#15360)
• (🏁 15 days ago) codeintel: Remove serializer interface (#15419)
• (🏁 15 days ago) dead code: Remove close methods on lsif stores (#15421)
• (🏁 15 days ago) codeintel: Remove sqlite from build process (#15422)
• (🏁 15 days ago) codeintel: Add bundle id to bundle store methods (#15424)
• (🏁 15 days ago) codeintel: Add bundle id to bundle database methods (#15425)
• (🏁 15 days ago) chore: Move dumpID field assignment into bundle database layer (#15468)
• (🏁 15 days ago) codeintel: Use one store instance in bundle manager client (#15469)
• (🏁 14 days ago) codeintel: Remove bundle manager client (#15474)
• (🏁 14 days ago) codeintel: Remove enterprise/internal/codeintel/bundle/client package (#15481)
• (🏁 14 days ago) codeintel: Remove enterprise/internal/codeintel/bundles/persistence/s… (#15485)
• (🏁 14 days ago) codeintel: Remove enterprise/internal/codeintel/bundles/persistence/postgres package (#15487)
• (🏁 14 days ago) codeintel: Merge LSIF stores (#15489)
• (🏁 14 days ago) codeintel: Merge bundle database into LSIF store (#15492)
• (🏁 14 days ago) codeintel: Remove enterprise/internal/codeintel/bundles/types package (#15494)
• (🏁 14 days ago) codeintel: Remove enterprise/internal/codeintel/bundles package (#15495)
• (🏁 14 days ago) codeintel: Reorganize files/defs in lsifstore package (#15496)
• (🏁 14 days ago) codeintel: Inline commits updater into the periodic handler (#15497)
• (🏁 14 days ago) codeintel: Move httpapi package into enterprise/cmd/frontend/internal (#15498)
• (🏁 14 days ago) codeintel: Remove dead queue package (#15499)
• (🏁 14 days ago) codeintel: Move auto-indexing packages (#15501)
• (🏁 14 days ago) codeintel: Move resolver/api packages (#15502)
• (🏁 14 days ago) codeintel: Rename uploadstore package (#15504)
• (🏁 14 days ago) codeintel: Group store packages in subdir (#15505)
• (🏁 14 days ago) chore: Match package name to directory (#15507)
• (🏁 14 days ago) codeintel: Remove bundle manager url envvar (#15508)
• (🏁 11 days ago) codeintel: Remove bundle manager (#15490)
• (🏁 10 days ago) codeintel: Remove observability layers (#14609; PRs: #15534) 1.00d 🧶
• (🏁 10 days ago) codeintel: Update metrics (#15541)
• (🏁 10 days ago) doc: Add page on how to add support for a language (#15594)
• (🏁 10 days ago) language support: Treat pc/pcc files as C/C++ (#15595)
• (🏁 9 days ago) Tracking issue: Remove the precise code intel bundle manager (#14820) 3.00d
• (🏁 8 days ago) Go to definition takes me to wrong line; wrong tokens highlighted on hover (#14015) 🐛
• (🏁 8 days ago) event_logs: Reduce query expression duplication (#15665)
• (🏁 8 days ago) event_logs: Collect codeintel and search data separately (#15666)
• (🏁 6 days ago) Update code intelligence ping data (#14574; PRs: #15635) 0.50d
• (🏁 4 days ago) Add total number of unique users on code intel to pings (#14673; PRs: #15635) 0.50d
• (🏁 3 days ago) codeintel: Add feature flags for auto-indexing (#15804)
• (🏁 3 days ago) codeintel: Retry pulling upload from MinIO (#15808)
• (🏁 3 days ago) codeintel: Drop unused commit column (#15192)
• (🏁 1 day ago) doc: Update codeintel indexer docs (#15818)
• (🏁 1 day ago) RFC 201: Tracking issue (#13891) 1.50d

@gbrik

• doesn't work on arch linux (#1) 🐛
• Create successful and reproducible indexes of 20 OSS repos (#12)
• remove all the dead code (#19)
• no output produced for seemingly well-formed compile_commands.json (#4) 🐛
• merge MVP of incremental indexing (#14863)

Completed

• (🏁 14 days ago) fix imports (#24)
• (🏁 4 days ago) definitions aren't output for templates (#14; PRs: #28) 🐛
• (🏁 today) add LSIF repl (#15987)

@macraig: 1.00d

Completed: 1.00d

• (🏁 15 days ago) Find reference & go to definition not showing up sometimes (sourcegraph.com) (#13802) 0.50d 🐛
• (🏁 6 days ago) Update code intelligence ping data (#14574; PRs: #15635) 0.50d

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🎩 Quality of life
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[macraig]

Last Week [Oct 19 to Oct 23]

• Migrated all GitHub projects to new Iterations and Backlog projects
• Cleaned/followed up on some issues that were stale
• Wrote the first version of Indexing a popular Go repo tutorial
• Gathered info and docs to prepare for CE training session
• Started documenting State of Indexers and when should we consider an indexer ready

Next Week [Oct 26 to Oct 30]

• Tweak tutorial to make it useful for both CE team and general public users
• Discuss and document the indexer's breakdown with the team to unblock roadmap revision

[efritz]

General notes

We began revisiting our team's goals. I don't think our plan is wrong, but the prose we use to describe it in our team page and in the product roadmap has been identified as problematic. We're having meetings as a team to better define the terminology we need to effectively communicate this to the greater Sourcegraph community. Doing this before a new manager onboards will certainly help them understand/align with the current heading of the team more effectively.

Completed 2020/10/20-23

Wrote and received approvals for RFC 253: Sunset the bundle manager. Started implementing the storage layer to replace the bundle manager so we can remove an entire service worth of complexity from the the code intel path. Waiting on reviews.

Met with @AlicjaSuska to get feedback on the code intel administrative UI and reworked it visually. This was a necessary pre-step to adding additional features into these pages (particularly the configuration of auto-index jobs) as the code was written by a react amateur (still am, but was too) in a way that would have made it tedious to modify. Feels nice now. Waiting on an approval from someone who knows frontend things.

Met with @sourcegraph/campaigns to have them live-review some PRs for the auto-indexer code. I am currently refactoring the auto-indexer code into a generic firecracker executor service, and a generic queue service that feeds the executor jobs. This may or may not be used by campaigns or other teams in the future, but was still an effective way to refresh the code now that we understand the use cases a little better. Code is organized better and the tests are more comprehensive. These reviews are approved and I'm waiting to merge these Monday.

Planned 2020/10/26-30

Will merge the approved executor/executor-queue PRs and continue working on https://github.com/sourcegraph/sourcegraph/issues/14822. The work here is already done in an experimental branch, so I just have to cut the remaining pieces into digestible PRs. The remainder of the work only affects the executor proxy in the frontend to the (which already exists as the precise-code-intel-indexer service, but targets slightly different routes). The deployment of this code should coincide with the deployment of the new executors in the sourcegraph-code-intel GCP project. Not doing so will make it so nothing is consuming the index queue.

Will continue to excise the bundle manager from the rest of the code base. The hard part here is mostly done. The rest may go slowly but it's due to deployment barriers (simply the turn around time of having something merged into cloud).

Might try to select a few of the open issues that aren't nested in another tracking issue to get them out of the way. Then, I'll jump back into https://github.com/sourcegraph/sourcegraph/issues/13891 as my next big focus.

[Strum355]

Completed 2020/10/20 - 2020/10/23

• Basic gradle integration that works for standard single-project scenarios
• Above integration has some support for multi-project scenarios (symbol resolving is whack, has not been dove into)
• Completed rewrite to Kotlin to support JRE 8+ while also having all the comfy niceties of Kotlin (espc regarding nulls)

Planned 2020/10/27 - 2020/10/30

• Improve multi-project scenarios, may involve compiling depending sources on demand TBD
• Explore using gradle init scripts with the tooling API for a more buildscript querying capabilities than the tooling API generally exposes to us (bad gradle 😠 )
• Maybe explore some concurrency/multithreading if I have the time