Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code Intelligence 3.21 Tracking issue #13987

Closed
efritz opened this issue Sep 21, 2020 · 13 comments
Closed

Code Intelligence 3.21 Tracking issue #13987

efritz opened this issue Sep 21, 2020 · 13 comments
Assignees
Labels
team/graph Graph Team (previously Code Intel/Language Tools/Language Platform) tracking
Milestone

Comments

@efritz
Copy link
Contributor

efritz commented Sep 21, 2020

Plan

Goal: Get the top three most popular language indexers: lsif-go, lsif-java, lsif-clang stable by working directly on implementation with customers to identify and resolve any remaining issues that prevent the tool from being use in a wide release. This goal takes top priority for this milestone and each member of the team is leading the initiative on an indexer.

Goal: Improve the reliability and the precise code intel backend by migrating data away from SQLite and into a more suitable database, Postgres.

Goal: Improve best effort auto-indexing for supported languages (see issues related to RFC 201).

Availability

Period is from 2020-09-20 to 2020-10-20. Please write the days you won't be working and the number of working days for the period.

  • Aida: 10d
  • Eric: 19d (off 2020/09/28-30)
  • Noah: 21d (off 2020/10/02)
  • Garo: 10d (off 2020/10/05-09)

Tracked issues

@unassigned

Completed

  • (🏁 7 days ago) codeintel: Generalize postgres bulk upload (#14608; PRs: #14637) 🧶

@Strum355

  • Find references to common Java class method name without false-positives (#3418)
  • 🚚 LSIF-Java Delivery (#13017)

Completed

  • (🏁 27 days ago) LSIF-JVM Research Spike (#14057) 🕵️
  • (🏁 8 days ago) codeintel: Faster online find closest commit operations (#14486; PRs: #14581)

@aidaeology: 1.00d

Completed: 1.00d

  • (🏁 27 days ago) Add history of code intelligence (#1628) :shipit:
  • (🏁 22 days ago) Add overview and instructions for using campaigns to enable precise code intel (#14224) 0.50d
  • (🏁 4 days ago) Navigation to definition unavailable from occurrence/usage of symbol in source (#13200) 0.50d 🐛

@beyang: 4.00d

  • 🚚 LSIF-clang Delivery (#12349) 4.00d

@efritz: 9.00d

  • RFC 235 Tracking issue: Move LSIF data to Postgres (#13882) 4.50d
    • Write announcement post for Postgres change (#14009)
    • (🏁 25 days ago) codeintel: RFC 235: Soft delete upload records (#13822) :shipit:
    • (🏁 18 days ago) RFC 235: Add code intel postgres container (#13883; PRs: #13864, #13904, #13924) 1.00d
    • (🏁 18 days ago) RFC 235: Configure connection to codeintel database (#13884; PRs: #13864, #13952) 0.50d
    • (🏁 18 days ago) RFC 235: Add migration infrastructure to codeintel database (#13885; PRs: #13903, #13943) 1.00d
    • (🏁 14 days ago) RFC 235: Update bundle manager to read from Postgres (#13886; PRs: #13923, #13924) 0.50d
    • (🏁 13 days ago) RFC 235: Update bundle manager janitor (#13887; PRs: #13923, #14468)
    • (🏁 13 days ago) RFC 235: Migrate SQLite data to Postgres (#13888; PRs: #13923, #13932) 0.50d
    • (🏁 13 days ago) RFC 235: Update worker to write to Postgres (#13889; PRs: #13923, #13946) 0.50d
    • (🏁 32 days ago) RFC 235: Add code intel postgres image (#13912; PRs: #13913) 0.50d
    • (🏁 32 days ago) chore: Co-locate dev scripts to interact with postgres (#13942) :shipit:
    • (🏁 32 days ago) chore: Set exec bit on docker-images/codeintel-db/build.sh (#13955) :shipit:
    • (🏁 18 days ago) fix: Ensure frontend connects to frontend database (#14379) :shipit:
    • (🏁 8 days ago) RFC 235: Do not mush data together in the server deployment (#14464; PRs: #14487)
    • (🏁 13 days ago) codeintel: Run migrations on startup for codeintel database (#14484) :shipit:
    • (🏁 13 days ago) codeintel: Do not block bundle manager startup with migrations (#14493) :shipit:
    • (🏁 6 days ago) codeintel: Remove bundle manager hop from query path (#14507) :shipit:
    • (🏁 12 days ago) codeintel: Lower number of goroutines during SQLite -> Postgres migration (#14520) :shipit:
    • (🏁 12 days ago) codeintel: De-multiply writes during SQLite -> Postgres migration (#14536) :shipit:
    • (🏁 12 days ago) codeintel: Clean up bundle files after SQLite -> Postgres migration (#14537) :shipit:
    • (🏁 12 days ago) codeintel: Add indexes to codeintel database tables (#14538) :shipit:
    • (🏁 7 days ago) RFC 235: Update janitor (#14566; PRs: #14641)
    • (🏁 6 days ago) RFC 235: Update metrics (#14568; PRs: #14695)
    • (🏁 7 days ago) RFC 235: Find equivalent of the now useless "freeSpace" operation in the janitor (#14569; PRs: #14641)
    • (🏁 7 days ago) RFC 235: Document minimum time required to be spent in 3.21 (#14604; PRs: #14635)
    • (🏁 7 days ago) RFC 235: Update architecture diagram (#14605; PRs: #14633)
  • RFC 201: Tracking issue (#13891) 2.00d
    • RFC 201: Configure CNCF repositories for indexing (#14343; PRs: #14344, #14415) 1.00d
    • RFC 201: Update metrics (#14567) 1.00d
    • (🏁 19 days ago) RFC 201: Write a JSONC index configuration format parser (#13892; PRs: #14356)
    • (🏁 18 days ago) RFC 201: Update auto indexer execution (#13894; PRs: #14359)
    • (🏁 15 days ago) RFC 201: Update index scheduler to publish new payload (#13895; PRs: #14358)
    • (🏁 14 days ago) RFC 201: Create index record based on configuration file (#13896; PRs: #14358, #14416)
    • (🏁 14 days ago) RFC 201: Detect configuration from repo contents (#13897; PRs: #14418)
    • (🏁 27 days ago) codeintel: Refactor command runner in indexer (#14102) :shipit:
    • (🏁 27 days ago) codeintel: Lower indexer output to debug level (#14103) :shipit:
    • (🏁 27 days ago) codeintel: Refactor index command construction (#14105) :shipit:
    • (🏁 27 days ago) codeintel: Move away from hardcoding image names (#14114) :shipit:
    • (🏁 27 days ago) codeintel: Refactor construction of copyfiles flags (#14116) :shipit:
    • (🏁 27 days ago) codeintel: Add orderedKeys to indexer (#14117) :shipit:
    • (🏁 19 days ago) codeintel: Make docker/firecracker abstraction in indexer (#14121) :shipit:
    • (🏁 18 days ago) migration: Update default indexer/indexer_args in lsif_indexes table (#14360) :shipit:
    • (🏁 18 days ago) codeintel: Update default indexer args in index scheduler (#14378) :shipit:
    • (🏁 18 days ago) migration: Fix bad default in indexer records (#14380) :shipit:
    • (🏁 18 days ago) codeintel: Fix bad default lsif-go arguments (#14386) :shipit:
    • (🏁 15 days ago) RFC 201: Add disk size to VM (#14390; PRs: #14412)
    • (🏁 15 days ago) RFC 201: Create index record based on database configuration (#14409; PRs: #14415)
    • (🏁 15 days ago) codeintel: Remove temporary files from firecracker VMs on startup (#14411) :shipit:
    • (🏁 14 days ago) codeintel: Add go mod download step to go index jobs (#14452) :shipit:
    • (🏁 11 days ago) codeintel: Store index job commands and output in the database (#14457) :shipit:
  • Slow Precise Code-Intel Tooltips in C/C++ Code (#13733)
  • codeintel: git diffing fails graphql requests related to force-pushed commits (#12588) 0.50d 🧶
  • 🚚 LSIF-Go Delivery (#13015)
  • Update RFC 236 (#14007) 1.00d
  • codeintel: Store commits as bytea in lsif_nearest_uploads (#14679) :shipit:
  • Remove -endpoint from src-cli usage (#11878) 0.50d 🧶

Completed: 5.00d

  • (🏁 32 days ago) chore: Co-locate dev scripts to interact with postgres (#13942) :shipit:
  • (🏁 32 days ago) RFC 235: Add code intel postgres image (#13912; PRs: #13913) 0.50d
  • (🏁 32 days ago) chore: Set exec bit on docker-images/codeintel-db/build.sh (#13955) :shipit:
  • (🏁 29 days ago) 504 Gateway Timeouts when mousing over after the page has loaded for a while (#12930) 🐛
  • (🏁 29 days ago) LSIF uploads fail with abbreviated OID (#13957; PRs: #14005) 0.50d
  • (🏁 29 days ago) Fix retries in src-cli lsif upload (#14008)
  • (🏁 28 days ago) tracking-issue: Better nested tracking issue estimates (#14035) :shipit:
  • (🏁 27 days ago) codenotify: Configure efritz's subscriptions (#14060) :shipit:
  • (🏁 27 days ago) codeintel: Refactor command runner in indexer (#14102) :shipit:
  • (🏁 27 days ago) codeintel: Lower indexer output to debug level (#14103) :shipit:
  • (🏁 27 days ago) codeintel: Refactor index command construction (#14105) :shipit:
  • (🏁 27 days ago) codeintel: Move away from hardcoding image names (#14114) :shipit:
  • (🏁 27 days ago) codeintel: Refactor construction of copyfiles flags (#14116) :shipit:
  • (🏁 27 days ago) codeintel: Add orderedKeys to indexer (#14117) :shipit:
  • (🏁 26 days ago) dbworker: Pass sql options to TransactableHandle (#14044; PRs: #14061, #14063)
  • (🏁 26 days ago) tracking-issue: Order finished work chronologically (#14124) :shipit:
  • (🏁 25 days ago) codeintel: RFC 235: Soft delete upload records (#13822) :shipit:
  • (🏁 19 days ago) codeintel: No longer able to upload repos which are currently cloning (#14052; PRs: #14141) 🐛
  • (🏁 19 days ago) codeintel: Explicitly list CNCF repositories for auto-indexing (#14344) :shipit:
  • (🏁 19 days ago) codeintel: Remove some wrappers from a previous abstraction (#14142) :shipit:
  • (🏁 19 days ago) codeintel: Make docker/firecracker abstraction in indexer (#14121) :shipit:
  • (🏁 19 days ago) codeintel: Fix values in fake repo usage stats for CNCF repos (#14353) :shipit:
  • (🏁 19 days ago) codeintel: Ensure CNCF values override real values (#14355) :shipit:
  • (🏁 19 days ago) RFC 201: Write a JSONC index configuration format parser (#13892; PRs: #14356)
  • (🏁 18 days ago) migration: Update default indexer/indexer_args in lsif_indexes table (#14360) :shipit:
  • (🏁 18 days ago) Tracking issue to track Eric's tracking issue improvements in 3.21 (#14264)
  • (🏁 18 days ago) RFC 235: Add migration infrastructure to codeintel database (#13885; PRs: #13903, #13943) 1.00d
  • (🏁 18 days ago) fix: Ensure frontend connects to frontend database (#14379) :shipit:
  • (🏁 18 days ago) codeintel: Update default indexer args in index scheduler (#14378) :shipit:
  • (🏁 18 days ago) RFC 201: Update auto indexer execution (#13894; PRs: #14359)
  • (🏁 18 days ago) migration: Fix bad default in indexer records (#14380) :shipit:
  • (🏁 18 days ago) codeintel: Fix bad default lsif-go arguments (#14386) :shipit:
  • (🏁 18 days ago) RFC 235: Add code intel postgres container (#13883; PRs: #13864, #13904, #13924) 1.00d
  • (🏁 18 days ago) RFC 235: Configure connection to codeintel database (#13884; PRs: #13864, #13952) 0.50d
  • (🏁 15 days ago) RFC 201: Update index scheduler to publish new payload (#13895; PRs: #14358)
  • (🏁 15 days ago) codeintel: Remove temporary files from firecracker VMs on startup (#14411) :shipit:
  • (🏁 15 days ago) RFC 201: Add disk size to VM (#14390; PRs: #14412)
  • (🏁 15 days ago) RFC 201: Create index record based on database configuration (#14409; PRs: #14415)
  • (🏁 14 days ago) RFC 201: Create index record based on configuration file (#13896; PRs: #14358, #14416)
  • (🏁 14 days ago) codeintel: Fix proper suffix for project root check (#14428) :shipit:
  • (🏁 14 days ago) RFC 201: Detect configuration from repo contents (#13897; PRs: #14418)
  • (🏁 14 days ago) codeintel: Add go mod download step to go index jobs (#14452) :shipit:
  • (🏁 14 days ago) RFC 235: Update bundle manager to read from Postgres (#13886; PRs: #13923, #13924) 0.50d
  • (🏁 14 days ago) Server deployment panic due to precise-code-intel-bundle-manager (#14467; PRs: #14469)
  • (🏁 13 days ago) RFC 235: Update bundle manager janitor (#13887; PRs: #13923, #14468)
  • (🏁 13 days ago) codeintel: Run migrations on startup for codeintel database (#14484) :shipit:
  • (🏁 13 days ago) RFC 235: Update worker to write to Postgres (#13889; PRs: #13923, #13946) 0.50d
  • (🏁 13 days ago) RFC 235: Migrate SQLite data to Postgres (#13888; PRs: #13923, #13932) 0.50d
  • (🏁 13 days ago) codeintel: Do not block bundle manager startup with migrations (#14493) :shipit:
  • (🏁 12 days ago) codeintel: Lower number of goroutines during SQLite -> Postgres migration (#14520) :shipit:
  • (🏁 12 days ago) dev: Do not crash startup script on unbound PG variables (#14491) :shipit:
  • (🏁 12 days ago) codeintel: De-multiply writes during SQLite -> Postgres migration (#14536) :shipit:
  • (🏁 12 days ago) codeintel: Clean up bundle files after SQLite -> Postgres migration (#14537) :shipit:
  • (🏁 12 days ago) codeintel: Add indexes to codeintel database tables (#14538) :shipit:
  • (🏁 11 days ago) codeintel: Store index job commands and output in the database (#14457) :shipit:
  • (🏁 8 days ago) codeintel: Faster online find closest commit operations (#14486; PRs: #14581)
  • (🏁 8 days ago) RFC 235: Do not mush data together in the server deployment (#14464; PRs: #14487)
  • (🏁 7 days ago) crashfix: SendUserEmailOnFieldUpdate crashes on users without an email (#14648) :shipit:
  • (🏁 7 days ago) RFC 235: Update architecture diagram (#14605; PRs: #14633)
  • (🏁 7 days ago) dbutil: Extract and reuse postgres batch inserter (#14637) :shipit:
  • (🏁 7 days ago) Add postgres bulk insertion mode for tracing (#14624) :shipit:
  • (🏁 7 days ago) RFC 235: Update janitor (#14566; PRs: #14641)
  • (🏁 7 days ago) codeintel: Stop invalid GetState queries from being executed (#14677) :shipit:
  • (🏁 7 days ago) RFC 235: Document minimum time required to be spent in 3.21 (#14604; PRs: #14635)
  • (🏁 7 days ago) RFC 235: Find equivalent of the now useless "freeSpace" operation in the janitor (#14569; PRs: #14641)
  • (🏁 6 days ago) codeintel: Remove bundle manager hop from query path (#14507) :shipit:
  • (🏁 6 days ago) RFC 235: Update metrics (#14568; PRs: #14695)
  • (🏁 6 days ago) chore: Fix enterprise frontend build (#14721) :shipit:
  • (🏁 6 days ago) chore: Fix enterprise server build (#14722) :shipit:
  • (🏁 6 days ago) codeintel: Clean up internal HTTP server setup (#14733) :shipit:

@gbrik: 5.00d

  • 🚚 LSIF-clang Delivery (#12349) 4.00d

Completed: 1.00d

  • (🏁 19 days ago) infer project root automatically if not specified (#15)
  • (🏁 19 days ago) doesn't build on ubuntu 18.04 (#3) 🐛
  • (🏁 1 day ago) Investigate effort for Bazel integration (#13202) 1.00d 🕵️
  • (🏁 1 day ago) github.com/nlohmann/json on macOS fails (#13) 🐛

@macraig: 0.50d

Completed: 0.50d

  • (🏁 13 days ago) Code Intel - LSIF Indexing CNCF effort tracking (#14348)
  • (🏁 4 days ago) Navigation to definition unavailable from occurrence/usage of symbol in source (#13200) 0.50d 🐛

Legend

  • 👩 Customer issue
  • 🐛 Bug
  • 🧶 Technical debt
  • 🎩 Quality of life
  • 🛠️ Roadmap
  • 🕵️ Spike
  • 🔒 Security issue
  • :shipit: Pull Request
@efritz efritz added team/graph Graph Team (previously Code Intel/Language Tools/Language Platform) tracking labels Sep 21, 2020
@efritz efritz added this to the 3.21 milestone Sep 21, 2020
@efritz efritz self-assigned this Sep 21, 2020
@efritz
Copy link
Contributor Author

efritz commented Sep 21, 2020

Planned 2020/09/21-25

This week I will focus on breaking down the work on RFC 235 (migrating SQLite data to Postgres) into manageable/reviewable changes. I'll also be available for pairing, design, and shadowing some of Aida's tasks to take over my share of the interim management responsibilities.

@Strum355
Copy link
Member

Planned 2020/09/21-2020/09/25

  • Finish RFC 238
  • Mystery hack with Eric 👻 👁️ oooo
  • Act on RFC 238 or be cast into limbo to convince the spirits that lsif-java should be rewritten in Kotlin

@efritz efritz changed the title WIP: Code Intelligence 3.21 Tracking issue Code Intelligence 3.21 Tracking issue Sep 24, 2020
@Strum355
Copy link
Member

Completed 2020/09/21 - 2020/09/25

Planned 2020/09/28 - 2020/10/01

  • More of com.sun.source MVP
  • Discuss taking on easy-issue for code-intel in sourcegraph/sourcegraph

@shrouxm
Copy link
Contributor

shrouxm commented Sep 28, 2020

Completed 2020/09/21 - 2020/09/25

  • successfully uploaded LSIF index of tensorflow + linux kernel
  • impromptu started bazel investigation in order to upload tensorflow
  • rewrote lsif-clang docs

Planned

2020/09/21

  • Apply common lsif-clang arguments automatically.
  • Create lsif-clang showcase using existing OSS repos.

2020/09/23

  • Research Bazel.
  • Write RFC proposing path forward for Bazel integration.

2020/09/24

  • Finish outstanding work from first two days.
  • Continue to clean up minor issues on the lsif-clang repository.

Reminder: Out next week

@efritz
Copy link
Contributor Author

efritz commented Sep 29, 2020

Completed 2020/09/21-25

Lots of tracking issue improvements. Implemented the majority of RFC 235/RFC 201 work. Most of the effort remaining here will just be on getting the necessary infra reviews and deploying in the correct order.

Planned 2020/10/01-02

Will start to work more deeply with @garo on RFC 201 work, as this has a lot of overlap with incremental indexing.

@shrouxm
Copy link
Contributor

shrouxm commented Oct 2, 2020

Completed 2020/09/28-2020/10/2

Cleaned up the way lsif-clang accepts arguments, can now be invoked simply as lsif-clang compile_commands.json and it will be easier to add new arguments. Expanded the lsif-clang docs with examples. Switched priority entirely from the Bazel investigation to multiple different delivery efforts, which involved requirements gathering, indexing new repos, and more wading around in build systems.

Planned 2020/10/05-2020/10/09

🛌

@efritz
Copy link
Contributor Author

efritz commented Oct 5, 2020

Completed 2020/10/01-02

Short week for me. Mainly discussed priorities around CNCF and RFC 201. We decided that completion of RFC 201 will give us what we need with respect to preparing the CNCF repogroup, so we'll continue to do that.

Planned 2020/10/05-09

Complete efforts on RFC 235 (deployment priority) and RFC 201 (CNCF priority).

@Strum355
Copy link
Member

Strum355 commented Oct 5, 2020

Completed 2020/09/28 - 2020/10/01

  • definitions emitting with comsunsource ✔️
  • mostly did understanding work of the existing comsunsource repos

Planned 2020/10/05 - 2020/10/09

  • same (and hopefully cross) file refs with comsunsource for simple projects

@efritz
Copy link
Contributor Author

efritz commented Oct 9, 2020

Completed 2020/10/05-09

Performed migration from SQLite to Postgres (a bit of blood was shed, but a manageable amount). Finished all of the big work to have user-defined index operations run on our side 😎 .

We are now automatically indexing typescript repositories, and we can drop a configuration in the database for any repository we'd like. Will work with Garo next week to see what we can automate for CNCF C++ repositories.

Planned 2020/10/12-16

Tie up a few loose ends in both RFC201 and 235 (I've created additional issues in those tracking issues). I've also planned future work which is not in this milestone (so will show in the RFC tracking issue but not this one). Will prioritize whatever needs to be in private instance for before branch cut.

@Strum355
Copy link
Member

Strum355 commented Oct 9, 2020

Completed 2020/10/05 - 2020/10/09

  • more learning of comsunsource was done ✔️
  • first successful cross-file ref was also done ✔️ Aida would be proud

Planned 2020/10/12 - 2020/10/16

  • Index more construct definitions and same/cross file refs
  • Start to amp up the complexity of the codebases indexed

@shrouxm
Copy link
Contributor

shrouxm commented Oct 12, 2020

Completed 2020/10/05 - 2020/10/09

image

Planned 2020/10/12 - 2020/10/16

@efritz
Copy link
Contributor Author

efritz commented Oct 16, 2020

Completed 2020/10/12-16

Finished the remaining work for RFC 235 (SQLite -> Postgres migration) and RFC 201 (auto-indexer configuration) that was planned for this milestone. Effort will be applied to the remainder of RFC 201 in the next milestone.

Had a meeting with @sourcegraph/campaigns engineers showing them the infrastructure for how code intelligence does safe user-supplied code execution. Did a quick proof of concept to see if we can run src campaigns apply in the same infrastructure. It was a success 🎉.

Plan 2020/10/19

Planning to write (and publish) a blog post before the 20th explaining some engineering decisions that went into introducing a new Postgres instance.

Will spend remaining time working on refactoring the code intel auto-indexer services into something that is not specific to code intelligence. In doing this refactor, code quality, testing, and docs of the indexer will be improved.

@macraig
Copy link
Contributor

macraig commented Oct 16, 2020

Completed 2020/10/05 - 2020/10/09

  • Created new LSIF delivery status tracking doc
  • Reviewed state of code intel docs
  • Reviewed open issues/bugs/feature requests for Code intel milestone planning
  • Started migrating Code Intel GitHub projects to Backlog and Iteration projects (2 projects vs 5)

Plan 2020/10/19

  • Go over new code intel docs and changes
  • Write tutorial for indexing a Go repo
  • Finish migrating issues to new projects

@efritz efritz closed this as completed Oct 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/graph Graph Team (previously Code Intel/Language Tools/Language Platform) tracking
Projects
None yet
Development

No branches or pull requests

5 participants