internal contributor 3.25 Tracking issue

[emidoots]

Plan

Availability

If you have planned unavailability this iteration (e.g., vacation), you can note that here.

Tracked issues

@slimsag

Completed

• (🏁 today) code insights: backend MVP implementation (#17218)

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🎩 Quality of life
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[github-actions]

Heads up @Joelkw @felixfbecker - the "team/code-insights" label was applied to this issue.

[emidoots]

(back from holiday break)

Jan 12th update:

• I've created a Code Insights :: Backend project board and filed a few handfuls of initial issues for tracking work towards the backend MVP https://github.com/orgs/sourcegraph/projects/122
• I am actively working from a codeinsights branch/PR (which will be unstable/buggy/broken frequently) and my aim is to have that branch by EOW run a very basic code insights backend in dev environments, including all the app infra/changes we need, running timescaledb, doing the DB migrations to create the schema, and the GraphQL schema/API/backend making requests and serving data from timescale.
• Throughout the week I'll be sending out PRs to take changes from that PR/branch (once polished enough to merge) and land them into main. Throughout all of this I will post updates on this PR https://github.com/sourcegraph/sourcegraph/pull/17227 and will relay them here as well, starting with this one:

[emidoots]

Jan 15th update:

• Running a bit behind my original goal/target of having a basic version working in dev mode by EOW
• Still moving quickly and have a clear path forward. ETA now next Tue/Wed night for dev version.
• No blockers

[emidoots]

Jan 22nd update:

• Original goal this week was to have the basic version working in prod, I am running behind on that goal by about a week (I had an unexpectedly large amount of unrelated calls/discussions this week, which made me do a lot of context switching)
• I spent a fair amount of time researching how we're doing GraphQL backends for Campaigns, Code Intel, and Code Monitoring as things have changed significantly here since I've last worked with our GraphQL backend code. This turned out to be a fairly large time sink, as there is a non-trivial amount of complexity here I've had to learn and think about how we'll communicate this to our first backend hire.
• I discussed general direction with Tomas, feel really good about the direction of everything, and have made some substantial progress today - I anticipate having the dev env merged into main mid-next week and it mostly working in prod by next Fri, and then iterating from there.

[emidoots]

This week

For https://app.hubspot.com/contacts/2762526/company/407948923/ - I helped with handover to Jonah, and joined discussions between them and Campaigns team about how to get campaigns working for them. I also helped to prepare Q4 highlights for them with Jonah and assisted with other general issues they had.

For the CSE team, I reviewed 3 applications and interviewed 1 candidate.

For the Code Insights team, I spent a majority of my time this week with ~13 PRs landing foundational work for the new code insights backend. I also got the new backend to a point where the GraphQL schema, settings schema, API resolvers, DB store+schema+migrations+dev deployment are mostly finished. See the Code Insights team update for a screenshot demonstrating a GraphQL query with some mock data.

Next week

I am highly focused on code insights backend work and trying to land a working version of it in production as soon as possible so we can iterate further. The primary work that remains to be done is around executing search queries, webhooks, deploying the DB to all deployments, and sending/landing a number of PRs which I have in my work-in-progress branch.

Aside from that, I have 3 CSE applications to review + 2 CSE candidates to interview scheduled currently, so I expect this will take some time away.

[emidoots]

Last week

For the CSE team, I reviewed ~10 CSE hiring project submissions (~30m each), and interviewed 4 CSE candidates.

For the codeintel team, I discussed how we do DB migrations with Eric.

For the code insights team, I have made good progress on the testing suite for the backend GraphQL resolvers and DB store, which should be the last step to getting this thing merged and running prod. I have been investing in architecture diagrams, etc. along the way. Also did an analysis/write-up of whether or not using TimescaleDB compression makes sense.

I discussed IC role/future with managers, completed my 360 reviews, and had a regular amount of 1:1s/customer syncs.

This week

I anticipate having much less meetings/busywork, and think it's time to get code insights running in prod. Aiming to have that merged by EOD tomorrow and then continue iterating from there.

[emidoots]

Last week

I interviewed a CSE candidate, and two Distribution team candidates. I made substantial progress on the code insights backend, getting everything to query insights merged and running in prod and very nearly got everything running end-to-end in production, including background workers that insert the data, etc. minus a PR or two that actually trigger all of that to run. Some highlights:

• Dev server returning data the background workers inserted
• insights frontend architecture diagram
• insights "repo-updater" worker architecture diagram

This week

I am heads down on code insights backend work and expect to spend all of my time here as I polish up what remains for the 3.25 release, which I feel confident we'll have a good outcome in. Aside from doing the polish & technical work here for 3.25, I anticipate doing some work that will continue until end of next week to land extensive documentation, follow-up issues, architecture diagrams, etc. for the next person who picks up the code insights backend. I anticipate this being the place where I can call this project "finished" (although there may of course be more work the insights team may wish to pull me back in for later, or minor things I missed.)

[emidoots]

Last week

We got the new Code Insights backend hooked up to the frontend. I got the new insights backend and TimescaleDB deployed in all production environments, including Sourcegraph.com which required merging upstream (a few hours of work.) I fixed a number of bug/polish issues in the backend before the 3.25 release. After a discussion around next blockers for going to customers, we realized the need for back-filling data on insights (so you don't have to wait months to get useful insights) was extremely important and I shifted focus to working on that.

This week

I estimate there are 3 remaining weeks in total on code insights work, in order of priority:

1. Fixing some lingering bugs (this week)
2. Adding data backfilling support (this week)
3. Brief revised proposal for why we should do bi-weekly releases at Sourcegraph (this week)
4. Supporting narrowing down to repository-level insights (next week)
5. Improving testing infrastructure (next week)
6. Basic webhook support (next week)
7. documentation, architecture diagrams, onboarding material for 1st backend hire (third week)