[WIP] Codeinsights MVP backend

[emidoots]

See #17218 for the general plan / work being done on code insights.

I am working from this codeinsights branch (which will be unstable/buggy/broken sometimes) and my aim is to have this branch by EOW run a very basic code insights backend in dev environments, including all the app infra/changes we need, running timescaledb, doing the DB migrations to create the schema, and the GraphQL schema/API/backend making requests and serving data from timescale.

This branch itself won't be merged, rather I will be taking changes from here once they are solid and sending individual PRs for review (so they meet our quality bar, are not "one big PR", etc.) I will be posting updates here as I send out those PRs.

TODO:

• Grafana dashboards -- issue filed elsewhere
• testing: An end-to-end test where we use the GraphQL API to modify user settings/add insights, and the GraphQL API to query data after a while? -- issue filed elsewhere
• Investigate adding more tests for enterprise/internal/insights/background/* - see https://github.com/sourcegraph/sourcegraph/pull/18267 -- issue filed elsewhere
• Basic webhook fetching support -- decided to adopt push model, see other issue
• Deploy TimescaleDB in non-dev envs
• Generate schema.insights.md documentation
• dev/db/squash_migrations.sh starts Postgres 9.6 if the connected DB version != 9.6, so it doesn't work with these migrations.
• internal/db/schemadoc/main.go starts Postgres 9.6 if the connected DB version != 9.6, so it doesn't work with this DB schema yet.
• Add code TODO for retroactively updating repo names in DB

Feb 21-26:

• insights: correct over-reporting / aggregation of data points insights: correct over-reporting / aggregation of data points #18632

Feb 15-19:

• Basic search query execution & storage
• Investigate using go-mockgen for GraphQL resolver tests: https://github.com/sourcegraph/sourcegraph/pull/18075#discussion_r572359571

Feb 8-9:

• Implement DB store tests
• Merge DB schema
• Implement GraphQL backend tests

Feb 1-5:

• Implement DB store
• Implement GraphQL backend
• Investigate replacing repo_names and metadata with https://docs.timescale.com/latest/using-timescaledb/compression (Tomas suggested, looks really cool) -- CONCLUSION: https://github.com/sourcegraph/sourcegraph/pull/17227#issuecomment-773704129
• Start TImescaleDB in testing env; run migrations enterprise/internal/insights: add basic store package + testing infrastructure #17733 insights: execute database migrations #17586

Jan 18-22:

• Determine where to store insights (user/org/global settings? likely not DB for now, but file an issue for later?)
• Run migrations on server startup
• Plan DB schema & queries/inserts
• Generate separate schema.md files for codeintel/frontend DBs so we can generate schema.insights.md later
• Tag TimescaleDB Docker image as sourcegraph/codeinsights-db
• Run codeinsights-db as part of dev server.
• Add migrations/codeinsights DB migration foundation
• Add GraphQL backend scaffolding / stubs

[emidoots]

PR to generate separate schema.md files for codeintel/frontend DBs so we can generate a schema.md for code insights DB schema soon: https://github.com/sourcegraph/sourcegraph/pull/17228

[codecov]

Codecov Report

Merging #17227 (ec41897) into main (b96444a) will increase coverage by 0.10%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #17227      +/-   ##
==========================================
+ Coverage   51.88%   51.98%   +0.10%     
==========================================
  Files        1713     1707       -6     
  Lines       84963    84926      -37     
  Branches     7748     7510     -238     
==========================================
+ Hits        44082    44150      +68     
+ Misses      36978    36873     -105     
  Partials     3903     3903              

Flag	Coverage Δ
go	51.03% <ø> (+0.19%)	⬆️
integration	30.53% <ø> (-0.16%)	⬇️
storybook	30.05% <ø> (-0.36%)	⬇️
typescript	54.29% <ø> (-0.12%)	⬇️
unit	34.80% <ø> (+0.04%)	⬆️

Impacted Files	Coverage Δ
internal/db/dbutil/dbutil.go	16.82% <ø> (ø)
cmd/frontend/graphqlbackend/search_structural.go	44.56% <0.00%> (-15.44%)	⬇️
client/web/src/enterprise/site-admin/routes.ts	6.66% <0.00%> (-10.99%)	⬇️
cmd/frontend/graphqlbackend/search_symbols.go	10.00% <0.00%> (-8.83%)	⬇️
...nal/campaigns/resolvers/changeset_apply_preview.go	58.97% <0.00%> (-5.48%)	⬇️
internal/db/users.go	47.49% <0.00%> (-3.37%)	⬇️
cmd/frontend/auth/user.go	76.11% <0.00%> (-2.99%)	⬇️
cmd/frontend/graphqlbackend/zoekt.go	75.46% <0.00%> (-2.98%)	⬇️
internal/db/external_accounts.go	61.11% <0.00%> (-2.87%)	⬇️
enterprise/internal/campaigns/syncer/syncer.go	61.71% <0.00%> (-2.61%)	⬇️
... and 157 more

[emidoots]

Ongoing discussion about this: https://sourcegraph.slack.com/archives/C014ZCKMCAV/p1610764544015200

[emidoots]

Ongoing discussions about this:

• repo vs. global vs. other association of events: https://sourcegraph.slack.com/archives/C014ZCKMCAV/p1610763511014200
• what happens if repositories get renamed/removed/etc: https://sourcegraph.slack.com/archives/C014ZCKMCAV/p1610762810012800

[emidoots]

PR to tag the TimescaleDB image as sourcegraph/codeinsights-db: https://github.com/sourcegraph/sourcegraph/pull/17427

[emidoots]

PR to run codeinsights-db (TimescaleDB) as part of dev server: https://github.com/sourcegraph/sourcegraph/pull/17431

[emidoots]

PR to add codeinsights-db migrations foundation: https://github.com/sourcegraph/sourcegraph/pull/17432

[emidoots]

• insights: change series_id to string (hash) insights: change series_id to string (hash); add index #18230

[emidoots]

• insights: store: query metadata & other minor improvements insights: store: query metadata & other minor improvements #18254

[emidoots]

• insights: store: add support for recording data points insights: store: add support for recording data points #18255

[emidoots]

• insights: add new discovery package for locating insights (and use it) insights: add new discovery package for locating insights (and use it) #18264

[emidoots]

• migrations: add insights_query_runner_jobs table migrations: add insights_query_runner_jobs table #18265

[emidoots]

Data size & retention policy speculation

In the original RFC we speculated the backend:

• Must support insights over 16,000 repos.
• Ideally over our largest customers ~500k+ repos.
• We speculated customers would at max have ~500 dashboards (spread across ~2k or so devs)
• We speculated there may be 5 panels per dashboard, each with a one to a handful of data series recorded per-repo.
• Examples of series cardinality included:
  • A panel showing 20 java versions, recorded per repo
  • A panel showing 40 languages, recorded per repo
  • A panel showing PR states(open/close/merged), recorded per repo - assuming a max of like 20,000 PRs.
• In an ideal world, keeping data for some long period of time like 6-12mo (but maybe with limitations as we start to talk about older data.)

We will obviously need horizontal scaling in the future to achieve the much larger scales (which should be easy to add with TimescaleDB, but I haven't looked into it.)

In theory, the best interval at which to record data points would be whenever repositories receive new commits. We are doing global searches (currently) at some interval, though, as that is a fair bit easier to implement - and so a question is at what interval we should do that for all repositories.

I would speculate that given all repositories on a Sourcegraph instance - up to 500k - we are unlikely to see a commit push frequency higher than every 5 minutes in general.

We can then speculate on the # of table rows per recording interval based on:

• 20-500k repos
• 10-500 dashboards with ~5 panels each (50-2500 panels)
• 5-40 semi-unique series per dashboard (let's say 50% are reused / same queries.)
• # of table rows per recording interval == repos * dashboard_panels * unique_series_per_dashboard

Some examples:

• 20k_repos50_panels5_series == 20,000,000 rows / recording interval (e.g. 5 minutes)
• 500k_repos2500_panels40_series == 50,000,000 rows/recording interval (e.g. 5 minutes)

Obviously that's a lot of data and will most likely not "just work" magically. https://docs.timescale.com/latest/faq#scaling says they regularly test with 10+ billion rows and inserting 100-200k rows / second. I we assume we have a budget of 10 billion rows, and recorded 20mil/5min we would only be able to store the last 500 recording intervals (1.7 days).

But that's all very speculative, let's try some real

Quick benchmarks

Inserting 1 year of data points for a single repository, with no metadata at a 5 minute interval takes 6-10s using:

INSERT INTO series_points(
    time,
    series_id,
    value,
    metadata_id,
    repo_id,
    repo_name_id,
    original_repo_name_id)
SELECT time,
    0,
    random()*80 - 40,
    (SELECT id FROM metadata WHERE metadata = '{"hello": "world", "languages": ["Go", "Python", "Java"]}'),
    2,
    (SELECT id FROM repo_names WHERE name = 'github.com/gorilla/mux-renamed'),
    (SELECT id FROM repo_names WHERE name = 'github.com/gorilla/mux-original')
    FROM generate_series(TIMESTAMP '2021-01-01 00:00:00', TIMESTAMP '2022-01-01 00:00:00', INTERVAL '5 min') AS time;

Idle memory / CPU usage is quite minimal (41 MiB / 1% CPU), still, and disk usage grows by about 60 MiB. Most of the costs outside of disk storage will probably be incurred at query time - which can be helped via precomputed aggregations Timescale offers.

But the 60 MiB disk usage gives us a pretty good idea: 20,000 repos * 60 MiB == ~1171 GiB of disk storage for 1 year of data points recorded every 5min for 20k repos.

Based on this, I will go with a default recording interval of 10 minutes (we can adjust this if needed later) which would give us ~40k repos worth of data points for a single series type on a reasonable deployment, i.e. it gets us decently far for the initial version.

The next important things to look at for scaling are:

• A data retention policy that deletes data points after 1yr.
• Continuous aggregation, we may be able to do this and keep lesser precision historical data but still at a per-repo level (e.g. one day point per day if older than 3mo.)
• Keep repository-level metrics for the past ~3 months, but then only keep global (non-repo-specific) insights after that.

Lots of stuff/options to explore here in the TimescaleDB docs.

[emidoots]

insights: actually enqueue/record insights from global settings #18314

[emidoots]

• ci: build docker-images/codeinsights-db ci: build docker-images/codeinsights-db #18350
• dev: allow codeinsights-db more time to startup dev: allow codeinsights-db more time to startup #18359
• insights: turn on for Sourcegraph.com insights: turn on for Sourcegraph.com #18360

[emidoots]

insights: correct over-reporting / aggregation of data points #18632