insights: investigate DB snapshotting in CI setup [spike]

[coury-clark]

As part of a patch retro (3.39.1) we discussed that having some snapshots of useful data may make it easier to test various scenarios that are otherwise challenging to test. This issue is to investigate:

1. Do we have any tooling being developed that might help with this? As I understand the dev-ex team might be working on something similar.
2. If not - what tooling do we need to build?

[sourcegraph-bot-2]

Heads up @Joelkw @felixfbecker @vovakulikov @unclejustin - the "team/code-insights" label was applied to this issue.

[jhchabran]

@coury-clark it depends on where you'd want to run this. If that's in CI, not yet but that should not be hard. If that's on preprod, I have an ongoing PR that enables to restore snapshots at the beginning of a build, so in that case, we could take a snapshot after having put the preprod in the state we're after.

[Joelkw]

Goal: spike this in iteration 18 to understand if this helps solve our testing problems.

[leonore]

What we want to achieve, essentially, is to be able to get a snapshot of code insights data from a db in an environment, and then apply that snapshot into another environment, for 2 purposes:

1. This would allow us to add some integration tests we can't currently test for. This is because there's some data (e.g. series_points) that we can't insert through graphql queries.
2. This would allow us to do local debugging of more complex scenarios that we might encounter on remote environments.

[jhchabran]

Update: we worked with @leonore and @davejrt on POC'ing this by pulling out a database dump from the demo and importing the data in local to run some queries. @leonore confirmed that she was able to perform meaningful requests in local that would be good to take into a test suite.

Next step is to POC doing this in CI.

[jhchabran]

@leonore (cc @sourcegraph/code-insights-backend) it was a bit more complicated that expected as the single server image doesn't run code insights and the preprod doesn't either because it runs in dotcom mode.

In the end we opted for deploying a cluster on the fly within the CI and pointing those tests toward it as you can see here https://buildkite.com/sourcegraph/sourcegraph/builds/154809#018167f2-dab4-43b6-8403-084cf47778cd/582-583

We have a bit of flakiness to provision the database, but that's something we can probably work out (cc @davejrt).

At this stage, it validates the POC from our perspective, and we should discuss what's next.

The remaining items I'm seeing here are:

• How can we generate the dataset that your tests will use, so we don't have to snapshot the demo database which is probably going to cause some flakes / annoying to diagnose failures further down the line.
• How can we separate your test suite from the rest.
  • That should not be hard, we can probably extract those from the gqltests or hide them behind a flag.

[leonore]

repurposing this issue for db snapshotting as part of CI to keep track of the conversations/PRs

opened another issue for easier approach to just have a manual dump and apply tool for insights and use dump data for local test: #40024