Deal with concurrent checkpointing from both old and new pageserver during timeline migration

[LizardWizzard]

Started from discussion here #874 (comment)

There is a dangerous possibility of two pageservers writing data concurrently to the same underlying s3 storage. This is scary even if it works given the incremental format we use.

There are some questions and invariants we need to uphold. We shouldn't be able to overwrite local metadata of an active timeline with something older/newer from s3. As far as I understand the opposite is not a problem because we save metadata to each checkpoint. Currently local overwrite shouldn't happen because we do not schedule downloads for timelines that are present locally. So this is good.

• What if we have some non determinism in checkpointer so it can create different files on two pageservers?
• There will be two checkpoints on the same lsn, or they'll interleave somehow?
• Even if there are no nondeterminism what if there are two different pageserver versions which use different code to produce checkpoints? E g new format version, changed layout, new files, missing files etc.

Feel free to correct me, maybe I'm missing something

[LizardWizzard]

The current vision on that is that we should allow concurrent checkpointing without ant problems. I created a test for that, but faced unrelated errors which are currently being fixed on main. So I'll continue my investigation

[LizardWizzard]

Note: while concurrent checkpointing shouldn't lead to correctness issues we still might want to avoid that in some future scenarios when we have timeline attached to two pageservers e.g to spread get page requests or to support failover. Currently concurrent checkpointing might happen in the process of tenant migration, when new and old pageservers are active simultaneously

[stepashka]

this is waiting for #1396

[LizardWizzard]

Things changed and we decided to introduce etcd. The new vision is to use it in order to prevent concurrent uploads from happening.

[problame]

I think we can close this, given we're set to implement relocation as specified in RFC #3868

[LizardWizzard]

I think this is still relevant. So issue describes the problem that RFC should solve in one way or another. In first iteration we decided to not have this problem by detaching before the attach so there is no concurrent background activity from more than one pageserver at a time. The project currently takes into account only first stage, so I'm not sure whether we should keep the issue in the project (keep it with separate label?). WDYT @problame?

[shanyp]

fixed by generations