Create Repository API should validate file system repository paths for collisions

[cjcenizal]

Per convo with @original-brownbear, we advise users against mounting the same path in multiple repositories, or else this will result in weird errors or corruption.

We can improve the UX and help users avoid this situation by updating the Create Repository API to comparing the path defined in the request against those defined in existing repositories, and rejecting the request if there's a collision.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Can you give us a sense of how often users encounter problems that this proposal would fix @cjcenizal? Also a sense of where the protections that exist today break down? I think the errors you get in recent versions are fairly clear, they'll indicate that the repository was concurrently modified.

Avoiding collisions at registration time is a hard problem to solve: we can check for literal path equality against other repositories in the cluster but should we also try and resolve symlinks? What about duplicate NFS mounts? With other repository types there are other ways to alias the same storage to different locations, do we need to detect them too? And we can't easily prevent a different cluster from mounting the same repository either. I see that we can make it a little stricter at registration time but it won't be much stricter, certainly not watertight, so the question is how valuable this is to add.

[cjcenizal]

Can you give us a sense of how often users encounter problems that this proposal would fix

No, sorry, I don't have any data on how frequently users hit this error or create file system repositories with the UI. Does the ES team have any telemetry on the percentage of repos that use the file system? That would give a rough idea of the potential for hitting this problem among our users.

we can check for literal path equality against other repositories in the cluster but should we also try and resolve symlinks? What about duplicate NFS mounts? With other repository types there are other ways to alias the same storage to different locations, do we need to detect them too?

Just a check for literal path equality would have helped me in this case. I'd settle for solving just the concrete problem I hit --"progress over perfection." :)

[Leaf-Lin]

I'm seeing quite a few of this happening lately, some of these are with frozen nodes. I'm guessing with the searchable snapshot setup, it may not be entirely clear to users that they can re-use the same repo?

It would be great if Elasticsearch is able to return errors at register time if the repo has been written to and is not registered with a readonly flag. Is it possible to create a marker in the path so that when another snapshot getting registered on the same root path, the marker should refuse to be overwritten and result in an error to be returned to user?

[kunisen]

Previously it seems the issue happened more when duplicate snapshot repositories are registered. However, @renshuki and I also observed it may happen even when there's a single snapshot repository.
Error message search took me here.
Thanks!

[DaveCTurner]

What error message do you mean @kunisen?

[kunisen]

Thanks and sorry I forgot to mention that @DaveCTurner
It's this error message:

[2023-01-31T04:00:00,001][ERROR][org.elasticsearch.xpack.slm.SnapshotLifecycleTask] 
[instance-00000000XX] failed to create snapshot for snapshot lifecycle policy [cloud-snapshot-policy]: 
RepositoryException[[found-snapshots] Could not read repository data because the contents of the repository do not match its expected state. 
This is likely the result of either concurrently modifying the contents of the repository by a process other than this cluster or an issue with the repository's underlying storage. 
The repository has been disabled to prevent corrupting its contents. 
To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents.]

Basically, we've seen this several times due to registering dup repositories.
But this time we found it's a single repository.

We didn't actually know the reason but unmounting and remounting the repository did the trick.
I will tag you in the internal ticket in case you have interest to look at it.

[DaveCTurner]

The internal ticket was a case where multiple clusters ended up writing to the same repository.