Missing lock file should be skipped

[ProactiveServices]

If a rest-server cannot access a lock file, which is deleted during the restic session, restic (client) will continue to try to remove the non-existent lock file because it is not aware the lock file has disappeared. To resolve this perhaps restic could get a list of the locks dir before each attempt to query/remove locks. A corner case but it's not my call as to whether to address it, I thought it worth reporting anyway.

Output of restic version

restic 0.9.6 compiled with go1.13.4 on linux/amd64

How did you run restic exactly?

Against a rest-server:
restic check -r <rest server> -p <password> --cacert <file>

using temporary cache in /tmp/restic-check-cache-423914273
repository c4dd5ad3 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-423914273
create exclusive lock for repository
Load(<lock/651e1dc9f8>, 0, 0) returned error, retrying after 582.280027ms: <lock/651e1dc9f8> does not exist
<repeats>

The rest server didn't have permission to access a lock file, which was an unrelated problem of my making. The rest server logs:

GET /repo/locks/651e1dc9f857feb10236dc0678e67a971c2c0d74a276061a6de2049b489d4730
GetBlob()
open /repo/locks/651e1dc9f857feb10236dc0678e67a971c2c0d74a276061a6de2049b489d4730: permission denied

While restic is waiting I removed the lockfile on the server manually then restic (client) retried access to the lockfile, rest-server reported:

GET /repo/locks/651e1dc9f857feb10236dc0678e67a971c2c0d74a276061a6de2049b489d4730
GetBlob()
open /repo/locks/651e1dc9f857feb10236dc0678e67a971c2c0d74a276061a6de2049b489d4730: no such file or directory

...and they continued on.

Restic assumed that the lockfile existed with every retry, meanwhile it had been deleted, so the lock query could never succeed in the session.

[MichaelEischer]

This could be solved rather elegantly by making the Remove operation of a backend idempotent, that is the operation should success even if repeated multiple times in a row. This would require restic to accept file does not exist as success for the Remove command. Actually this might be the only clean solution (besides a horrible hack that uses Test to check if the file existed before and no longer exists afterwards) as with a remote backend (think distributed system) an operation may succeed without us every getting the confirmation. In that case it would also be rather useful for restic to not fail. This would also make restic more resilient to temporary network problems.

[MichaelEischer]

@ProactiveServices do you remember whether the lock loading error repeated more than 10 or 20 times? I'm no longer sure how I ended up talking about the Remove command (probably too much reading about other lock issues...). If the error only showed up that many times, then these are just unnecessary retries like in #1523.

[ProactiveServices]

When trying to access a lock to which it has no permission there were 10 retries before restic aborted. When trying to access a pre-existing lock which initially existed, but was then removed, there were also 10 retries. Looks like I jumped the gun when I saw the repeated retries; for some reason I assumed it would continue retrying indefinitely.

[MichaelEischer]

Duplicate of #1523.