periodically check for stale restic repo locks

[skriss]

Signed-off-by: Steve Kriss krisss@vmware.com

Fixes #1511

I think this is all we need to do, but I'm still working on testing it.

[carlisia]

This seems fine.

[skriss]

I've confirmed that when there are no stale locks, this is a no-op, and when there is a stale lock, this successfully clears it out.

I also tweaked the ensureRepo function in the restic repo controller to only try to initialize a repo if the error gotten from attempting to connect to the repo indicates that the repo doesn't exist. We were seeing issues where, if the repo had a stale lock, Velero would try to re-initialize it rather than reporting that it was locked, which just led to confusing error output for the user.

[skriss]

One other note - restic considers locks "stale" if they've been around for >30min, even though in the normal case, they get refreshed every 5min. We might want to consider a PR to restic to make this "stale threshold" a CLI flag, so that we could say restic unlock --older-than 10m, for example.

[skriss]

Hold on merging this, checking one thing.

[skriss]

OK, looks fine, I just had some versioning issues.

[prydonius]

lgtm