First time Replication Destinations fail in v0.8.1, succeed in v0.8.0

[no1here0003]

Describe the bug
Creating a brand new replicationdestination in VolSync v0.8.1 fails with:

VolSync restic container version: unknown
restore
restic 0.16.4 compiled with go1.21.6 on linux/amd64
Testing mandatory env variables
=== Starting restore ===
Fatal: unable to open config file: Stat: The specified key does not exist.
Is there a repository at the following location?

v0.8.0 works.

Steps to reproduce
Create a new replication destination in a bucket that it has never been in before as a first time deployment in version v0.8.1, the jobs error and the pvc associated with it is stuck in Pending.

Do the same in v.0.8.0 and it works.

Expected behavior
It should behave as v0.8.0 did

Actual results
PVC is stuck in pending and VolSync jobs error with the above error message.

Additional context
This was a first time install of VolSync, I went straight to v0.8.1 and this happened. Downgrading to v0.8.0 worked.

[no1here0003]

In case it helps, the replication endpoint is the latest release of minio.

[adampetrovic]

See #1163 as well

[no1here0003]

In this case I can definitely reproduce it on demand. If there's any additional information I can gather please let me know 👍

[tesshuflower]

There was a fix in v0.8.1 (#1005) for a bug where if the auth or connection to the data store hosting the restic backup failed, the replicationdestination would succeed and not throw any errors.

If you are trying to restore a backup from a bucket that cannot authenticate or does not exist, then you will get an error. Generally this is desired as we want to know if a restore from a backup does not succeed.

[no1here0003]

Would it be possible to allow an override in the install to not error? I have a GitOps set up, so having the destination on all pvc definitions makes life a lot easier, including new ones.

I would imagine that's what others are doing as well.

[no1here0003]

Wait. Is it if a bucket doesn't exist or a path? In my case a bucket did and a path did not.

[onedr0p]

If you are trying to restore a backup from a bucket that cannot authenticate or does not exist, then you will get an error. Generally this is desired as we want to know if a restore from a backup does not succeed.

I am using one bucket with sub-folders per replicationsource, would this affect me as well? I'll try to find some time to test it but if this error happens in that case it makes the process a bit cumbersome since I would have to commit the PVC and replicationsource in one commit and then wait for the buckets sub-folder/restic repo to be created and then commit the replicationdestination in another commit. Not really ideal IMO and with v0.8.0 I can commit them all in one shot.

[tesshuflower]

@onedr0p and @no1here0003 Yes you're correct, it wouldn't just be the bucket, but the path itself. Essentially it means that path cannot be found and/or is not initialized as a restic repo.

Potentially we could look at trying to initialize the path if doing restic operations such as listing snapshots on it fails (initing the path would then fail if the path has already been initialized so I think that would be pretty safe) - however the issue with ignoring errors when the path doesn't exist is it means a restore from a backup may look like it succeeds even if your data was blown away.

Generally I would prefer to prioritize an actual restore from a backup and know the status of this restore than for an initial setup type situation where the bucket or path does not exist.

I think I understand your scenario better now however, so will discuss and get back to you.

[tesshuflower]

An update: I've discussed with @JohnStrunk and I think we're going to put in a fix to try and make a restore from empty (or no path) not throw an error, which should then restore the behaviour you're used to, but still fail the job if the credentials are incorrect or cannot connect.

The fix would be this on restore:

• Do the same check we currently do for listing snapshots in the restic path (if this fails we don't know if it's because of bad creds or path not existing/not inited)
• if it fails, try to init the path
• if this also fails, we fail the job (credentials are bad or not working, or path could not be accessed or inited)
• Now we continue to list snapshots, there will be none, so no data will be restored (will try to make sure we log something in the log in this case - for debugging purposes)

I think this should solve most issues and will allow you to restore from empty/non existing repos. The side effect will be that it will init the path ahead of your replicationsource doing it (I don't believe this should cause any issue).

I'm going to try to get this fix into the v0.9.1 release. Just FYI we have a v0.9.0 release that may come out as soon as tomorrow - but this will not contain the fix yet.

[onedr0p]

Thanks for the update @tesshuflower that seems like a great solution indeed.

[no1here0003]

Awesome, thank you very much!