Missing directories in data/ will cause backup to fail

[dvicory]

Output of restic version

debug enabled
restic 0.8.1 (v0.8.1-0-g546d6f36)
compiled with go1.9.2 on darwin/amd64

How did you run restic exactly?

Configure for a local directory backend. Take an initial backup:

restic -r . backup path/to/files

Delete all empty folders in data/:

for i in {0..255}; do rmdir data/$(printf "%2.2x\n" $i); done

Add some files in path/to/files and then take another backup:

restic -r . backup path/to/files

You will get errors like the following and backing up won't work:

debug enabled
enter password for repository:
password is correct
using parent snapshot 01f77097
scan [/path/to/files]
scanned 56 directories, 192 files in 0:00
Save(<data/6d315bd048>) returned error, retrying after 395.927319ms: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 738.53627ms: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 1.490479005s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 2.506716072s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 3.27260705s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 4.004076866s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 3.201206403s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 5.203822631s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 8.396998087s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
Save(<data/6d315bd048>) returned error, retrying after 26.32182325s: OpenFile: open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
[0:55] 100.00%  595.599 KiB/s  31.990 MiB / 31.990 MiB  248 / 248 items  0 errors  ETA 0:00
duration: 0:55, 0.58MiB/s
open data/6d/6d315bd0481c9910b472720d790ee0e38d1166dd311affbdc1bdd400b60d181c: no such file or directory
OpenFile
github.com/restic/restic/internal/backend/local.(*Local).Save
	src/github.com/restic/restic/internal/backend/local/local.go:142
github.com/restic/restic/internal/backend.(*RetryBackend).Save.func1
	src/github.com/restic/restic/internal/backend/backend_retry.go:71
github.com/restic/restic/vendor/github.com/cenkalti/backoff.RetryNotify
	src/github.com/restic/restic/vendor/github.com/cenkalti/backoff/retry.go:37
github.com/restic/restic/internal/backend.(*RetryBackend).retry
	src/github.com/restic/restic/internal/backend/backend_retry.go:37
github.com/restic/restic/internal/backend.(*RetryBackend).Save
	src/github.com/restic/restic/internal/backend/backend_retry.go:65
github.com/restic/restic/internal/cache.(*Backend).Save
	src/github.com/restic/restic/internal/cache/backend.go:55
github.com/restic/restic/internal/repository.(*Repository).savePacker
	src/github.com/restic/restic/internal/repository/packer_manager.go:107
github.com/restic/restic/internal/repository.(*Repository).Flush
	src/github.com/restic/restic/internal/repository/repository.go:306
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot
	src/github.com/restic/restic/internal/archiver/archiver.go:767
main.runBackup
	src/github.com/restic/restic/cmd/restic/cmd_backup.go:478
main.glob..func2
	src/github.com/restic/restic/cmd/restic/cmd_backup.go:49
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).execute
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:698
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:783
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).Execute
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:736
main.main
	src/github.com/restic/restic/cmd/restic/main.go:69
runtime.main
	/usr/local/Cellar/go/1.9.2/libexec/src/runtime/proc.go:195
runtime.goexit
	/usr/local/Cellar/go/1.9.2/libexec/src/runtime/asm_amd64.s:2337

What backend/server/service did you use to store the repository?

Local directory backend.

Expected behavior

Check that the necessary folders exist and create them if necessary.

Actual behavior

Fails and will not continue the backup.

Steps to reproduce the behavior

See above.

Do you have any idea what may have caused this?

Missing directories in data/.

Do you have an idea how to solve the issue?

Did restic help you or made you happy in any way?

Appreciate the help on IRC! 😄

[fd0]

Hm, I'm unable to reproduce the issue.

[fd0]

Oh, gotcha. The bug triggers only when the local repository is specified as a relative path.