Running rebuild-index on s3 with new default layout corrupts index

[vsivsi]

Output of restic version

restic 0.6.0 (v0.6.0-0-g820c88ea)
compiled with go1.8.3 on darwin/amd64

How did you start restic exactly? (Include the complete command line)

Init repo with new -o s3.layout=default option

Script used for below in this gist:
https://gist.github.com/vsivsi/b45b2064c75a38fac80ac6a964b1ce0d

Full reproduction:

export RESTIC_REPOSITORY=s3:http://localhost:9000/restictest/restic
export RESTIC_PASSWORD=Password
export MINIO_ACCESS_KEY=AccessKey
export MINIO_SECRET_KEY=SecretKey
export AWS_ACCESS_KEY_ID=$MINIO_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY

minio server ./test_server &

# Endpoint:  http://192.168.3.211:9000  http://127.0.0.1:9000
# AccessKey: AccessKey 
# SecretKey: SecretKey 
# Region:    us-east-1
#
# Browser Access:
#   http://192.168.3.211:9000  http://127.0.0.1:9000
#
# Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
#    $ mc config host add myminio http://192.168.3.211:9000 AccessKey SecretKey
#
# Object API (Amazon S3 compatible):
#   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
#   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
#   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
#   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
#
# Drive Capacity: 928 GiB Free, 1.0 TiB Total

sleep 5

./restic init -o s3.layout=default

# created restic backend bfd8e53e9c at s3:http://localhost:9000/restictest/restic
# Please note that knowledge of your password is required to access
# the repository. Losing your password means that your data is
# irrecoverably lost.

# Create 10 16MB test files with long random names, containing random data:
mkdir test_data
for ((x=0;x<10;x++))
do
   FN=$(xxd -l 32 -p -c 32 /dev/urandom)
   head -c 16000000 /dev/urandom > ./test_data/$FN
done

./restic backup ./test_data

# scan [/Users/admin/restic6/test_data]
# scanned 1 directories, 10 files in 0:00
# [0:01] 100.00%  103.550 MiB/s  152.588 MiB / 152.588 MiB  11 / 11 items  0 errors  ETA 0:00 
# duration: 0:01, 89.86MiB/s
# snapshot 85f2d1a0 saved

./restic snapshots

# ID        Date                 Host        Tags        Directory
# ----------------------------------------------------------------------
# 85f2d1a0  2017-05-31 06:11:40  dev.local               /Users/admin/restic6/test_data

./restic check

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs

./restic check --read-data

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs
# Read all data
# [0:00] 0.00%  0 / 31 items
# duration: 0:00

./restic rebuild-index

# counting files in repo
# [0:00]   0 / 0 packs
# finding old index files
# saved new index as 39a40733
# remove 1 old index files

./restic check

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs
# error for tree fb9ed061:
#  id fb9ed0615766b7b057cd950a9160732978a3daedbc5483f02b5f25a22832acc2 not found in any index
# Fatal: repository contains errors

./restic check --read-data

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs
# error for tree fb9ed061:
#   id fb9ed0615766b7b057cd950a9160732978a3daedbc5483f02b5f25a22832acc2 not found in any index
# Read all data
# [0:00]   0 / 0 items
# duration: 0:00
# Fatal: repository contains errors

What backend/server/service did you use?

S3 with latest Minio version:

Version: 2017-05-05T01:14:51Z
Release-Tag: RELEASE.2017-05-05T01-14-51Z
Commit-ID: 40985cc4e3eec06b7ea82dc34c8d907fd2e7aa12

Expected behavior

rebuild-index works

Actual behavior

rebuild-index corrupts repo

Steps to reproduce the behavior

See above...

Do you have any idea what may have caused this?

Maybe the same issue I fixed in my fork with this commit:
vsivsi@654c69f

[vsivsi]

BTW, I'd try to merge the fix in the commit above into a PR to fix this but I'm literally on a ship in the middle of the Pacific Ocean (at 31N 158W to be exact) with the worlds crappiest satellite internet connection (and a million other things to do), so this time it's all I can to do just report this and hope someone else will take it from there. It took me 90 minutes to download the 0.6.0 darwin binary today to try it out, but I will happily try it again when 0.6.1 is released with a fix for this bug!
Thanks, -Vaughn

[middelink]

I'm not 100% sure this is just for S3, for laughs I ran a restic check myself and got a nasty surprise.

$ /usr/local/sbin/restic -r rest:http://qnap.ch.polyware.nl:8000 check
Create exclusive lock for repository
Load indexes
pack 1b182587 contained in several indexes: {c18786e0 e818c425}

run `restic rebuild-index' to correct this
Check all packs
pack eca9eacf586b67d0e9c0cfc8b70539bfc5eff517d0b5d1e93e2713b315d50322: not referenced in any index
pack ecaa076063c957af1b7381bb90c837a764622d0f0614038beaff2d9b9b494948: not referenced in any index
pack ecac32f93108ff4b688323c6b35fd953404b7ce9ea832d0c48c9fb05ef720df6: not referenced in any index
pack ecd6eb8e403a6756d512e190e59138e134247ad4df76c8b1c88c101ec0ca2236: not referenced in any index
pack ecdd4693007a3fa98f7b6467ea129bca81640b51a24d7f38b52d52f4fd6fecdb: not referenced in any index
pack ecdd8e1cf7f5bf27c5a006ce6d41c60be72d458dc1d63dd0e0d07e8e9de218be: not referenced in any index
pack ece206120bc8603c46103a41f960e7e810c7be4ed15fe0437b973b9767ed2bab: not referenced in any index
pack ecf1a7c0403576a6ac46e8da66a18d9620c5686e1f81ecfc8ca404f30dbef85a: not referenced in any index
pack ecf1f95aef4409dc1e4d0089a800d804afb67b777e6db01032e069342b682577: not referenced in any index
pack ecf7b976f02d5827257d50ace21ba369c2c1553a9ccfc21afab91ba75a39278d: not referenced in any index
pack ecfae977fad87bec45694026442aaf6c7679043a441acfe83911dc15b33db173: not referenced in any index
pack ecfaf0b77434bb27d5e434bc69c9160e447751ade8d6650b4d44fbc923c4aed8: not referenced in any index
pack ecfea1c91711141efca2980c242b9363889e38b62798caa2fa536eadc835b3e0: not referenced in any index
pack ed0452179342b340420a1e6332536f8458427be34d2e680798241e65e869553d: not referenced in any index
pack ed06a0cfc8063742da2f092764594fa4a228a3ed430f200c863420a0fb1d636b: not referenced in any index
pack ed1733f00a2b4de339fb1d82da220ab86b21e4cb2da4c0aecd7719a4720cb802: not referenced in any index
pack ed1aa22918c3d6e5d0418043aea11695a570585684990ea103d9a8b4e1d2e36d: not referenced in any index
pack ed1c246bb460453de8f5c87ed74f449afed5ceba3d01692644b6ffaf3831f330: not referenced in any index
pack ed1d00d249750e740d7810ee7abdeba2844647e11d39856aff5ea6238d84b955: not referenced in any index
pack ed25a4abf9850447168cba2315561518741ca3deb8838e8738e8c08d25ee5f1d: not referenced in any index
pack ed2b7519c1cdb52bbc2addab6c7ea2da395f92b06d26ad4ecaae964b816b6d41: not referenced in any index
pack ed3a57ec997af9007c5f3a9c52bddcb17b622fc2aee4cef4b88778acf2728bb6: not referenced in any index
pack ed3d5efbd2ffc074e344009eae00390929cc988b1a7d0914f3d930ed50321cdd: not referenced in any index
pack ed3db8d06f272151af2dbd11bb27f1f95b5ed9646519e851daae8e51dc78dec1: not referenced in any index
pack ed41fa097946138941bfc6496ca4af36b7f68e26579959276a1888e2882dbd4e: not referenced in any index
pack ed43c070458b5af48bc9ae066e64743d11c376ba4ec4731bac4d7b4e66945ae5: not referenced in any index
pack ed46b464403bc6890fcbea99399367bb80f2d2cfc49e7e7ca8a7d36421c973bb: not referenced in any index
pack ed4cdeefb501bc9f038ac58dcc596395a1c5084ae31cd2088e9d60aac6e9c1fc: not referenced in any index
pack ed4e898f4c00fee7ac6c97cc4afe636f672f2f005b04c89699aca617145d7725: not referenced in any index
pack ed510e257fe7f804c809d4886fa02ab3faae27ea4dbf82d380b406beceaa65b9: not referenced in any index
pack ed5c036e3adb29cadf3851a4cb49ea870fefb3c4a45bdbd1147fe6d79bc58120: not referenced in any index
pack ed5c9640f92173871c729549da177bf6a7b959b83a1119cbfbb845675e6ce69c: not referenced in any index
pack ed5f2e755ebbfac8835c94caac3fa513bb86eab9938cede0346693e66c2b642f: not referenced in any index
pack ed674f53e7c51241e07e6097cb968034bd19786252258e3edce1661adc73a502: not referenced in any index
pack ed6acd8e1baef4158ad0791271b45a03d1ea1a541a94f6ddda53ad54180564d1: not referenced in any index
pack ed6c2db6c71e8258a6f6a3b80b1993aa7cd74805321f55571421cac40b2713f8: not referenced in any index
pack ed6cff0f875a7dbe56cdd489df36739559091d0a95dcf7cf5a766b092d0d2730: not referenced in any index
pack ed6d59c3b6ca87dad3061ff5a529113720816a937fc516d93fa134153309d177: not referenced in any index
pack ed6fc9a5b4e6d0c9acb414d62413a06963c3a0a24bfcc36ca180cc4da4ef0761: not referenced in any index
pack ed7065fe1c19d8e1d8c88ddc7894b5e28d460980b3208b57d5d07aba63bf203e: not referenced in any index
pack ed71589cf3dff22b76700e97475d3822053ae0c28088bdb62a4552a3ded0d968: not referenced in any index
pack ed7796e9f46703c8e797bf6540dd953b075acbb373c19c1c31beaa3b52412679: not referenced in any index
pack ed7a34e4bdeceb0e89cc763fde75750c156e4a7a18841d55c9375350c3a942b4: not referenced in any index
pack ed7eae7382404187f3d01edf0914f7176aea07237d505dba743e3cf52938964b: not referenced in any index
pack ed846dab5ca5a4d2fa5f4f58f5959fec99273a642c8c04f79a613adf0b59984e: not referenced in any index
pack ed857768ceb0da527bf5aad91e4410655577d265224f6a17476303ae5dac96d5: not referenced in any index
pack ed85c37025dfade297e22f5517416cd2935043b14dbece93b1b3cd36495068cb: not referenced in any index
pack ed884d1673574a894ea3eb24243c6dffee6a91699a1e5e6776e484b8572feda8: not referenced in any index
pack ed94673420ccec78c909d312ef98e308222930adf715f33fa1ff98e15ca0d82a: not referenced in any index
pack ed9532d0be70531b565a0db666b9e6d9df65f5bcca61bb013eba8ca181b785dd: not referenced in any index
pack ed953330d15d3603ef23fbc33c444e82bc574b42fd2de98c313d4e0670abfec6: not referenced in any index
pack ed95cebdabaccab1e8eb7bf0680f8b39a104d143ad1ab6d1596bee0f88ff1179: not referenced in any index
pack ed980b65604eb60cde0ced4fb332464ec3b506ca4d99a27f5053044c228ba07b: not referenced in any index
pack ed9935ea1c33485cfa1bdaadb51d2f3d2d2615dbde946e8d365995d03c5ec048: not referenced in any index
pack ed9e558c649d687975ad27b16083d2e8f9a763cf8640e71a6b2da427170e6f0d: not referenced in any index
pack ed9e5f4f0622a6a48ac741b2aa97b30f5dedccedb972bf38ab2b6647e66b0def: not referenced in any index
pack eda376dab4630e0d4d43c981cc0c6a97fab7805fa6641f437fe2526beedd0118: not referenced in any index
pack eda3e41090074bd8a53693aeb665d785a1a9703686e92ce10ce0e0d4f8cbc9c2: not referenced in any index
pack eda7018cce5a8a73fc2d93f9e88a9839dd01b6bdd588f8b6a0317f2ffcccdbbe: not referenced in any index
pack eda7037d4e7059e9ef5b890e9532decf230c60f4eb07114c238c928362af469e: not referenced in any index
pack eda711e5551ed9add092d6bb2ec4d3023e22750ca9b7f39b2420f7de2d66ace0: not referenced in any index
pack edb398295e295711e26462599864eed006530d88e05d3b50cb08ad04eb35b57e: not referenced in any index
pack edb6d021ee31a756293541e814c23c1f9920ae8fdd7665a63df0b74e52147d59: not referenced in any index
pack edb860750f2dd3b50853ce4801ee967f54269ea09d6a92ce1fe3dc068de7d5e0: not referenced in any index
pack edbe8143fdd0b3b86f10e3d4305b282e50401c1cab864bfc098cd48e49253a9b: not referenced in any index
pack edbf2c5e5193a63cb1b367d81ccc21b6f3c2100cfcc06021c6e9342182030c16: not referenced in any index
pack edc51b604ac1d58c90f419b9ed9545c4a11ea67730c77da46142f7eec8461518: not referenced in any index
pack edd672371a606815cd0f9e6c1517fa5dcf5d1f054473f8ad8432c09c09976fe4: not referenced in any index
pack edd67638f6c25266de43bde9cf001c6bd64fa245137e5805f9afa33e01b20ca7: not referenced in any index
pack edd71d60779ef9d26d4ecc068b3ce3cfe4f3a5317dcae7460e15ff3830bb2094: not referenced in any index
pack edde2286202e3a4a160118ad652d2a3a13447c00db81121780469b582c71bcd1: not referenced in any index
pack ede8eda38dd9b0759b228c22e9d9b3b3d415f2feff2f4d599834f124be249860: not referenced in any index
pack ede95658852d5791257f68c692b16df86698c33f5da1ac6500ba74d65f683ca8: not referenced in any index
pack ede9e2fe1fdf1272e4049f81cd0d602a9a615d498ec29ecf750c4c58ef0aae5f: not referenced in any index
pack edeb23e4eacd4116f22bfc0921b2558440c635ac491ecc2ddfe98f410830b920: not referenced in any index
pack edf3e53163c36d8ee137d9827b2bc23e476ba7118c6f1020ff67ae62a9fcc2b1: not referenced in any index
pack edf83015fa1b0432ca694457b1529a5773d385bcd09756d02b94bef8f15f2dfe: not referenced in any index
pack edf9cb73fdda2695ec070f7b4ec298cfc9530733111be535155d0842164b32f8: not referenced in any index
pack edfa73d20809a3cb481614a50d4a84b28478bed2035dc3e17cee6eccc16b2a95: not referenced in any index
pack edfde86af2ae600847bf7b838facda6dd5bc9d2ea415a2c1740a03e2bbf41df6: not referenced in any index
pack edfe95a5e135ffa4aac4c04d2a099e6978f90a39cdb585970772a5c0672e7527: not referenced in any index
pack ee014e5a7c6d1b6de8c7c620983bd366c2bcfe996b380ac20df1162b92f1d5ad: not referenced in any index
pack ee020635a043211d0a4f310139f9c7ced4fb0f1a0059b97849eff558b3275822: not referenced in any index
pack ee033f767f4e7e6c7ecc6a9402c021bbbd7b4413e14f34e5b5faa6cff6dc8158: not referenced in any index
pack ee06cb6779ce9de7a46ae0467fa7f04371d989d178f23deb8190f5f1fcf119cd: not referenced in any index
pack ee0d679211843d6da82d7246d0d9d50d26e49ef0b9a6b050edf742a5565f7ab8: not referenced in any index
pack ee0ed6884b48a114f26590a01df7b18a5858d7dd549fc89f418af42dd66f16a9: not referenced in any index
pack ee12cb7b5eba0a9d6a5813adf1599dd739692b7672c39f12d95f57da38edb082: not referenced in any index
pack ee16930ae5705e292afc1e04808914efb84b4902b9eacfda4a241b1a0ee3ae01: not referenced in any index
pack ee177e09dc5275f193d2eb5cf6f3bdc6c7c025b32550296f0e74d034a5064df4: not referenced in any index
pack ee22678d4f410048e87b6906f5e95e69de4c86b7154ec2d5522fa602019d0171: not referenced in any index
pack ee263d5539dbc100d52d30ae225bf6d5ccb8d852984dbd670c44957c35d67704: not referenced in any index
pack ee29e354f062b265a4b8e85baadb4bb2421973f93d127283c0a32c81ab13d197: not referenced in any index
pack ee2ac5cefb17e60ab0c746006b401c3a4e47dcc927854caab118f06ef49db7b3: not referenced in any index
pack ee3144f5853aeb2a5d7e2e3bdd5a8279d92570215e894a95afc2cb4d698b44ca: not referenced in any index
pack ee388622e89a57abec903f8d77841062e0f6bde50d87fc9e9e6fe8df30c131b4: not referenced in any index
pack ee39068cd7216bd6665089decc7165983b89eeaf8f152c691dd2b482f62f448f: not referenced in any index
pack ee3b64c04d056458824ee7b973518d207338d13fd28e86081f2c1a693dc8f6e0: not referenced in any index
pack ee3eca22704db6d778fae29edd22da7dfe0b837d2aec88983922568fe9888acf: not referenced in any index
pack ee408dfb4823c8738c8020e516d0ba896c0336dba207d2d6217723c89c931adc: not referenced in any index
pack ee5e071df702dc2066c165560950ee519c8e2541135a1e0c85cab81dc4c2af76: not referenced in any index
pack ee65149637628bc3139594398b7f5d1effcf5437295c855b718cb86018b69c9b: not referenced in any index
pack ee81c2f0fd19a6e4f341c9a2b0a5f9651f05030a9a08b0a1e41869a61c8306a9: not referenced in any index
pack ee8405fe56bcff502def3f221ed11a6aba2cb2b6fda82b699b2eb0781d3b9049: not referenced in any index
pack ee8473b5f865fdfaeed331ac5ddc00e0aed79d77687dd236abd4987a1a992f61: not referenced in any index
pack ee9275c21543d515653f945f0be8f9a6c3e1597f2a190c1dd5dd3d9aac126aa4: not referenced in any index
pack f8abd1fe67afb4fb611f26c02cf6fce2d065beb84a65945ba7b190dabd403667: not referenced in any index
Check snapshots, trees and blobs
error for tree dfd61fab:
  id dfd61fab1a4ee3aa4f14dd7e0c2938a024344c8d5372beaf9909fb7e34651faf not found in any index
error for tree dbc579ff:
  id dbc579ffd37f91df2fee86385b19c7f7e46f720ab8be3badc56408d63509382d not found in any index
error for tree b3c9e379:
  id b3c9e37930e3d171ac254c7135dced644515b80a5c2ffa0b8cb254b47791eed0 not found in any index
error for tree 59f815e2:
  id 59f815e29478c04a69b3f77c24d966833f8849d6cf856da9c692bae223581d5e not found in any index
error for tree a4ac832a:
  id a4ac832a9e076a584cebf1201456ec869efeeace30de127cfa08cc2d9d0ed1c1 not found in any index
error for tree a986655c:
  id a986655cceac838573d85de3f7e85ae8e686730e4e4d95929f086b72a48e0a35 not found in any index
error for tree a4a5e036:
  id a4a5e0362bb601fcf4a5cfe600b76ca86d1e31c4d649256e6d304b0ce928d1bf not found in any index
error for tree 98ac2e26:
  id 98ac2e264222c3377e893f5f539cfd316bc1b05d55b095347753455b64892380 not found in any index
error for tree 129f69ae:
  id 129f69ae7dc18a161e42a04e85427bb2c77b2d92e0b069f5c7588ff2067a5296 not found in any index
error for tree 34e1c4a1:
  id 34e1c4a11ff0797d37581d7f8c76fada90bcb07d41ca3e8a6d49888cdd3a3208 not found in any index
error for tree 77963d25:
  id 77963d2510ab3482a2aa4d03cd0e6573501d705482664d3ec566a1a3772a2905 not found in any index
error for tree 71173899:
  id 71173899bea06aee0330554528995f593f9d31990273e431078d0a0a733a8b49 not found in any index
error for tree 7398f0ee:
  id 7398f0eec764cd26699d690d994aaa695602996edfae15a2c0963af91c2d454d not found in any index
error for tree 7abe7d96:
  id 7abe7d966d68d556aab73087f53207213de704a8cca05ee8a70da45647095785 not found in any index
error for tree 3bfa1ee9:
  id 3bfa1ee9ccda57ff226ec943e0416a25b8441ad85a814b558f29deb040dd42f7 not found in any index
error for tree a3eed9aa:
  id a3eed9aae6e8054d05bbc970181a4384e13ab203636825b2585968b62c8fe484 not found in any index
error for tree 543775b1:
  id 543775b1fe8ebb80d513a50b86ec3472102b63a8f64ed9a471ce54ae07042063 not found in any index
error for tree e96e33d4:
  id e96e33d4cbfd8f79322a2b1d2c3cf69a52c6905d2293fa2245cb5f571a93fe4c not found in any index
error for tree d484ca3e:
  id d484ca3e5f4f0372dd1ebf589606306ab70ae7ee428cb521e3d5274921736259 not found in any index
error for tree da511ada:
  id da511ada7fd398eb6ef0ab272ee559c16d73acfa27454417e63fec23f1889719 not found in any index
error for tree fef3e679:
  id fef3e67999a37fdf0ee1394b66f6ed796904f21a0a97b30975bfaaad3d4bdf7c not found in any index
error for tree 7ae2f675:
  id 7ae2f675570524d7a5089a312940370b73562f70689dcb7e6a53a6e40720024c not found in any index
error for tree 80d0b856:
  id 80d0b856955027dcf2e234f3325070fc0f24157d59dad59fa58094ff64619e4b not found in any index
error for tree 35a2c7ab:
  id 35a2c7ab442270240bfa33a7797c00b407ff9c00207347e79f432e7506dab209 not found in any index
error for tree ccc3e20e:
  id ccc3e20e1d881dcee2ae861714766207356f15aea0d255491f578f117a81971e not found in any index
error for tree 62c4992b:
  id 62c4992bb1443a22d6499bd4cc3372f2250c22818d27cacff533b2f4f7af1c02 not found in any index
error for tree fffe4ac8:
  id fffe4ac841980afe351bd52fba57ce758f123f86fc3fbc9da130feaed2c9d1c3 not found in any index
error for tree 66811069:
  id 6681106927277462b217e504cb889e6b51dcf0c90ff400cfca3dfe446f222deb not found in any index
error for tree 68038c12:
  id 68038c12e15e1c7abcf4c6f2412e40e1bce01da08d4d51edd1fd1f6c15e56df0 not found in any index
error for tree 4b0c7bd8:
  id 4b0c7bd8953912bb0e2c3003a215fb7b469031926ebe3e64d01014f52367d83d not found in any index
error for tree 179a71c2:
  id 179a71c2695b3562c947fa6dedb02b681506bad777f033734d951f2ebaf257b8 not found in any index
error for tree 8f854700:
  id 8f854700fb9f90ea37eb36eadad846c2b9a8815712a5c326ed3c54e3af782b05 not found in any index
error for tree db3c3f4c:
  id db3c3f4c410398648790efa537c940c073f40fa263e050b765f17670ef34e00f not found in any index
error for tree da3a7c5d:
  id da3a7c5d474f49d36f542032f6f3e1c568938f5d17397790a0a19d37930a5b7b not found in any index
error for tree ae75b074:
  id ae75b074a3a9a5fdc17f067670244f8474db120277dfbab5052fe6aa80f4ddb5 not found in any index
error for tree 1ff37782:
  id 1ff37782f92ae4254d37947f2a87426c9ecd5d3ab113bb26861e2cb32c9b3e6b not found in any index
error for tree 32a88971:
  id 32a889716b6c30ce28b1a889ade713a305f891e376a03f04e37757303245edd7 not found in any index
error for tree d2cf2790:
  id d2cf27901bbd20e80a2260ee8c88f8a24049419b3461384353729011364df16f not found in any index
error for tree 93f4538b:
  tree 93f4538b, blob 018dcdf7: not found in index
error for tree 6d056908:
  tree 6d056908, blob fb69df7d: not found in index
error for tree 946c0f3f:
  tree 946c0f3f, blob ad68d5c0: not found in index
error for tree 02ba04ae:
  tree 02ba04ae, blob ad68d5c0: not found in index
error for tree 662fc57e:
  tree 662fc57e, blob fb69df7d: not found in index
error for tree c601567a:
  tree c601567a, blob 7e784209: not found in index
  tree c601567a, blob d583bb34: not found in index
  tree c601567a, blob 252086d0: not found in index
error for tree b10815a7:
  tree b10815a7, blob f8185565: not found in index
  tree b10815a7, blob afd007cf: not found in index
  tree b10815a7, blob c07eef18: not found in index
  tree b10815a7, blob dc55f2d8: not found in index
  tree b10815a7, blob 08b5a91c: not found in index
  tree b10815a7, blob 491f8642: not found in index
  tree b10815a7, blob e8d5cf51: not found in index
  tree b10815a7, blob be612d39: not found in index
error for tree 6bafa778:
  id 6bafa7784097de4de149e6afc388ce74ada8415bc112737b7af46c30827f4a0e not found in any index
error for tree 33083318:
  tree 33083318, blob f8185565: not found in index
  tree 33083318, blob 610189d8: not found in index
  tree 33083318, blob 1de2b25f: not found in index
  tree 33083318, blob afd007cf: not found in index
  tree 33083318, blob 287dbc96: not found in index
  tree 33083318, blob c07eef18: not found in index
  tree 33083318, blob 5eecc50e: not found in index
  tree 33083318, blob dc55f2d8: not found in index
  tree 33083318, blob 08b5a91c: not found in index
  tree 33083318, blob 491f8642: not found in index
  tree 33083318, blob e8d5cf51: not found in index
  tree 33083318, blob be612d39: not found in index
error for tree 695311e1:
  id 695311e15a8c44f39379797a5e5517c036d48da2ca8b0cf1b725450fa839acba not found in any index
error for tree d998c45f:
  id d998c45f64e86d88a3344fafebea5e11ed2f72100d7633e2e14922361e0ae785 not found in any index
error for tree daec9cf6:
  tree daec9cf6, blob fb69df7d: not found in index
error for tree de5eccc9:
  tree de5eccc9, blob ad68d5c0: not found in index
error for tree 0aececca:
  tree 0aececca, blob ad68d5c0: not found in index
error for tree 06407a04:
  tree 06407a04, blob fb69df7d: not found in index
error for tree 2a4bc5da:
  tree 2a4bc5da, blob 018dcdf7: not found in index
error for tree e8133e82:
  tree e8133e82, blob b8e75fca: not found in index
error for tree 233b5ee9:
  id 233b5ee90a04ccdfb9c1ae109db8a59c09a3a92c2fa860541291a446b9107687 not found in any index
error for tree 8bae0453:
  id 8bae0453178200d48e2ecb03cc48079a9093996207ceafc947581a8caa46065f not found in any index
error for tree 2fc05701:
  id 2fc0570126da6aad1f98ba1bbf3d80441f1972a9924ff0c7cb4bb0663c367382 not found in any index
error for tree 9ac17b23:
  tree 9ac17b23, blob ad68d5c0: not found in index
error for tree d26fbce2:
  tree d26fbce2, blob fb69df7d: not found in index
error for tree 739e2c17:
  tree 739e2c17, blob 7e784209: not found in index
  tree 739e2c17, blob d583bb34: not found in index
  tree 739e2c17, blob 252086d0: not found in index
error for tree 995b5741:
  tree 995b5741, blob 7e784209: not found in index
  tree 995b5741, blob d583bb34: not found in index
  tree 995b5741, blob 252086d0: not found in index
error for tree 84453170:
  tree 84453170, blob fb69df7d: not found in index
error for tree c2c0a3f5:
  tree c2c0a3f5, blob ad68d5c0: not found in index
error for tree fabca633:
  tree fabca633, blob ad68d5c0: not found in index
error for tree 07260e1b:
  tree 07260e1b, blob fb69df7d: not found in index
error for tree be5ca81e:
  tree be5ca81e, blob fb69df7d: not found in index
error for tree fb241cd5:
  tree fb241cd5, blob ad68d5c0: not found in index
error for tree 9a283d5b:
  tree 9a283d5b, blob 7e784209: not found in index
  tree 9a283d5b, blob d583bb34: not found in index
  tree 9a283d5b, blob 252086d0: not found in index
error for tree e312027f:
  tree e312027f, blob ad68d5c0: not found in index
error for tree 31cd4651:
  tree 31cd4651, blob fb69df7d: not found in index
error for tree b51e6a16:
  tree b51e6a16, blob ad68d5c0: not found in index
error for tree 80f519ac:
  tree 80f519ac, blob fb69df7d: not found in index
error for tree d8c6349b:
  tree d8c6349b, blob ad68d5c0: not found in index
error for tree 8d73974a:
  tree 8d73974a, blob fb69df7d: not found in index
error for tree 9e3521be:
  id 9e3521be4665a1ace03e4ba0d51e5f11f24589d42af866d7e0089d171e650c8b not found in any index
error for tree 92668021:
  id 926680211b708232e22455a8db55d927d46b610cb55bb94e5e32a60db340a22d not found in any index
error for tree 3317042f:
  id 3317042fce19fa93350a1759c3cbe351983c6e237751f95a84cba325f344de5c not found in any index
error for tree d5e93206:
  tree d5e93206, blob fb69df7d: not found in index
error for tree 363f2cd8:
  tree 363f2cd8, blob ad68d5c0: not found in index
error for tree bfc7d003:
  id bfc7d00315cf655739865c4acc5e7228816ffb0fd4532a94680254ef86a8a0e6 not found in any index
error for tree 50aa00dc:
  id 50aa00dc98c1ad16b4460142eac6f7c5955ad4a04e336158d60ec950dce241f0 not found in any index
error for tree 701d766d:
  id 701d766d3ffd98918e8a04ab2e5b204508d43d18cc005c11a7076b128bcaf1e9 not found in any index
error for tree ea106896:
  tree ea106896, blob fb69df7d: not found in index
error for tree 87fee5da:
  tree 87fee5da, blob ad68d5c0: not found in index
error for tree 66333a30:
  tree 66333a30, blob 018dcdf7: not found in index
error for tree 8b5b57b2:
  tree 8b5b57b2, blob b8e75fca: not found in index
error for tree db4120b8:
  tree db4120b8, blob 018dcdf7: not found in index
error for tree 898b866d:
  tree 898b866d, blob ad68d5c0: not found in index
error for tree efa32682:
  tree efa32682, blob fb69df7d: not found in index
error for tree 13912cfb:
  id 13912cfbb3ffccc8b87adedb547a9a266c9af794e758dbbe0a7fee6871c4b27c not found in any index
error for tree d7bfdfaa:
  id d7bfdfaa62f2fdd4d817f9fbf09659287b27cb500e23b37714e3bac00efcf8c6 not found in any index
error for tree 1ed757df:
  id 1ed757dfbcb876766da81670b01a8f6b72e3f9e33aec6daa71896d9b5d885946 not found in any index
error for tree cd95e9a1:
  id cd95e9a10369bf33eeb8ec0fc127027ce7eec6df2198a22e3e646b5df4a2269d not found in any index
error for tree 3092b2f7:
  id 3092b2f734c3c603e26cfb5853a350b910abc9f074e40b2dfecac708f7d1feda not found in any index
error for tree 543cee9b:
  id 543cee9b309650e6785bcfeafff7a09fd61459339d9e5ad55692c373ce1e27d1 not found in any index
error for tree ab49f956:
  id ab49f956c098bd08b1be70bda780b7bd6cb3dcac259a1df8d71ac8480c049c00 not found in any index
error for tree 42a7541e:
  id 42a7541e8d10e5b04374055d8bad41a2ea876286bb3fa92500e6468ccd6c39bb not found in any index
error for tree 14f7bf4b:
  id 14f7bf4b793d2a86c022572efd0ee0de053a86cb51896092a9af6667ba71bac3 not found in any index
error for tree 37002681:
  id 3700268113100594b893cc949c197ad43c5e3ed5e97e674c8b273057a6748779 not found in any index
error for tree 49da46b7:
  id 49da46b7a90851bd82eced32d9d6f56784f2c3f16eb5fc4593ed1d6e49cfe94c not found in any index
error for tree 0ae6192f:
  id 0ae6192fccb2b9f17297aea7a2c6aebcf0afb9942dee4f7284bfa3d7cb3790fc not found in any index
error for tree 359ca8f2:
  id 359ca8f2dd0db784abddd1d3cadc50d640099590e83ee917740fbd6ce214a9f7 not found in any index
error for tree cd4089a4:
  id cd4089a4afc1000f862745f32b0223683fb41be876634a99ca00e1e6b82d3275 not found in any index
error for tree 949bbe08:
  id 949bbe08b6a2e016bbd7ecceec6f0add572e49cbc4a20341a07b5f08c3a37a35 not found in any index
error for tree fe9c7184:
  id fe9c7184966f0704684a42c450b98d2d8d7b569f19223c34258db4dc7eef4438 not found in any index
error for tree 7debca39:
  id 7debca39e1a26977114e19faf3e8639b526b2f655150fa1cfa3f92d77b80ac51 not found in any index
error for tree b70d5ac9:
  id b70d5ac9f1a02186848f48581f7509dc3e29a781aaa991af93a80ffe878dccf4 not found in any index
error for tree 50921d00:
  id 50921d0031aab9fa163d5186a674b04c14cf87ae58c6a7aeae0c8807e7bf0009 not found in any index
error for tree 3b8ae290:
  id 3b8ae290aacfa4b2e42ef92da0d53e79384da4709553b0f75fff331c10918ca7 not found in any index
error for tree de3ead62:
  id de3ead62973169185ddc82ce6db8c656c0fef0bd46bcfd8df5961c5237ffa4d8 not found in any index
error for tree e2782924:
  id e2782924001a8473d1f5a1ee74a72afa2361e80af1a7da83b054d308b9f77260 not found in any index
error for tree a87c2626:
  id a87c2626b4318bafec7440c8610721e0ceab71c5bb23b68033af9ba79b31816f not found in any index
error for tree efd019f5:
  id efd019f5156038f42e605e87478e7f1c7002bca2ad2fad7bb8094893cbbd5c1a not found in any index
error for tree 410aa445:
  id 410aa44594ff97981075135c64e80de17c8d4cdceefc771cf5264e867897e4ea not found in any index
error for tree afc71ebe:
  id afc71ebe4eb4d5508b188dc8a29a15a919e7355e1a471971c28e63b2d36c67d1 not found in any index
error for tree f670f866:
  id f670f86689407148cf9a4ec1003a485f24933a3657d7d21632ea02d43097dced not found in any index
error for tree 07041343:
  id 07041343e91bc7858aa50481bc8596052513726b7df5318fb2bb54ee140e1236 not found in any index
error for tree 7c9d15db:
  tree 7c9d15db, blob cee337c2: not found in index
error for tree 81d1f923:
  tree 81d1f923, blob 7f210da4: not found in index
error for tree b5df24f8:
  tree b5df24f8, blob 3f41e3d4: not found in index
  tree b5df24f8, blob 4c80aac8: not found in index
error for tree ea012a94:
  tree ea012a94, blob 7e784209: not found in index
  tree ea012a94, blob d583bb34: not found in index
  tree ea012a94, blob 252086d0: not found in index
error for tree 935b04d8:
  id 935b04d8b174280857d7d0b35f463d1d4020e05c1ab6285c0e5db6a0a538a604 not found in any index
error for tree 80db2ce8:
  id 80db2ce8238df5e10aeb4352fc1bcabb04493c85ef6c3d44e5e4f6ba6359499a not found in any index
error for tree d1b485ed:
  id d1b485ed6b3497b10969a961d17a573b2b222fdef50f57735cbb4738cf1a7afb not found in any index
error for tree 822d37ee:
  id 822d37ee6cd4dd3702b12dbc247c3547eec9daa60e86d8532f6c067daf3aab71 not found in any index
error for tree f6c55535:
  tree f6c55535, blob ad68d5c0: not found in index
error for tree f3e5a2e9:
  tree f3e5a2e9, blob ad68d5c0: not found in index
error for tree 522f9adb:
  tree 522f9adb, blob fb69df7d: not found in index
error for tree fc349fe1:
  id fc349fe1fcbada58a03ca23d4630072e57b3b4146455d3854d3b970b7669ca30 not found in any index
error for tree 0fbc7814:
  id 0fbc78144e88c90cfc2ef445fcd65aa4569e5e630e6ae9d571056c8f281e153c not found in any index
error for tree e16ab722:
  id e16ab72283e17b51ab77323d526c367094d05e3a749a647da5ce953ff78df79b not found in any index
error for tree c7069cbf:
  id c7069cbfe849dd006f8fe29ba45b04022d5927e38321bda00362d71757ac49a4 not found in any index
error for tree 313e95a1:
  id 313e95a1e6e2a56f47b6d8e5216cf61c792c4d77c66376930f163da2c638e08b not found in any index
error for tree 49dc4ba4:
  id 49dc4ba4e1a1b9010495051bd7c3a3b961f0c0706e7aa7421d9ddf0563c0bb08 not found in any index
error for tree 31b8eb93:
  id 31b8eb93dbcae85e1fd4b738236a0f964409b378cff8453001d2f3ba14d66afc not found in any index
error for tree 9a8a9618:
  tree 9a8a9618, blob ad68d5c0: not found in index
error for tree 990c7954:
  tree 990c7954, blob ad68d5c0: not found in index
error for tree f5e0295d:
  id f5e0295d3f838e364749563fe5f10b6f7c2c8f90f1588cc703381c3c5b5eeff1 not found in any index
error for tree 2dd8a023:
  tree 2dd8a023, blob ad68d5c0: not found in index
error for tree 2b548096:
  tree 2b548096, blob fb69df7d: not found in index
error for tree bb576ee1:
  tree bb576ee1, blob f8185565: not found in index
  tree bb576ee1, blob afd007cf: not found in index
  tree bb576ee1, blob c07eef18: not found in index
  tree bb576ee1, blob dc55f2d8: not found in index
  tree bb576ee1, blob 08b5a91c: not found in index
  tree bb576ee1, blob 491f8642: not found in index
  tree bb576ee1, blob e8d5cf51: not found in index
  tree bb576ee1, blob be612d39: not found in index
error for tree 83cbdefc:
  tree 83cbdefc, blob 7e784209: not found in index
  tree 83cbdefc, blob d583bb34: not found in index
  tree 83cbdefc, blob 252086d0: not found in index
error for tree 953f43e9:
  tree 953f43e9, blob ad68d5c0: not found in index
error for tree ad0ed77d:
  tree ad0ed77d, blob fb69df7d: not found in index
error for tree 038901e7:
  tree 038901e7, blob ad68d5c0: not found in index
error for tree 772258d1:
  tree 772258d1, blob fb69df7d: not found in index
error for tree 80c6e69e:
  tree 80c6e69e, blob ad68d5c0: not found in index
error for tree a5096423:
  tree a5096423, blob ad68d5c0: not found in index
error for tree 8a6436d7:
  tree 8a6436d7, blob fb69df7d: not found in index
<stopped looking...>

I'm going to run restic rebuild-indexes to see if it fixes things up. (As one of my VMs OOMs every night due to restic ballooning, it is likely I have half baked packs in the repo, but all the missing tree's are unexpected. ... Are we sure restic check is not lying to us?

[fd0]

Oh dear, thanks for reporting! Good thing we haven't enabled the default layout by default yet... I'll have a look ASAP. It looks like the checker is using the wrong layout, so it does not find any indexes or data files.

[vsivsi]

@fd0 Yes, that's exactly what I ended up fixing in the commit I referenced at the end of the issue above.

[fd0]

btw, you can force the layout with -o s3.layout=default for the rebuild-index operation.

[middelink]

BTW, In my case I was not using S3, so it could not have been the layout issue. My problem was fixed after running restic rebuild-index though, so I can only presume restic check has some problem with old, superseded indexes.

[fd0]

Yes, I think your issue is different. Maybe that the there are unfinished pack files (I know that we need to work on that).

[fd0]

Ok, rebuild-index for s3 with the default layout creates an empty index

[fd0]

Listing pack files on s3 with the default layout is broken, patch is here:https://gist.github.com/fd0/15039537035ce2d6f8f203bd247c1724

I'll commit this fix later today.

[fd0]

Fixed in 5c59484.

[vsivsi]

@fd0

Is this change in the earlier patch not necessary for the fix? Just noticed that it didn't make it into the commit you reference above...

https://gist.github.com/fd0/15039537035ce2d6f8f203bd247c1724#file-s3defaultfix-patch-L9-L10

[fd0]

The first hunk isn't needed (it is even wrong), the second hunk is the one that's important.

[fd0]

I'll release 0.6.1 shortly. Together with #986 this warrants a new release (btw, the s3 backend used with Minio should also be also affected by #985).