archivist does not report about missing files

[vogel]

Here is example with ledger-00378d7f.xdr.gz missing on core_live_003:

wget http://history.stellar.org/prd/core-live/core_live_003/ledger/00/37/8d/ledger-00378d7f.xdr.gz
--2017-08-25 12:25:43--  http://history.stellar.org/prd/core-live/core_live_003/ledger/00/37/8d/ledger-00378d7f.xdr.gz
Resolving history.stellar.org (history.stellar.org)... 52.218.16.98
Connecting to history.stellar.org (history.stellar.org)|52.218.16.98|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-08-25 12:25:44 ERROR 404: Not Found.

stellar-archivist does not catch this error

./stellar-archivist --low 3600000 --high 3700000 scan --verify --thorough http://history.stellar.org/prd/core-live/core_live_003/
2017/08/25 12:23:23 Scanning checkpoint files in range: [0x0036ee3f, 0x0038753f]
2017/08/25 12:23:35 Archive: 1564 history, 1564 ledger, 968 transactions, 0 results, 0 scp
2017/08/25 12:23:55 Checkpoint files scanned with 0 errors
2017/08/25 12:23:55 Archive: 1564 history, 1564 ledger, 1564 transactions, 1564 results, 1564 scp
2017/08/25 12:24:01 Archive: 312 buckets total, 312 referenced
2017/08/25 12:24:01 Examining checkpoint files for gaps
2017/08/25 12:24:01 Examining buckets referenced by checkpoints
2017/08/25 12:24:01 No checkpoint files missing in range [0x0036ee3f, 0x0038753f]
2017/08/25 12:24:01 No missing buckets referenced in range [0x0036ee3f, 0x0038753f]
2017/08/25 12:24:01 Error: mismatched hash on ledger header 0x0037983f: expected 214bba8213a2e58e3e40ba1ff57882306b97c305404c5f5d2506917be56e48cf, got 0000000000000000000000000000000000000000000000000000000000000000
2017/08/25 12:24:01 Error: 1 ledger headers (of 63168 checked) have unexpected hashes
2017/08/25 12:24:01 Verified 63168 transaction sets have expected hashes
2017/08/25 12:24:01 Verified 63168 transaction result sets have expected hashes
2017/08/25 12:24:01 Verified 312 buckets have expected hashes
2017/08/25 12:24:01 Detected 1 objects with unexpected hashes

[robertDurst]

Hey! Having issues replicating this. Here are the steps I took:

1. Mirrored a history archive to my local disk

stellar-archivist mirror http://s3-eu-west-1.amazonaws.com/history.stellar.org/prd/core-testnet/core_testnet_001 file://local-archive --low 3600000 --high 3700000

2. Run stellar-archivist

stellar-archivist --low 3600000 --high 3700000 scan --verify --thorough file:///Users/[MY_USERNAME]/local-archive/

I received this output:

2018/03/02 14:41:08 Scanning checkpoint files in range: [0x0036ee3f, 0x0038753f]
2018/03/02 14:41:09 Archive: 1344 history, 233 ledger, 762 transactions, 824 results, 928 scp
2018/03/02 14:41:10 Checkpoint files scanned with 0 errors
2018/03/02 14:41:10 Archive: 1564 history, 1564 ledger, 1564 transactions, 1564 results, 1564 scp
2018/03/02 14:41:10 Scanning all buckets, and those referenced by range
2018/03/02 14:41:11 Archive: 2037 buckets total, 2037 referenced
2018/03/02 14:41:11 Examining checkpoint files for gaps
2018/03/02 14:41:11 Examining buckets referenced by checkpoints
2018/03/02 14:41:11 No checkpoint files missing in range [0x0036ee3f, 0x0038753f]
2018/03/02 14:41:11 No missing buckets referenced in range [0x0036ee3f, 0x0038753f]
2018/03/02 14:41:11 Verified 100096 ledger headers have expected hashes
2018/03/02 14:41:11 Verified 100096 transaction sets have expected hashes
2018/03/02 14:41:11 Verified 100096 transaction result sets have expected hashes
2018/03/02 14:41:11 Verified 2037 buckets have expected hashes

As you can see this worked as expected.

3. Removed ledger-00378d7f.xdr.gz

4. Run stellar-archivist again

stellar-archivist --low 3600000 --high 3700000 scan --verify --thorough file:///Users/[MY_USERNAME]/local-archive/

5. Received the following:

2018/03/02 15:00:16 Scanning checkpoint files in range: [0x0036ee3f, 0x0038753f]
2018/03/02 15:00:17 Archive: 1376 history, 229 ledger, 772 transactions, 809 results, 909 scp
2018/03/02 15:00:18 Checkpoint files scanned with 0 errors
2018/03/02 15:00:18 Archive: 1564 history, 1563 ledger, 1564 transactions, 1564 results, 1564 scp
2018/03/02 15:00:18 Scanning all buckets, and those referenced by range
2018/03/02 15:00:19 Archive: 2037 buckets total, 2037 referenced
2018/03/02 15:00:19 Examining checkpoint files for gaps
2018/03/02 15:00:19 Examining buckets referenced by checkpoints
2018/03/02 15:00:19 Missing ledger (1): 0x00378d7f
2018/03/02 15:00:19 No missing buckets referenced in range [0x0036ee3f, 0x0038753f]
2018/03/02 15:00:19 Error: mismatched hash on ledger header 0x00378d7f: expected e9b8e5356394f4ca192fcccddd5b93dd9bfb98428548e0d7eb0050e820a41ca5, got 0000000000000000000000000000000000000000000000000000000000000000
2018/03/02 15:00:19 Error: 1 ledger headers (of 100032 checked) have unexpected hashes
2018/03/02 15:00:19 Verified 100032 transaction sets have expected hashes
2018/03/02 15:00:19 Verified 100032 transaction result sets have expected hashes
2018/03/02 15:00:19 Verified 2037 buckets have expected hashes
2018/03/02 15:00:19 Detected 1 objects with unexpected hashes

As you can see, the error was caught.

2018/03/02 15:00:19 Missing ledger (1): 0x00378d7f

Is it possible this has been fixed?

[cecilygardner]

Hi - @robertDurst was unable to reproduce this issue on the livenet or the testnet environment. Given that, we're going to close the issue. Please do let us know if you're still encountering the problem.