rp_storage_tool fails to decode manifest in ShadowIndexingManyPartitionsTest.test_many_partitions_shutdown

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/30740#01889338-b7f1-46f5-9406-183a83b1c38e

Module: rptest.tests.e2e_shadow_indexing_test
Class:  ShadowIndexingManyPartitionsTest
Method: test_many_partitions_shutdown

[INFO  - 2023-06-07 00:33:08,859 - redpanda - _cloud_storage_diagnostics - lineno:2470]: Fetching manifest f0000000/meta/kafka/panda-topic/122_9/manifest.bin
[WARNING - 2023-06-07 00:33:08,862 - redpanda - _cloud_storage_diagnostics - lineno:2486]: Failed to decode f0000000/meta/kafka/panda-topic/122_9/manifest.bin
[INFO  - 2023-06-07 00:33:08,862 - redpanda - _cloud_storage_diagnostics - lineno:2470]: Fetching manifest f0000000/meta/kafka/panda-topic/33_9/manifest.bin
[WARNING - 2023-06-07 00:33:08,865 - redpanda - _cloud_storage_diagnostics - lineno:2486]: Failed to decode f0000000/meta/kafka/panda-topic/33_9/manifest.bin
[INFO  - 2023-06-07 00:33:08,865 - redpanda - _cloud_storage_diagnostics - lineno:2470]: Fetching manifest f0000000/meta/kafka/panda-topic/88_9/manifest.bin
[WARNING - 2023-06-07 00:33:08,870 - redpanda - _cloud_storage_diagnostics - lineno:2486]: Failed to decode f0000000/meta/kafka/panda-topic/88_9/manifest.bin

rp_storage_tool is failing to decode manifests at the end of the test

[VladLazar]

The failure to decode is interesting, but it's not the cause of the test failure.
The test failed at:

[ERROR - 2023-06-07 00:32:52,898 - cluster - wrapped - lineno:90]: Test failed, doing failure checks on RedpandaService-0-139916909148720

, but the first decoding failure happens later during the tear-down:

[WARNING - 2023-06-07 00:33:08,416 - redpanda - _cloud_storage_diagnostics - lineno:2486]: Failed to decode 00000000/meta/kafka/panda-topic/10_9/manifest.bin

[VladLazar]

Let's keep this ticket for the decoding failures and chase the timeout in #11268

[VladLazar]

I tried to decode this manifest with rp_storage_tool from the tip of dev (933ad04) and it worked fine. Maybe the tests are using an older version?

[jcsp]

If it decodes with tip of dev and we haven't seen the issue again, I'm inclined to think this is fixed.

[rockwotj]

While debugging another test I hit this locally. Is there anything useful in my local environment I can grab to debug?