"Reader cannot read before start of the log" logging endlessly on enabling TS

[daisukebe]

Version & Environment

Redpanda version: (use rpk version): v23.3.7

Under a certain condition, brokers ends up with complaining below forever on enabling cloud_storage_enabled

WARN  2024-03-14 14:10:40,229 [shard 1:main] raft - [group_id:1, {kafka/foo/0}] state_machine_manager.cc:419 - exception thrown from background apply fiber for archival_metadata_stm - std::runtime_error (Reader cannot read before start of the log 0 < 11)

What went wrong?

What works

• Produce
• Show / alter topic configs

What doesn't work

• Consume
• List partitions

What should have happened instead?

Warning should not last forever and nothing visible to users should happen.

How to reproduce the issue?

1. Have a cluster with TS disabled
2. Create a topic and produce some records
3. Let retention kick in and move start offset forward
4. Produce some other records, hence, making end_offset > start_offset (non zero)
5. Enable TS and restart the broker. Here're configs changed

cloud_storage_enabled: true
cloud_storage_enable_remote_write: true
cloud_storage_enable_remote_read: true
cloud_storage_azure_container: givencontainer
cloud_storage_azure_storage_account: givenaccount
cloud_storage_azure_shared_key: [redacted]
cloud_storage_segment_max_upload_interval_sec: 10

6. You'll hit the issue

Additional information

Please attach any relevant logs, backtraces, or metric charts.

[Lazin]

On step 3 do we need retention to remove all data?

[daisukebe]

Not necessarily. The problem arises with either case below on step 3.

• LOG-START-OFFSET == HIGH-WATERMARK
• LOG-START-OFFSET < HIGH-WATERMARK