When topics are created via auto-create Tiered storage is not enabled correctly

[jason-da-redpanda]

Version & Environment

23.2.8

What went wrong?

If a cluster is configured so that cloud storage is enabled correctly ...
per https://docs.redpanda.com/current/manage/tiered-storage/#set-up-tiered-storage
and it defaults so that new topics are created with TS enabled

e.g

redpanda.remote.read          true        
redpanda.remote.write         true  

if topics created via CLI/console.. topics work as expected in that segments are uploaded to cloud storage +
rpk topic describe-storage shows the topic as CLOUD-STORAGE-MODE =full

if the topic is created via the "auto-create" mechanism

https://docs.redpanda.com/current/manage/tiered-storage/#set-up-tiered-storage

then
rpk topic describe-storage' shows CLOUD-STORAGE-MODE disabled`

which is incorrect and means segments are not uploaded to cloud..

What should have happened instead?

topics created via auto-create mechanism should have TS enabled correctly ....

How to reproduce the issue?

1. configure tiered storage per https://docs.redpanda.com/current/manage/tiered-storage/#set-up-tiered-storage
2. enable auto topic create on cluster
   rpk cluster config set auto_create_topics_enabled true
3. create a topic via CLI
   rpk topic create plonky
4. confirm that rpk topic describe-storage plonky shows CLOUD-STORAGE-MODE =full (as expected)
   and rpk topic describe plonky
   shows

redpanda.remote.read          true        
redpanda.remote.write         true  

5. produce to a non-existant topic... so topic gets created via auto topic create feature

rpk topic produce plonky2 --allow-auto-topic-creation

6. confirm that rpk topic describe-storage plonky2' shows CLOUD-STORAGE-MODE disabled (not expected..bug) and rpk topic describe plonky2`
   shows

redpanda.remote.read          true        
redpanda.remote.write         true 

Additional information

Please attach any relevant logs, backtraces, or metric charts.

[andijcr]

this seems to be also in dev.

I am able to confirm that via rpk describe topic config, both auto-created and manually-created topics are almost the same

[ERROR - 2023-09-19 16:26:21,002 - topic_creation_test - topic_autocreate_test - lineno:158]: auto_topic='autocreated'        config={'cleanup.policy': ('delete', 'DEFAULT_CONFIG'),       'compression.type': ('producer', 'DEFAULT_CONFIG'), 'max.message.bytes': ('1048576', 'DEFAULT_CONFIG'), 'message.timestamp.type': ('CreateTime', 'DEFAULT_CONFIG'), 'redpanda.remote.delete': ('true', 'DEFAULT_CONFIG'), 'redpanda.remote.read': ('true', 'DEFAULT_CONFIG'), 'redpanda.remote.write': ('true', 'DEFAULT_CONFIG'), 'retention.bytes': ('-1', 'DEFAULT_CONFIG'), 'retention.local.target.bytes': ('-1', 'DEFAULT_CONFIG'), 'retention.local.target.ms': ('86400000', 'DEFAULT_CONFIG'), 'retention.ms': ('604800000', 'DEFAULT_CONFIG'), 'segment.bytes': ('16000000', 'DEFAULT_CONFIG'), 'segment.ms': ('1209600000', 'DEFAULT_CONFIG')}

[ERROR - 2023-09-19 16:26:22,592 - topic_creation_test - topic_autocreate_test - lineno:167]: manual_topic='manuallycreated', config={'cleanup.policy': ('delete', 'DYNAMIC_TOPIC_CONFIG'), 'compression.type': ('producer', 'DEFAULT_CONFIG'), 'max.message.bytes': ('1048576', 'DEFAULT_CONFIG'), 'message.timestamp.type': ('CreateTime', 'DEFAULT_CONFIG'), 'redpanda.remote.delete': ('true', 'DEFAULT_CONFIG'), 'redpanda.remote.read': ('true', 'DEFAULT_CONFIG'), 'redpanda.remote.write': ('true', 'DEFAULT_CONFIG'), 'retention.bytes': ('-1', 'DEFAULT_CONFIG'), 'retention.local.target.bytes': ('-1', 'DEFAULT_CONFIG'), 'retention.local.target.ms': ('86400000', 'DEFAULT_CONFIG'), 'retention.ms': ('604800000', 'DEFAULT_CONFIG'), 'segment.bytes': ('16000000', 'DEFAULT_CONFIG'), 'segment.ms': ('1209600000', 'DEFAULT_CONFIG')}

The only difference is {'cleanup.policy': ('delete', 'DYNAMIC_TOPIC_CONFIG') for the manually created topic and {'cleanup.policy': ('delete', 'DEFAULT_CONFIG'), for the automatic topic.

cloud_storage/status API endpoint returns a different story:

auto_topic_cs_status=  {'cloud_storage_mode': 'disabled',  'metadata_update_pending': False, 'total_log_size_bytes': 99, 'cloud_log_size_bytes': 0, 'stm_region_size_bytes': 0, 'archive_size_bytes': 0, 'local_log_size_bytes': 99, 'stm_region_segment_count': 0, 'local_log_segment_count': 1, 'local_log_start_offset': 0, 'local_log_last_offset': 0} manual_topic_cs_status={'cloud_storage_mode': 'full', 'ms_since_last_manifest_upload': 1357, 'metadata_update_pending': False, 'total_log_size_bytes': 253, 'cloud_log_size_bytes': 0, 'stm_region_size_bytes': 0, 'archive_size_bytes': 0, 'local_log_size_bytes': 253, 'stm_region_segment_count': 0, 'local_log_segment_count': 1, 'local_log_start_offset': 0, 'local_log_last_offset': 0}
manual_topic_cs_status={'cloud_storage_mode': 'full',    'ms_since_last_manifest_upload': 1357, 'metadata_update_pending': False, 'total_log_size_bytes': 253, 'cloud_log_size_bytes': 0, 'stm_region_size_bytes': 0, 'archive_size_bytes': 0, 'local_log_size_bytes': 253, 'stm_region_segment_count': 0, 'local_log_segment_count': 1, 'local_log_start_offset': 0, 'local_log_last_offset': 0}

cloud_storage_mode disabled vs cloud_storage_mode full.

I am trying to find the difference in codepath.

In theory, if the produce request is done to the non-leader of the cluster, this bug should not appear because the follower, instead of creating the topic on itself sends to the controller leader a standard create topic request.

[andijcr]

PR to fix this behavior is up, but will likely need some further work to land in

[michael-redpanda]

Slack thread: https://redpandadata.slack.com/archives/C04AVDG7N58/p1695224914461349?thread_ts=1695224638.984009&cid=C04AVDG7N58