Skip to content

Comments

rpk: Add guardrails for Tiered Storage properties#25470

Merged
Lazin merged 3 commits intoredpanda-data:devfrom
Lazin:pr/rpk-confirm-ts-disable
Apr 3, 2025
Merged

rpk: Add guardrails for Tiered Storage properties#25470
Lazin merged 3 commits intoredpanda-data:devfrom
Lazin:pr/rpk-confirm-ts-disable

Conversation

@Lazin
Copy link
Contributor

@Lazin Lazin commented Mar 20, 2025

Prompt the user for confirmation if they are trying to set cloud_storage_enable_remote_write cluster config to 'false'. Do the same if they're trying to set topic property redpanda.remote.write to 'false'. This operation is potentially unsafe because the data loss might occur if the Tiered Storage was active. In both cases the message says Warning: disabling Tiered Storage may lead to data loss. If you only want to pause Tiered Storage temporarily, use the 'cloud_storage_enable_segment_uploads' option. Abort? and the default choice is Yes. So the warning suggests a safe alternative and in order to use non-safe option the user have to chose it explicitly.

This is a part of the safe pause/resume feature - redpanda-data/docs#1017 (docs PR).

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.1.x
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@Lazin Lazin requested review from a team, kbatuigas and r-vasquez as code owners March 20, 2025 14:42
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 20, 2025

Retry command for Build#63439

please wait until all jobs are finished before running the slash command



/ci-repeat 1
tests/rptest/tests/delete_retention_ms_test.py::DeleteRetentionMsTest.test_create_topic_configurations_cluster_defaults
tests/rptest/tests/alter_topic_configuration_test.py::AlterTopicConfiguration.test_shadow_indexing_config
tests/rptest/tests/cloud_storage_scrubber_test.py::CloudStorageScrubberTest.test_scrubber@{"cloud_storage_type":1}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset@{"cloud_storage_type":2}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset_spillover@{"cloud_storage_type":2}
tests/rptest/tests/alter_topic_configuration_test.py::ShadowIndexingGlobalConfig.test_overrides_remove
tests/rptest/tests/cloud_storage_scrubber_test.py::CloudStorageScrubberTest.test_scrubber@{"cloud_storage_type":2}
tests/rptest/tests/datalake/transactions_test.py::DatalakeTransactionTests.test_with_transactions@{"cloud_storage_type":1,"compaction":false}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset@{"cloud_storage_type":1}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset_spillover@{"cloud_storage_type":1}
tests/rptest/tests/alter_topic_configuration_test.py::ShadowIndexingGlobalConfig.test_overrides_set

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 20, 2025

CI test results

test results on build#63439
test_id test_kind job_url test_status passed
rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_shadow_indexing_config ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5a-4da9-9c4b-84f6a83239c5 FAIL 0/20
rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_shadow_indexing_config ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1e-49ac-bc16-2d86c69b137a FAIL 0/20
rptest.tests.alter_topic_configuration_test.ShadowIndexingGlobalConfig.test_overrides_remove ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5b-4db7-9f61-b51c9031295f FAIL 0/20
rptest.tests.alter_topic_configuration_test.ShadowIndexingGlobalConfig.test_overrides_remove ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1f-4987-98db-400d3936e567 FAIL 0/20
rptest.tests.alter_topic_configuration_test.ShadowIndexingGlobalConfig.test_overrides_set ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5c-414e-b841-36f5e5d96b14 FAIL 0/20
rptest.tests.alter_topic_configuration_test.ShadowIndexingGlobalConfig.test_overrides_set ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f20-4901-97a3-2964c35b3d70 FAIL 0/20
rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5c-414e-b841-36f5e5d96b14 FAIL 0/20
rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f20-4901-97a3-2964c35b3d70 FAIL 0/20
rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5a-47af-a107-9ea3c3df8d4f FAIL 0/20
rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1e-481f-8d6e-fe47ec5c9638 FAIL 0/20
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=.cancellation.dir.out.stage.preparing.use_alias.False ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5b-4db7-9f61-b51c9031295f FLAKY 1/2
rptest.tests.datalake.compaction_test.CompactionTest.test_compaction.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1e-49ac-bc16-2d86c69b137a FLAKY 1/2
rptest.tests.datalake.datalake_upgrade_test.DatalakeUpgradeTest.test_upload_through_upgrade.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f20-4901-97a3-2964c35b3d70 FLAKY 1/2
rptest.tests.datalake.transactions_test.DatalakeTransactionTests.test_with_transactions.cloud_storage_type=CloudStorageType.S3.compaction=False ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5c-414e-b841-36f5e5d96b14 FAIL 0/1
rptest.tests.delete_retention_ms_test.DeleteRetentionMsTest.test_create_topic_configurations_cluster_defaults ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5a-4da9-9c4b-84f6a83239c5 FAIL 0/20
rptest.tests.delete_retention_ms_test.DeleteRetentionMsTest.test_create_topic_configurations_cluster_defaults ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1f-4987-98db-400d3936e567 FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5b-4db7-9f61-b51c9031295f FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1e-49ac-bc16-2d86c69b137a FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5c-414e-b841-36f5e5d96b14 FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1f-4987-98db-400d3936e567 FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5b-4db7-9f61-b51c9031295f FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1e-49ac-bc16-2d86c69b137a FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5c-414e-b841-36f5e5d96b14 FAIL 0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b465-0f1f-4987-98db-400d3936e567 FAIL 0/20
rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_schema_id_validation.protocol=AVRO.client_type=Python.validate_schema_id=True.subject_name_strategy=SubjectNameStrategyCompat.TOPIC_RECORD_NAME.payload_class=com.redpanda.A.B.C.D.NestedPayload.compression_type=CompressionTypes.ZSTD ducktape https://buildkite.com/redpanda/redpanda/builds/63439#0195b444-3f5a-4da9-9c4b-84f6a83239c5 FLAKY 1/2
test results on build#63977
test_id test_kind job_url test_status passed
rptest.tests.datalake.cluster_restore_test.DatalakeClusterRestoreTest.test_restore_partition_spec.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP ducktape https://buildkite.com/redpanda/redpanda/builds/63977#0195ecd2-34d8-4ca7-9a2a-b896915acf8c FLAKY 1/2
rptest.tests.datalake.throttling_test.DatalakeThrottlingTest.test_basic_throttling.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63977#0195ecd2-34db-49e1-af43-810c416fdb15 FLAKY 1/2
rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade ducktape https://buildkite.com/redpanda/redpanda/builds/63977#0195eccf-f71e-4593-a176-5f78928e94ce FLAKY 1/2

out.Die("invalid arguments: %v, please use one of 'rpk cluster config set <key> <value>' or 'rpk cluster config set <key>=<value>'", args)
}
// Disabling Tiered Storage requires a confirmation from the user because it may lead to data loss.
if key == "cloud_storage_enable_remote_write" && value == "false" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, however, we should provide a way to avoid these confirmation prompts in the CLI. Similar to what we do in other commands with confirmation prompts.

Example: https://github.com/redpanda-data/redpanda/blob/dev/src/go/rpk/pkg/cli/security/role/delete.go#L71
we use this flag name across rpk

The same comment applies to topic config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@Feediver1
Copy link

@Lazin Not sure what to do with the pause/resume doc PR, as this has not yet been merged. Need some guidance on whether this will be added before GA. thx

Lazin added 3 commits March 31, 2025 10:11
Prompt if the user is trying to set 'cloud_storage_enable_remote_write'
to 'false'. This operation is potentialy unsafe because the data loss
might occur.

The prompt can be avoided by passing '--no-confirm'. This flag is
supposed to be used in tests and automation scripts.

Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
in case if the user is disabling 'redpanda.remote.write'. This is a
potentially dangerous operation that can lead to data loss. The prompt
suggest an alternaive for that (safe pause).

The '--no-confirm' flag that allows the user to bypass the prompt is
added. The flag is supposed to be used in scripts and CI.

Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
In order to disable TS the rpk user must provide positive confirmation
now. This breaks some tests. In order to avoid this the '--no-confirm'
flag is passed to the rpk.

Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
@Lazin Lazin force-pushed the pr/rpk-confirm-ts-disable branch from 2a55c7f to f47a5a1 Compare March 31, 2025 14:13
@Lazin Lazin requested a review from r-vasquez March 31, 2025 16:20
@Lazin
Copy link
Contributor Author

Lazin commented Apr 3, 2025

@Feediver1 I'm going to merge this and backport to v25.1.x but it's not part of the RC

@Lazin Lazin merged commit 0221de4 into redpanda-data:dev Apr 3, 2025
26 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v25.1.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants