Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

config: enable cluster metadata uploads by default #15188

Merged
merged 3 commits into from
Nov 30, 2023
Merged

config: enable cluster metadata uploads by default #15188

merged 3 commits into from
Nov 30, 2023

Conversation

andrwng
Copy link
Contributor

@andrwng andrwng commented Nov 29, 2023 •
edited
Loading

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.2.x
  • v23.1.x
  • v22.3.x

Release Notes

Features

  • In addition to topic data, Redpanda will now upload cluster-related metadata, such as cluster configs, security settings, and consumer offsets. This metadata can be restored along side topic data, allowing for the functional recovery of the entire cluster.

@andrwng andrwng added this to the v23.3.1-rc2 milestone Nov 29, 2023
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Nov 29, 2023 •
edited
Loading

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f9-0a27-45b2-87d7-b716600f7657:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_simple_end_to_end.partition_count=10.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.read_replica_e2e_test.ReadReplicasUpgradeTest.test_upgrades.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.offset_for_leader_epoch_read_replica_test.OffsetForLeaderEpochReadReplicaTest.test_offset_for_leader_epoch"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f3-bf2e-45b0-9a37-c893e4c39868:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_identical_hwms.partition_count=5.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_writes_forbidden.partition_count=10.cloud_storage_type=CloudStorageType.ABS"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f9-0a2b-42a3-9d4a-d5b1af3b6977:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_simple_end_to_end.partition_count=10.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f3-bf33-48a4-a19a-58b1bd2f6e8b:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_writes_forbidden.partition_count=10.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_identical_lwms_after_delete_records.partition_count=5.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f9-0a34-4a20-9e69-510c0d4bcf3e:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_writes_forbidden.partition_count=10.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_identical_lwms_after_delete_records.partition_count=5.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f3-bf38-4838-9ca6-89f6ef4496b6:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_simple_end_to_end.partition_count=10.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.read_replica_e2e_test.ReadReplicasUpgradeTest.test_upgrades.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f3-bf3d-4645-aeeb-a236adb11c0a:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_simple_end_to_end.partition_count=10.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.offset_for_leader_epoch_read_replica_test.OffsetForLeaderEpochReadReplicaTest.test_offset_for_leader_epoch"
"rptest.tests.cluster_config_test.ClusterConfigTest.test_cloud_validation"

new failures in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f9-0a30-485e-823e-4758127b05ac:

"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_identical_hwms.partition_count=5.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_writes_forbidden.partition_count=10.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cluster_config_test.ClusterConfigTest.test_cloud_validation"

new failures in https://buildkite.com/redpanda/redpanda/builds/41935#018c1a7e-e8c6-439c-b60e-ad9b0072df7c:

"rptest.tests.read_replica_e2e_test.ReadReplicasUpgradeTest.test_upgrades.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/41935#018c1a7e-e8ce-4a86-a31d-77c9d5367986:

"rptest.tests.timequery_test.TestReadReplicaTimeQuery.test_timequery"
"rptest.tests.cluster_config_test.ClusterConfigTest.test_cloud_validation"

new failures in https://buildkite.com/redpanda/redpanda/builds/41935#018c1aa5-0e7d-47c3-807c-524629ef022d:

"rptest.tests.read_replica_e2e_test.ReadReplicasUpgradeTest.test_upgrades.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.num_to_upgrade=0.with_tiered_storage=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/41935#018c1aa5-0e7f-4093-b23d-ef5de11ac554:

"rptest.tests.timequery_test.TestReadReplicaTimeQuery.test_timequery"
"rptest.tests.cluster_config_test.ClusterConfigTest.test_cloud_validation"

new failures in https://buildkite.com/redpanda/redpanda/builds/41981#018c1cf1-25e6-4a2d-bd01-33d11675d016:

"rptest.tests.cluster_config_test.ClusterConfigTest.test_cloud_validation"

new failures in https://buildkite.com/redpanda/redpanda/builds/41981#018c1cf1-25df-40a8-83a3-13b11951d9d5:

"rptest.tests.timequery_test.TestReadReplicaTimeQuery.test_timequery"

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Nov 29, 2023 •
edited
Loading

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f3-bf2e-45b0-9a37-c893e4c39868

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/41927#018c19f3-bf33-48a4-a19a-58b1bd2f6e8b

@andrwng
tests: explicitly disable metadata upload loop
55da2ce 
Many tests expect the metadata upload loop to be off by default. This
explicitly sets that.
@andrwng andrwng force-pushed the cluster-metadata-uploads-default branch 3 times, most recently from 218dc8c to 29e7093 Compare November 29, 2023 22:37
@andrwng
rptest: accommodate cluster metadata uploads in tests
efad83b 
The read replica clusters in tests are typically started without
creating a bucket for the cluster, which yields some error logs
complaining about missing buckets when configuring to upload cluster
metadata. So, this disables uploading cluster metadata in most of these
tests. I did the same for other tests that play around with bucket
configs.

The upgrade test is a special beast: there isn't an easy wa to have the
nnew cluster start up with a non-default config. So, I opted to create
a bucket for the read replica in this case.
@andrwng andrwng force-pushed the cluster-metadata-uploads-default branch from 29e7093 to efad83b Compare November 29, 2023 22:39
@andrwng andrwng requested review from VladLazar, Lazin, dotnwat and abhijat and removed request for VladLazar November 29, 2023 23:41
@andrwng
Copy link
Contributor Author

andrwng commented Nov 30, 2023

CI failure is #13275

dotnwat
dotnwat approved these changes Nov 30, 2023
View reviewed changes
Copy link
Member

@dotnwat dotnwat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rad

@piyushredpanda piyushredpanda merged commit 83611a9 into redpanda-data:dev Nov 30, 2023
18 of 20 checks passed
@github-actions github-actions bot mentioned this pull request Dec 22, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotnwat dotnwat dotnwat approved these changes

@Lazin Lazin Awaiting requested review from Lazin

@abhijat abhijat Awaiting requested review from abhijat

@VladLazar VladLazar Awaiting requested review from VladLazar

Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
v23.3.1-rc2
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@andrwng @vbotbuildovich @dotnwat @piyushredpanda