Skip to content

Comments

bazel: upgrade to bazel 8#25368

Merged
rockwotj merged 13 commits intoredpanda-data:devfrom
rockwotj:clang-19
Apr 17, 2025
Merged

bazel: upgrade to bazel 8#25368
rockwotj merged 13 commits intoredpanda-data:devfrom
rockwotj:clang-19

Conversation

@rockwotj
Copy link
Contributor

@rockwotj rockwotj commented Mar 13, 2025

Upgrade bazel to v8

and also upgrade clang-19 build and seastar. Kind of a mash of lots of build related stuff :)

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 16, 2025

CI test results

test results on build#63204
test_id test_kind job_url test_status passed
kafka_server_rpfixture.kafka_server_rpfixture unit https://buildkite.com/redpanda/redpanda/builds/63204#01959cd7-8645-4da4-93c6-995cb04b2822 FLAKY 1/2
rptest.tests.crash_loop_checks_test.CrashLoopChecksTest.test_crash_report_with_signal.signo=Signals.SIGILL.signal_shard=1 ducktape https://buildkite.com/redpanda/redpanda/builds/63204#01959d18-bde7-47ac-8610-2f5db61a5747 FLAKY 1/2
rptest.tests.datalake.datalake_upgrade_test.DatalakeUpgradeTest.test_upload_through_upgrade.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK ducktape https://buildkite.com/redpanda/redpanda/builds/63204#01959d1c-7aa9-4b30-9e38-41d639447f44 FLAKY 1/4
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic ducktape https://buildkite.com/redpanda/redpanda/builds/63204#01959d1c-7aa8-4644-acee-7cad327746fc FLAKY 1/3
test results on build#64359
test_id test_kind job_url test_status passed
rptest.tests.datalake.throttling_test.DatalakeThrottlingTest.test_basic_throttling.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/64359#01961ec2-c97f-4696-9e9c-a494cad29875 FLAKY 1/2
rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=False ducktape https://buildkite.com/redpanda/redpanda/builds/64359#01961ebe-1723-48c3-9090-ce61f4a96913 FLAKY 1/3
test results on build#64392
test_id test_kind job_url test_status passed
rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete ducktape https://buildkite.com/redpanda/redpanda/builds/64392#01962342-598d-46b7-b828-07e89c229c7e FLAKY 1/2
rptest.tests.datalake.disk_budget_test.DatalakeDiskUsageTest.test_idle_finish.num_partitions=30.concurrent_translations=1.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/64392#01962342-599c-4afd-9ad5-27f9db6374f1 FLAKY 1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic ducktape https://buildkite.com/redpanda/redpanda/builds/64392#01962342-599c-4afd-9ad5-27f9db6374f1 FLAKY 1/2
test results on build#64647
test_id test_kind job_url test_status passed
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=.cancellation.dir.in.stage.preparing.use_alias.True ducktape https://buildkite.com/redpanda/redpanda/builds/64647#019641b3-217c-4abd-aa21-356647b0392d FLAKY 14/21
rptest.tests.datalake.throttling_test.DatalakeThrottlingTest.test_basic_throttling.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/64647#0196419f-0484-4324-aaa1-0f9a4361467b FLAKY 14/21
rptest.tests.datalake.throttling_test.DatalakeThrottlingTest.test_basic_throttling.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP ducktape https://buildkite.com/redpanda/redpanda/builds/64647#0196419f-0485-4ca8-a8f5-227ab711a924 FLAKY 18/21
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=False.with_tiered_storage=True.with_iceberg=True.with_chunked_compaction=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/64647#0196419f-0483-43e9-8cc8-3428e11d6cb9 FLAKY 12/13
test results on build#64663
test_id test_kind job_url test_status passed
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_creating_and_listing_migrations ducktape https://buildkite.com/redpanda/redpanda/builds/64663#019642d4-a01a-4036-bb83-dbf6007cbd49 FLAKY 20/21

dotnwat
dotnwat previously approved these changes Apr 10, 2025
@rockwotj
Copy link
Contributor Author

Enabling seastar assertions in release mode trips up the fetch benchmark, will investigate more

external/+non_module_dependencies+seastar/include/seastar/core/sharded.hh:566: seastar::sharded<seastar::abort_source>::~sharded() [Service = seastar::abort_source]: Assertion `_instances.empty()` failed.
zsh: illegal hardware instruction  bazel-bin/src/v/kafka/server/tests/kafka_fetch_rpbench

@travisdowns
Copy link
Member

@rockwotj we are working (struggling) to get bazel microbenchmarks fully enabled in bazel with baseline results, it would be great to have that working and with some data points before this goes in (should be up tomorrow), WDYT?

cc @StephanDollberg

@rockwotj
Copy link
Contributor Author

Yeah that's a good idea. I'll wait a week before merging this

@rockwotj
Copy link
Contributor Author

Enabling seastar assertions in release mode trips up the fetch benchmark, will investigate more

external/+non_module_dependencies+seastar/include/seastar/core/sharded.hh:566: seastar::sharded<seastar::abort_source>::~sharded() [Service = seastar::abort_source]: Assertion `_instances.empty()` failed.
zsh: illegal hardware instruction  bazel-bin/src/v/kafka/server/tests/kafka_fetch_rpbench

Okay I believe I've fixed this issue. Will hold off on merging until microbenchmarks are stable in Bazel.

@travisdowns
Copy link
Member

Okay I believe I've fixed this issue. Will hold off on merging until microbenchmarks are stable in Bazel.

Thank you, we are getting there.

@rockwotj
Copy link
Contributor Author

I've reverted the clang-19 upgrade and just upgraded to bazel 8 which will help a ton with these silly lockfile conflicts.

@rockwotj rockwotj requested a review from dotnwat April 16, 2025 21:35
@rockwotj rockwotj changed the title bazel: upgrade to clang-19 bazel: upgrade to bazel 8 Apr 16, 2025
@rockwotj
Copy link
Contributor Author

I will upgrade to clang 19 after we have benchmarks stable, but I would love to get this in

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Apr 17, 2025

Retry command for Build#64639

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/rack_aware_replica_placement_test.py::RackAwarePlacementTest.test_rack_awareness_after_node_operations@{"rack_layout":"ABCDEF"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_topic_lifecycle@{"catalog_type":"nessie","cloud_storage_type":1}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"preparing"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/upgrade_test.py::UpgradeBackToBackTest.test_upgrade_with_all_workloads@{"single_upgrade":false}

From libc++19 release notes:

> libc++ no longer supports std::allocator<const T> and containers of const-qualified element type, such as std::vector<const T> and std::list<const T>. This used to be supported as an undocumented extension. If you were using std::vector<const T>, replace it with std::vector<T> instead. The _LIBCPP_ENABLE_REMOVED_ALLOCATOR_CONST macro can be defined to temporarily re-enable this extension. to temporarily re-enable this extension to make it easier to update user code This macro will be honored for one released and ignored starting in LLVM 20. To assist with the clean-up process, consider running your code through Clang Tidy, with std-allocator-const enabled.
From libc++19 release notes:

> libc++ no longer supports std::allocator<const T> and containers of const-qualified element type, such as std::vector<const T> and std::list<const T>. This used to be supported as an undocumented extension. If you were using std::vector<const T>, replace it with std::vector<T> instead. The _LIBCPP_ENABLE_REMOVED_ALLOCATOR_CONST macro can be defined to temporarily re-enable this extension. to temporarily re-enable this extension to make it easier to update user code This macro will be honored for one released and ignored starting in LLVM 20. To assist with the clean-up process, consider running your code through Clang Tidy, with std-allocator-const enabled.
To fix an warning that crops up in Bazel 8
This picks up the new SEASTAR_ASSERT macro, which removes some warnings
from the build in Bazel.
This brings vastly better lockfile management and will cause many less
merge conflicts because of the silly lockfile in Bazel.
This was not cleaned up properly, which since we moved to a newer
version of Seastar that had asserts turned on in release mode, it caught
these issues of not properly stopping the sharded abort source.
Also upgrade re2 to get past bazelbuild/bazel#24426
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Apr 17, 2025

Retry command for Build#64647

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"preparing"},"use_alias":true},"transfer_leadership":true}
tests/rptest/tests/datalake/throttling_test.py::DatalakeThrottlingTest.test_basic_throttling@{"catalog_type":"nessie","cloud_storage_type":1}

@rockwotj
Copy link
Contributor Author

/ci-repeat 1
skip-unit
skip-redpanda-build
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"preparing"},"use_alias":true},"transfer_leadership":true}
tests/rptest/tests/datalake/throttling_test.py::DatalakeThrottlingTest.test_basic_throttling@{"catalog_type":"nessie","cloud_storage_type":1}

@StephanDollberg
Copy link
Member

I will upgrade to clang 19 after we have benchmarks stable, but I would love to get this in

I thought they were now but looks like they are broken again because of some task issues. Will be looking shortly.

@rockwotj
Copy link
Contributor Author

/ci-repeat 1
skip-unit
skip-redpanda-build
tests/rptest/tests/data_migrations_api_test.py
tests/rptest/tests/datalake/throttling_test.py

@vbotbuildovich
Copy link
Collaborator

Retry command for Build#64663

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_creating_and_listing_migrations

@rockwotj rockwotj merged commit 56fb767 into redpanda-data:dev Apr 17, 2025
16 of 19 checks passed
@rockwotj rockwotj deleted the clang-19 branch April 17, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants