Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved logging around tx processing #14739

Merged
merged 2 commits into from
Nov 8, 2023

Conversation

mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented Nov 3, 2023

Made transaction related logs more structured. Now each log line is prefixed with [tx_id=<value>] which allows to easily query for the transaction id flow.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.2.x
  • v23.1.x
  • v22.3.x

Release Notes

Improvements

  • easier to understand transaction related logs

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40404#018b964d-414e-4759-a409-e3f9fe6e84d4: "rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx_unique_keys"
"rptest.tests.tx_feature_flag_test.TxFeatureFlagTest.test_disabling_transactions_after_they_being_used"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=False"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_1"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_transactions"
"rptest.tests.transactions_test.TransactionsTest.delete_topic_with_active_txns_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_topic"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_tx_coordinator_test"
"rptest.tests.tx_abort_index_test.TxAbortSnapshotTest.test_index_removal"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40404#018b9672-928e-45e1-96b6-1db196ee67e9: "rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=compact.workload=Workload.TX_UNIQUE_KEYS"
"rptest.tests.compaction_e2e_test.CompactionE2ERebootTest.test_write_reboot_read"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.local.target.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True"
"rptest.tests.scramless_eos_test.ScramlessEosTest.test_tx_init_passes"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_producers"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_list_transactions"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_test"
"rptest.tests.transactions_test.TransactionsTest.simple_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_tid"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_simple_get_transaction"
"rptest.tests.tx_reads_writes_test.TxReadsWritesTest.test_reads_writes"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40404#018b964d-4157-491e-b977-dd5118d3daee: "rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_last_timestamp_of_describe_producers"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_etag"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_mark_transaction_expired"
"rptest.tests.tx_overflow_test.TxOverflowTest.underflow_test"
"rptest.tests.transactions_test.TransactionsTest.rejoin_member_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40404#018b9672-929b-41d4-b18d-ffa8c7aadf3c: "rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=compact.workload=Workload.TX"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=delete.workload=Workload.TX_UNIQUE_KEYS"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_last_timestamp_of_describe_producers"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_etag"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_mark_transaction_expired"
"rptest.tests.tx_overflow_test.TxOverflowTest.underflow_test"
"rptest.tests.transactions_test.TransactionsTest.rejoin_member_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40404#018b9672-9291-4613-98a3-39164d91accc: "rptest.tests.tx_feature_flag_test.TxFeatureFlagTest.test_disabling_transactions_after_they_being_used"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.local.target.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx_unique_keys"
"rptest.tests.shadow_indexing_tx_test.ShadowIndexingTxTest.test_shadow_indexing_aborted_txs.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=False"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_1"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_transactions"
"rptest.tests.transactions_test.TransactionsTest.delete_topic_with_active_txns_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_topic"
"rptest.tests.tx_abort_index_test.TxAbortSnapshotTest.test_index_removal"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_tx_coordinator_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40404#018b9672-9297-4be2-aa71-b918ae47a9df: "rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=delete.workload=Workload.TX"
"rptest.tests.shadow_indexing_tx_test.ShadowIndexingTxTest.test_shadow_indexing_aborted_txs.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_2"
"rptest.tests.transactions_test.TransactionsTest.change_static_member_test"
"rptest.tests.transactions_test.TransactionsTest.expired_tx_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_all_transactions"
"rptest.tests.transactions_test.TransactionsTest.init_transactions_creates_eos_topics_test"
"rptest.tests.tx_overflow_test.TxOverflowTest.overflow_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_topic_from_ongoin_tx"
"rptest.tests.tx_verifier_test.TxVerifierTest.test_all_tx_tests"

@vbotbuildovich
Copy link
Collaborator

@piyushredpanda
Copy link
Contributor

/ci-repeat 1

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9854-070d-4c40-8703-71385024adf1: "rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx"
"rptest.tests.tx_feature_flag_test.TxFeatureFlagTest.test_disabling_transactions_after_they_being_used"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_1"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_transactions"
"rptest.tests.transactions_test.TransactionsTest.delete_topic_with_active_txns_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_topic"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_tx_coordinator_test"
"rptest.tests.tx_abort_index_test.TxAbortSnapshotTest.test_index_removal"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9854-0711-442a-8cdd-d8ef8f91b7a2: "rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx_unique_keys"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=False"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_2"
"rptest.tests.transactions_test.TransactionsTest.change_static_member_test"
"rptest.tests.transactions_test.TransactionsTest.init_transactions_creates_eos_topics_test"
"rptest.tests.transactions_test.TransactionsTest.expired_tx_test"
"rptest.tests.tx_overflow_test.TxOverflowTest.overflow_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_all_transactions"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_topic_from_ongoin_tx"
"rptest.tests.tx_verifier_test.TxVerifierTest.test_all_tx_tests"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9854-0718-4337-8a56-dad7009b0c6c: "rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True"
"rptest.tests.scramless_eos_test.ScramlessEosTest.test_tx_init_passes"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_producers"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_list_transactions"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_tid"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_test"
"rptest.tests.transactions_test.TransactionsTest.simple_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_simple_get_transaction"
"rptest.tests.tx_reads_writes_test.TxReadsWritesTest.test_reads_writes"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9854-0714-41e6-af3b-1b802684920d: "rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_last_timestamp_of_describe_producers"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_etag"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_mark_transaction_expired"
"rptest.tests.tx_overflow_test.TxOverflowTest.underflow_test"
"rptest.tests.transactions_test.TransactionsTest.rejoin_member_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9879-7789-464d-8ad4-b065109fddeb: "rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx_unique_keys"
"rptest.tests.shadow_indexing_tx_test.ShadowIndexingTxTest.test_shadow_indexing_aborted_txs.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=False"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_2"
"rptest.tests.transactions_test.TransactionsTest.change_static_member_test"
"rptest.tests.transactions_test.TransactionsTest.expired_tx_test"
"rptest.tests.transactions_test.TransactionsTest.init_transactions_creates_eos_topics_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_all_transactions"
"rptest.tests.tx_overflow_test.TxOverflowTest.overflow_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_topic_from_ongoin_tx"
"rptest.tests.tx_verifier_test.TxVerifierTest.test_all_tx_tests"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9879-778c-43b0-883a-0c84919d36fc: "rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.local.target.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=delete.workload=Workload.TX"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_last_timestamp_of_describe_producers"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_mark_transaction_expired"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_etag"
"rptest.tests.tx_overflow_test.TxOverflowTest.underflow_test"
"rptest.tests.transactions_test.TransactionsTest.rejoin_member_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9879-7790-430a-9a20-41b225f37e50: "rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.local.target.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=delete.workload=Workload.TX_UNIQUE_KEYS"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=compact.workload=Workload.TX"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True"
"rptest.tests.scramless_eos_test.ScramlessEosTest.test_tx_init_passes"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_producers"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_list_transactions"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_test"
"rptest.tests.transactions_test.TransactionsTest.simple_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_tid"
"rptest.tests.tx_reads_writes_test.TxReadsWritesTest.test_reads_writes"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_simple_get_transaction"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40445#018b9879-7786-4450-9586-8e166f0a19c6: "rptest.tests.compacted_verifier_test.CompactedVerifierTest.test_tx"
"rptest.tests.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=compact.workload=Workload.TX_UNIQUE_KEYS"
"rptest.tests.compaction_e2e_test.CompactionE2ERebootTest.test_write_reboot_read"
"rptest.tests.tx_feature_flag_test.TxFeatureFlagTest.test_disabling_transactions_after_they_being_used"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.shadow_indexing_tx_test.ShadowIndexingTxTest.test_shadow_indexing_aborted_txs.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"
"rptest.tests.scramful_eos_test.ScramfulEosTest.test_tx_init_passes_1"
"rptest.tests.transaction_kafka_api_test.TxKafkaApiTest.test_describe_transactions"
"rptest.tests.transactions_test.TransactionsTest.delete_topic_with_active_txns_test"
"rptest.tests.tx_admin_api_test.TxAdminTest.test_delete_non_existent_topic"
"rptest.tests.transactions_test.TransactionsTest.graceful_leadership_transfer_tx_coordinator_test"
"rptest.tests.tx_abort_index_test.TxAbortSnapshotTest.test_index_removal"

@vbotbuildovich
Copy link
Collaborator

@vbotbuildovich
Copy link
Collaborator

@vbotbuildovich
Copy link
Collaborator

src/v/cluster/tm_stm.cc Outdated Show resolved Hide resolved
src/v/cluster/tm_stm_cache.cc Outdated Show resolved Hide resolved
src/v/cluster/tx_gateway_frontend.cc Outdated Show resolved Hide resolved
@vbotbuildovich
Copy link
Collaborator

Some of the operators were incorrectly declared while all of them were
lacking definition

Signed-off-by: Michal Maslanka <michal@redpanda.com>
Signed-off-by: Michal Maslanka <michal@redpanda.com>
@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40616#018bab0c-5f5e-4601-9e1b-437a4fc6e595: "rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=delete"

@mmaslankaprv
Copy link
Member Author

/ci-repeat 1

@mmaslankaprv mmaslankaprv merged commit 172b311 into redpanda-data:dev Nov 8, 2023
26 checks passed
@mmaslankaprv mmaslankaprv deleted the tx-logging branch November 8, 2023 11:52
@vbotbuildovich
Copy link
Collaborator

/backport v23.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-14739-v23.2.x-472 remotes/upstream/v23.2.x
git cherry-pick -x f65999a837b1ffd82f53dfe2137d049492e832af b8f63a76bc1f9943503a61951321381a6f62013a

Workflow run logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants