Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ducktape: improve ubsan violation handling in dt tests #17529

Merged
merged 2 commits into from
Apr 2, 2024
Merged

ducktape: improve ubsan violation handling in dt tests #17529

merged 2 commits into from
Apr 2, 2024

Conversation

bharathv
Copy link
Contributor

@bharathv bharathv commented Apr 1, 2024

Currently the test continues despite a violation and includes a single log line that the violation has occurred which makes it difficult to debug further especially narrowing down the exact source that caused it..

This change makes the test halt on first violation include a full stack trace in the broker logs.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x

Release Notes

  • none

@bharathv
ducktape: improve ubsan violation handling in dt tests
fc8e7a0 
Currently the test continues despite a violation and includes a single
log line that the violation has occurred which makes it difficult to
debug further especially narrowing down the exact source that caused it..

This change makes the test halt on first violation include a full stack
trace in the broker logs.
@bharathv
Copy link
Contributor Author

bharathv commented Apr 1, 2024

Current:

/vectorized/llvm/bin/../include/c++/v1/__chrono/duration.h:429:35: runtime error: signed integer overflow: 3821777540142 + 9223372036854000000 cannot be represented in type 'rep' (aka 'long long')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /vectorized/llvm/bin/../include/c++/v1/__chrono/duration.h:429:35 in 

<-- continue test --->

after this change, behavior similar to asan error handling.

==285==WARNING: Failed to use and restart external symbolizer!
    #0 0x7f847c74a062  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x374a062) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #1 0x7f847c99f33c  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x399f33c) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #2 0x7f847c50d7f3  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x350d7f3) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #3 0x7f847c36b4b7  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x336b4b7) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #4 0x7f847c3c7d52  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x33c7d52) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #5 0x7f847c3c7c83  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x33c7c83) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #6 0x7f847c3c7983  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x33c7983) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #7 0x7f847c3c7814  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x33c7814) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #8 0x7f847c3c75e8  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x33c75e8) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #9 0x7f847c3c7307  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x33c7307) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #10 0x7f847c2fc00b  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x32fc00b) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #11 0x7f847c2f9a30  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x32f9a30) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #12 0x7f847d46d68d  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x446d68d) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #13 0x7f847d46cd05  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x446cd05) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #14 0x7f847d4441fa  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x44441fa) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #15 0x7f847d442bf8  (/work/redpanda/vbuild/debug/clang/lib/libv_v_raft.so+0x4442bf8) (BuildId: 2dc21deb552be437bb2ab613aa61dbf0c0a49d60)
    #16 0x7f84a2c5b160  (/work/redpanda/vbuild/debug/clang/lib/libv_v_cluster.so+0x1d25b160) (BuildId: b0a9d6303adab52d30e8ac685f262d6fb49fc3ec)
    #17 0x7f84a2cd1b2b  (/work/redpanda/vbuild/debug/clang/lib/libv_v_cluster.so+0x1d2d1b2b) (BuildId: b0a9d6303adab52d30e8ac685f262d6fb49fc3ec)
    #18 0x7f84a2cd151c  (/work/redpanda/vbuild/debug/clang/lib/libv_v_cluster.so+0x1d2d151c) (BuildId: b0a9d6303adab52d30e8ac685f262d6fb49fc3ec)
    #19 0x7f84509aa914  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x47aa914) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #20 0x7f84509b7b91  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x47b7b91) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #21 0x7f84509bddfb  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x47bddfb) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #22 0x7f8450b3b2d6  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x493b2d6) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #23 0x7f8450b39b60  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x4939b60) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #24 0x7f8450b39a20  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x4939a20) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #25 0x7f8450b399c4  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x49399c4) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #26 0x7f8450b34ff0  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x4934ff0) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #27 0x7f84ddb5fba1  (/work/redpanda/vbuild/debug/clang/lib/libv_v_application.so+0x8f5fba1) (BuildId: 14a055923cf8d300f9726b858ce4d680aa72f092)
    #28 0x7f84ddb5b708  (/work/redpanda/vbuild/debug/clang/lib/libv_v_application.so+0x8f5b708) (BuildId: 14a055923cf8d300f9726b858ce4d680aa72f092)
    #29 0x7f845078e6fc  (/work/redpanda/vbuild/debug/clang/rp_deps_install/lib64/libseastar.so+0x458e6fc) (BuildId: 6edcd393e08c7cade16a10125f1d930695e34619)
    #30 0x7f844baac896  (/work/redpanda/vbuild/debug/clang/dist/local/redpanda/lib/libc.so.6+0x8e896) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #31 0x7f844bb336fb  (/work/redpanda/vbuild/debug/clang/dist/local/redpanda/lib/libc.so.6+0x1156fb) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /work/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__chrono/duration.h:432:35 in 
AddressSanitizer:DEADLYSIGNAL
       295:     __vdso_gettimeofday

ballard26
ballard26 previously approved these changes Apr 1, 2024
View reviewed changes
Copy link
Contributor

@ballard26 ballard26 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Apr 1, 2024 •
edited

new failures in https://buildkite.com/redpanda/redpanda/builds/47164#018e9ab1-02fd-4e1c-aa93-1470c2138350:

"rptest.tests.redpanda_kerberos_test.RedpandaKerberosRulesTesting.test_kerberos_mapping_rules.rules=.RULE.1.1test.0.client.DEFAULT.kerberos_principal=client.rp_user=clienttestEXAMPLE.COM.expected_topics=.restricted.always_visible.acl=.restricted.clienttestEXAMPLE.COM.always_visible"

new failures in https://buildkite.com/redpanda/redpanda/builds/47164#018e9ab1-02f5-42c7-8d40-864972a8f868:

"rptest.tests.redpanda_kerberos_test.RedpandaKerberosRulesTesting.test_kerberos_mapping_rules.rules=.RULE.2.1testbad.0.client.RULE.1.1testgood.client.s.client.1redpanda.U.DEFAULT.kerberos_principal=client.rp_user=TESTGOODREDPANDA.expected_topics=.restricted.always_visible.acl=.restricted.TESTGOODREDPANDA.always_visible"

new failures in https://buildkite.com/redpanda/redpanda/builds/47164#018e9ab1-02fa-4a47-acf9-a56ba7f3f755:

"rptest.tests.redpanda_kerberos_test.RedpandaKerberosTest.test_init.req_principal=client.acl=True.topics=.needs_acl.always_visible.fail=False"
"rptest.tests.redpanda_kerberos_test.RedpandaKerberosConfigTest.test_non_default"

new failures in https://buildkite.com/redpanda/redpanda/builds/47164#018e9ab1-02f7-4ded-887b-5928c86f11cf:

"rptest.tests.redpanda_kerberos_test.GSSAPIReauthTest.test_gssapi_reauth"
"rptest.tests.redpanda_kerberos_test.RedpandaKerberosTest.test_init.req_principal=client.acl=False.topics=.always_visible.fail=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/47177#018e9c18-72fe-401f-9eff-c1683cc676fd:

"rptest.tests.full_disk_test.FullDiskReclaimTest.test_full_disk_triggers_gc"

dotnwat
dotnwat previously approved these changes Apr 1, 2024
View reviewed changes
@bharathv bharathv dismissed stale reviews from dotnwat and ballard26 via de4f877 April 1, 2024 22:49
@bharathv bharathv requested a review from a team as a code owner April 1, 2024 22:49
@bharathv bharathv requested review from savex and removed request for a team April 1, 2024 22:49
@bharathv
Copy link
Contributor Author

bharathv commented Apr 1, 2024

latest push added support for UBSAN suppressions file.

@bharathv
ducktape: support ubsan suppressions
b94de09 
Populated with currently known UB issues in krb5 library.
@vbotbuildovich
Copy link
Collaborator

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47177#018e9ca1-fd8d-49e2-a529-98ed960b5c49

@bharathv
Copy link
Contributor Author

bharathv commented Apr 2, 2024

Failure #17550 (new, unrelated)

@piyushredpanda piyushredpanda merged commit 6cef001 into redpanda-data:dev Apr 2, 2024
13 of 16 checks passed
@bharathv bharathv deleted the ubsan_ducktape branch April 2, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotnwat dotnwat dotnwat approved these changes

@savex savex Awaiting requested review from savex savex is a code owner automatically assigned from redpanda-data/devprod

@ballard26 ballard26 Awaiting requested review from ballard26

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@bharathv @vbotbuildovich @dotnwat @ballard26 @piyushredpanda