Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

messaging_service: Do TLS init early #14493

Closed
wants to merge 1 commit into from
Closed

messaging_service: Do TLS init early #14493

wants to merge 1 commit into from

Conversation

elcallio
Copy link
Contributor

@elcallio elcallio commented Jul 3, 2023

Fixes #14299

failure_detector can try sending messages to TLS endpoints before start_listen has been called (why?). Need TLS initialized before this. So do on service creation.

@mykaul mykaul requested a review from kbr-scylla July 3, 2023 16:08
@scylladb-promoter
Copy link
Contributor

CI state FAILURE - https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2204/

@scylladb-promoter
Copy link
Contributor

CI state ABORTED - https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2228/

@tchaikov tchaikov mentioned this pull request Jul 4, 2023
Closed
2 tasks
@mykaul
Copy link
Contributor

mykaul commented Jul 4, 2023

CI failed with a backtrace - why don't we decode it already (especially as it's in debug) - @yaronkaikov ?

const: Assertion `eq(co_await call(ExReg::exchange{i}, 100_t), ExReg::ret{i - 1})' failed.
 Aborting on shard 0.
 Backtrace:
   0xbca590
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x48197bc
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4819413
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x466bae9
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x469ae85
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x478c669
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x478c89e
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x478c6ea
   /lib64/libc.so.6+0x3cb1f
   /lib64/libc.so.6+0x8ce5b
   /lib64/libc.so.6+0x3ca75
   /lib64/libc.so.6+0x267fb
   /lib64/libc.so.6+0x2671a
   /lib64/libc.so.6+0x35655
   0xca8ce2
   0xcead16
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x468d278
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x469534e
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4699390
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4696fb6
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x42186c0
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4215f5a
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc1dce
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc16e0
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc15d0
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc103c
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4465738
   /lib64/libc.so.6+0x8b12c
   /lib64/libc.so.6+0x10cbbf
 === TEST.PY SUMMARY START ===

@scylladb-promoter
Copy link
Contributor

CI state SUCCESS - https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2232/

@elcallio
Copy link
Contributor Author

elcallio commented Jul 4, 2023

Is there any way to access the scylla exec used in the above stacktrace? To decode it...

@kbr-scylla
Copy link
Contributor

This was an assertion failure, we don't need any info from this particular backtrace.
This says everything:

const: Assertion `eq(co_await call(ExReg::exchange{i}, 100_t), ExReg::ret{i - 1})' failed.

it looks unrelated to the PR, a flaky test (although I'm seeing this for the first time). This test is not using messaging_service.

@kbr-scylla
Copy link
Contributor

I opened an issue: #14500

@elcallio
Copy link
Contributor Author

elcallio commented Jul 4, 2023

True, and the assert is in test code, so less interesting; But it is in a callback, and there are seastar frames in the trace. While they will in this case obviously be just future dispatch, in a more general sense it would be nice to quickly be able to get a decoded trace... As a general point...

@kbr-scylla
Copy link
Contributor

Agreed.
Build artifacts on the Jenkins job page contain this file
https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2228/artifact/00-Build.txt
You should be able to download the image or use http://backtrace.scylladb.com/index.html with the BuildID

@yaronkaikov
Copy link
Contributor

CI failed with a backtrace - why don't we decode it already (especially as it's in debug) - @yaronkaikov ?

const: Assertion `eq(co_await call(ExReg::exchange{i}, 100_t), ExReg::ret{i - 1})' failed.
 Aborting on shard 0.
 Backtrace:
   0xbca590
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x48197bc
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4819413
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x466bae9
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x469ae85
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x478c669
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x478c89e
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x478c6ea
   /lib64/libc.so.6+0x3cb1f
   /lib64/libc.so.6+0x8ce5b
   /lib64/libc.so.6+0x3ca75
   /lib64/libc.so.6+0x267fb
   /lib64/libc.so.6+0x2671a
   /lib64/libc.so.6+0x35655
   0xca8ce2
   0xcead16
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x468d278
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x469534e
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4699390
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4696fb6
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x42186c0
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4215f5a
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc1dce
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc16e0
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc15d0
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar_testing.so+0xc103c
   /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debug/seastar/libseastar.so+0x4465738
   /lib64/libc.so.6+0x8b12c
   /lib64/libc.so.6+0x10cbbf
 === TEST.PY SUMMARY START ===

I guess it's because it was never required. How should we decode it? only for debug mode?

@michoecho
Copy link
Contributor

Is there any way to access the scylla exec used in the above stacktrace? To decode it...

It wasn't a Scylla exec, but a test exec.

And to get failing test executables you navigate to (in this case) https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2228/artifact/scylla/build/x86_64/debug/FailingtestExe/randomized_nemesis_test. (You can get there from https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2228/artifact by clicking links).

But seastar and seastar_testing are shared libraries in debug, so to decode the seastar parts of the backtrace you would have to download them separately from the packages linked at https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2228/artifact/00-Build.txt, and adjust the paths in the backtrace to point at your downloaded libraries...

in a more general sense it would be nice to quickly be able to get a decoded trace... As a general point...

backtrace.scylladb.com is the best we have. But it won't work for tests.

@DoronArazii
Copy link

DoronArazii commented Jul 6, 2023 •
edited

@xemul @tgrabiec we need review here for moving it forward

@DoronArazii DoronArazii requested a review from xemul July 6, 2023 11:03
xemul
xemul reviewed Jul 6, 2023
View reviewed changes
message/messaging_service.hh Outdated Show resolved Hide resolved
@DoronArazii
Copy link

@avikivity please assist with reviewing this one (talked in our meeting).

@avikivity
Copy link
Member

@avikivity please assist with reviewing this one (talked in our meeting).

It has open review comments.

messaging_service: Do TLS init early
7a8aac5 
Fixes scylladb#14299

failure_detector can try sending messages to TLS endpoints before start_listen
has been called (why?). Need TLS initialized before this. So do on service creation.
@scylladb-promoter
Copy link
Contributor

CI state SUCCESS - https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2358/

@mykaul
Copy link
Contributor

mykaul commented Jul 10, 2023

@avikivity - is there anything still open, or can it be reviewed?

@elcallio
Copy link
Contributor Author

ping?

@avikivity
Copy link
Member

@kbr-scylla this is what I referred to re TLS / failure detector causing crashes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nyh nyh nyh left review comments

@xemul xemul xemul approved these changes

@kbr-scylla kbr-scylla kbr-scylla approved these changes

@avikivity avikivity Awaiting requested review from avikivity

@tgrabiec tgrabiec Awaiting requested review from tgrabiec

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Segmentation fault on node configured to use client encryption
10 participants
@elcallio @scylladb-promoter @mykaul @kbr-scylla @yaronkaikov @michoecho @DoronArazii @avikivity @nyh @xemul