Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV in destination controller #11162

Closed
bjoernw opened this issue Jul 24, 2023 · 4 comments
Closed

SIGSEGV in destination controller #11162

bjoernw opened this issue Jul 24, 2023 · 4 comments
Labels
bug

Comments

@bjoernw
Copy link

bjoernw commented Jul 24, 2023

What is the issue?

Sorry for the poor log exception parsing but I'm seeing the following issue in our destination controller pods. It resolves itself via an automatic restart.
Screenshot 2023-07-24 at 11 24 31 AM

Around this time we're also seeing Ser

How can it be reproduced?

Not sure, seeing this randomly.

Logs, error output, etc

See screenshot

output of linkerd check -o short

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2023-07-26T13:32:12Z
    see https://linkerd.io/2.13/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ cli is up-to-date
    is running version 2.13.4 but the latest stable version is 2.13.5
    see https://linkerd.io/2.13/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 2.13.4 but the latest stable version is 2.13.5
    see https://linkerd.io/2.13/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-6cfbff8455-9m5qj (stable-2.13.4)
	* linkerd-destination-6cfbff8455-gg6f5 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-gxpf6 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-kpzvx (stable-2.13.4)
	* linkerd-destination-6cfbff8455-m26r7 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-rzckz (stable-2.13.4)
	* linkerd-destination-6cfbff8455-sktd4 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-t6hrt (stable-2.13.4)
	* linkerd-destination-6cfbff8455-v2wqv (stable-2.13.4)
	* linkerd-destination-6cfbff8455-xr9ct (stable-2.13.4)
	* linkerd-identity-7d75c98558-2njhq (stable-2.13.4)
	* linkerd-identity-7d75c98558-5wcw8 (stable-2.13.4)
	* linkerd-identity-7d75c98558-b45vn (stable-2.13.4)
	* linkerd-identity-7d75c98558-f5mj9 (stable-2.13.4)
	* linkerd-identity-7d75c98558-j985m (stable-2.13.4)
	* linkerd-identity-7d75c98558-k9dgr (stable-2.13.4)
	* linkerd-identity-7d75c98558-ndlnc (stable-2.13.4)
	* linkerd-identity-7d75c98558-nj2bl (stable-2.13.4)
	* linkerd-identity-7d75c98558-phf5h (stable-2.13.4)
	* linkerd-identity-7d75c98558-stb9m (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-2dtqn (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-6rg92 (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-8j9pf (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-bhhzx (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-ck59g (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-l7l5x (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-phrxj (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-qp59l (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-sb55j (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-w9f7n (stable-2.13.4)
	* metrics-api-6b7d76799f-mqgv5 (stable-2.13.4)
	* prometheus-746b879d6b-hmhcw (stable-2.13.4)
	* smi-adaptor-6c9cb547b5-qj8lr (stable-2.13.4)
	* tap-844b4cd55c-p2lgq (stable-2.13.4)
	* tap-injector-8b7d8f856-nsczh (stable-2.13.4)
	* web-664cbdc7b5-4zxfp (stable-2.13.4)
    see https://linkerd.io/2.13/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    prometheus-5d6ffd95df-g8jkc running stable-2.12.3 but cli running stable-2.13.4
    see https://linkerd.io/2.13/checks/#l5d-cp-proxy-cli-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-6cfbff8455-9m5qj (stable-2.13.4)
	* linkerd-destination-6cfbff8455-gg6f5 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-gxpf6 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-kpzvx (stable-2.13.4)
	* linkerd-destination-6cfbff8455-m26r7 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-rzckz (stable-2.13.4)
	* linkerd-destination-6cfbff8455-sktd4 (stable-2.13.4)
	* linkerd-destination-6cfbff8455-t6hrt (stable-2.13.4)
	* linkerd-destination-6cfbff8455-v2wqv (stable-2.13.4)
	* linkerd-destination-6cfbff8455-xr9ct (stable-2.13.4)
	* linkerd-identity-7d75c98558-2njhq (stable-2.13.4)
	* linkerd-identity-7d75c98558-5wcw8 (stable-2.13.4)
	* linkerd-identity-7d75c98558-b45vn (stable-2.13.4)
	* linkerd-identity-7d75c98558-f5mj9 (stable-2.13.4)
	* linkerd-identity-7d75c98558-j985m (stable-2.13.4)
	* linkerd-identity-7d75c98558-k9dgr (stable-2.13.4)
	* linkerd-identity-7d75c98558-ndlnc (stable-2.13.4)
	* linkerd-identity-7d75c98558-nj2bl (stable-2.13.4)
	* linkerd-identity-7d75c98558-phf5h (stable-2.13.4)
	* linkerd-identity-7d75c98558-stb9m (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-2dtqn (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-6rg92 (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-8j9pf (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-bhhzx (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-ck59g (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-l7l5x (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-phrxj (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-qp59l (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-sb55j (stable-2.13.4)
	* linkerd-proxy-injector-77dcdd7dcb-w9f7n (stable-2.13.4)
	* metrics-api-6b7d76799f-mqgv5 (stable-2.13.4)
	* prometheus-746b879d6b-hmhcw (stable-2.13.4)
	* smi-adaptor-6c9cb547b5-qj8lr (stable-2.13.4)
	* tap-844b4cd55c-p2lgq (stable-2.13.4)
	* tap-injector-8b7d8f856-nsczh (stable-2.13.4)
	* web-664cbdc7b5-4zxfp (stable-2.13.4)
    see https://linkerd.io/2.13/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    prometheus-5d6ffd95df-g8jkc running stable-2.12.3 but cli running stable-2.13.4
    see https://linkerd.io/2.13/checks/#l5d-viz-proxy-cli-version for hints
× viz extension self-check
    Error calling Prometheus from the control plane: Query failed: "max(process_start_time_seconds{}) by (pod, namespace)": server_error: server error: 504
    see https://linkerd.io/2.13/checks/#l5d-viz-metrics-api for hints

(ignore the prometheus issue, it's crashlooping due to load right now)

Environment

  • K8S: v1.24.15-eks
  • Linkerd: 2.13.4
  • AMI: amazon

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None
@bjoernw bjoernw added the bug label Jul 24, 2023
@alpeb
Copy link
Member

alpeb commented Jul 24, 2023

Thanks for the report; this was also reported in #11135. I've raised #11163 detailing the likely cause. Will try to get the fix prioritized.

@bjoernw
Copy link
Author

bjoernw commented Jul 24, 2023

thank you!

@bjoernw
Copy link
Author

bjoernw commented Jul 25, 2023

@alpeb We had a 15min prod outage today due to the destination-controller cache having outdated pods and hence triggering failfast mode all throughout the mesh. I saw several of the SIGSEGVs log patterns I reported above.

@alpeb
Copy link
Member

alpeb commented Aug 2, 2023

The fix will make it into 2.13.6, coming out most likely next week.

@olix0r olix0r closed this as completed Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@olix0r @alpeb @bjoernw