Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sys_procexit_e has more then 1M instruction on GKE (COS) #1639

Closed
albe19029 opened this issue Jan 23, 2024 · 3 comments · Fixed by #1642
Closed

sys_procexit_e has more then 1M instruction on GKE (COS) #1639

albe19029 opened this issue Jan 23, 2024 · 3 comments · Fixed by #1642
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@albe19029
Copy link

Good day. Have found next issue starting from sysdig 0.33.1 version.
When I try to run sysdig on GKE cluster I get next error:

processed 40396 insns (limit 1000000) max_states_per_insn 1 total_states 4057 peak_states 4057 mark_read 73
-- END PROG LOAD LOG --
libscap: bpf_load_program() event=raw_tracepoint/filler/sys_procexit_e: Operation not permitted

I found our that this commit leads to the problem. 1e06bd3

I don't know why this leads only to problems on COS (also checked AWS and Azure - no problems running there), but only managed to run sysdig if I change values of MAX_THREADS_GROUPS and MAX_HIERARCHY_TRAVERSE to this one:

#define MAX_THREADS_GROUPS 25
#define MAX_HIERARCHY_TRAVERSE 35

Is it possible to investigate why on COS this limit is lower then on other Linux Distros. Or adopt values to make it possible to run sysdig starting from 0.33.1 also on GKE, as now it is broken.

Have tested values both on x64 and arm64 clusters. Be aware, that to run sysdig on arm64 this should be fixed also (draios/sysdig#2057 - this was original ticket with the problem)

To run sysdig on GKE I use next yaml file
scap.txt

kubectl apply -f scap.yaml

And then attach to pod:
kubectl exec --stdin --tty sysdig-0341 -- /bin/bash

And run sysdig

@Andreagit97
Copy link
Member

thank you for reporting! uhm this doesn't seem an issue with the number of instruction:

processed 40396 insns (limit 1000000) max_states_per_insn 1 total_states 4057 peak_states 4057 mark_read 73

as you can see we didn't overcome the limit of 1000000. BTW you are not the first one who reported us this kind of issue on GKE ... as a fix, I think that we can decrease the 2 macros (MAX_THREADS_GROUPS MAX_HIERARCHY_TRAVERSE) as you suggested!

@Andreagit97 Andreagit97 self-assigned this Jan 24, 2024
@Andreagit97 Andreagit97 added this to the 0.15.0 milestone Jan 24, 2024
@Andreagit97
Copy link
Member

BTW please note that this won't be fixed in sysdig 0.33.1. it will be fixed only in sysdig versions that will be based on the new patched libs versions

@albe19029
Copy link
Author

albe19029 commented Jan 25, 2024

Thanks for fix, will be wait for next sysdig release. We are migrating to version 0.34.1 now, so no problems. Hope new release with this fix will coming soon, as without it - GKE users won't be happy by this update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants