Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pyperf profiling not working with pyston environment #3125

Closed
keyolk opened this issue Mar 21, 2024 · 0 comments · Fixed by #3126
Closed

Pyperf profiling not working with pyston environment #3125

keyolk opened this issue Mar 21, 2024 · 0 comments · Fixed by #3126

Comments

@keyolk
Copy link
Contributor

keyolk commented Mar 21, 2024

Describe the bug

Trying pyperf ebpf profiling, it is failed on my uwsgi + pyston environment.
Here some error logs

ts=2024-03-15T07:40:08.003160632Z level=info msg="pyperf loaded" component=pyroscope.ebpf.local
ts=2024-03-15T07:40:08.137502618Z level=error msg="pyperf get python process data failed" component=pyroscope.ebpf.local err="GetPythonProcInfo error /proc/4740/maps: no python found" pid=4740 target="{__container_id__=\"b16d855b5f2fa6a82a52844d8779b6ffe207d9dc4bd19677c334c403d0e4730b\", __name__=\"process_cpu\", service_name=\"/api\"}"
-var rePython = regexp.MustCompile("/.*/((?:lib)?python)(\\d+)\\.(\\d+)(?:[mu]?(?:\\.so)?)?(?:.1.0)?$")
+var rePython = regexp.MustCompile("/.*/((?:lib)?python)(\\d+)\\.(\\d+)(?:[mu]?(-pyston\\d.\\d)?(?:\\.so)?)?(?:.1.0)?$")

After some updates on its regex https://github.com/grafana/pyroscope/blob/main/ebpf/python/procinfo.go#L21
Faced another error

ts=2024-03-20T12:45:11.312965197Z level=error msg="pyperf get python process data failed" component=pyroscope.ebpf.local err="failed to get python process libc could not determine libc version 4531, no libc found" pid=4531 target="{__container_id__=\"6f9c5812397d2f59745ba855090c9cda52b675cbeec70948e126633111b29eb7\", __name__=\"process_cpu\", service_name=\"/api\"}"

And next I tried with the diff to https://github.com/grafana/pyroscope/blob/main/ebpf/python/procinfo.go#L61

-			if strings.HasSuffix(m.Pathname, "/libc.so.6") {
+			if strings.HasSuffix(m.Pathname, "/libc.so.6") || strings.HasSuffix(m.Pathname, "/libc-2") {

Not sure if the approach is OK.

To Reproduce

Steps to reproduce the behavior:

using flow mode

$ cat <<EOF > /etc/grafana.river
logging {
  level  = "info"
  format = "logfmt"
}

discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}
  

discovery.relabel "local_containers" {
  targets = discovery.docker.linux.targets
  rule {
    action = "replace"
    source_labels = ["__meta_docker_container_name"]
    target_label = "service_name"
  }
}

pyroscope.ebpf "local" {
  targets    = discovery.relabel.local_containers.output
  forward_to = [
    pyroscope.write.central.receiver,
  ]
}

pyroscope.write "central" {
  endpoint {
    ...  
  }
}
EOF

$ docker run -e AGENT_MODE=flow --pid=host --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /etc/grafana.river:/etc/grafana.river -ti grafana/agent /etc/grafana.river

Expected behavior

No issue on profiling

Environment

  • Infrastructure: AWS ec2
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
  • Deployment tool: docker
docker version
Client: Docker Engine - Community
 Version:           25.0.4
 API version:       1.41 (downgraded from 1.44)
 Go version:        go1.21.8
 Git commit:        1a576c5
 Built:             Wed Mar  6 16:32:16 2024
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Additional Context

I'm using pyston based uWSGI.
Wrote a PR to fix this #3126

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant