New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sysdig on Proxmox - sysdig-probe fails to load #415

Closed
rmoff opened this Issue Aug 14, 2015 · 10 comments

Comments

Projects
None yet
5 participants
@rmoff

rmoff commented Aug 14, 2015

I'm trying to run sysdig on Proxmox, and the sysdig-probe fails to load.
Diagnostics:

root@proxmox04:/tmp# pveversion -v
proxmox-ve-2.6.32: 3.4-156 (running kernel: 2.6.32-39-pve)
pve-manager: 3.4-9 (running version: 3.4-9/4b51d87a)
pve-kernel-2.6.32-39-pve: 2.6.32-157
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-18
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
root@proxmox04:/tmp# uname -r
2.6.32-39-pve
root@proxmox04:/tmp# ls -l /lib/modules/$(uname -r)
total 2368
lrwxrwxrwx  1 root root     36 Aug 14 09:03 build -> /usr/src/linux-headers-2.6.32-39-pve
drwxr-xr-x 12 root root   4096 Jun  5 11:28 kernel
-rw-r--r--  1 root root 605998 May  8 10:40 modules.alias
-rw-r--r--  1 root root 580612 May  8 10:40 modules.alias.bin
-rw-r--r--  1 root root   5665 May  8 10:40 modules.builtin
-rw-r--r--  1 root root   7318 May  8 10:40 modules.builtin.bin
-rw-r--r--  1 root root 205635 May  8 10:40 modules.dep
-rw-r--r--  1 root root 298611 May  8 10:40 modules.dep.bin
-rw-r--r--  1 root root     80 May  8 10:40 modules.devname
-rw-r--r--  1 root root  77081 May  8 10:40 modules.order
-rw-r--r--  1 root root    131 May  8 10:40 modules.softdep
-rw-r--r--  1 root root 273187 May  8 10:40 modules.symbols
-rw-r--r--  1 root root 342206 May  8 10:40 modules.symbols.bin
drwxr-xr-x  2 root root   4096 Jun  5 11:28 zfs
root@proxmox04:/tmp# sysdig-probe-loader
* Unloading sysdig-probe, if present
Error: Module sysdig_probe is not currently loaded
* Running dkms autoinstall

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
make KERNELRELEASE=2.6.32-39-pve -C /lib/modules/2.6.32-39-pve/build M=/var/lib/dkms/sysdig/0.1.102/build....(bad exit status: 2)
Error! Bad return status for module build on kernel: 2.6.32-39-pve (x86_64)
Consult /var/lib/dkms/sysdig/0.1.102/build/make.log for more information.
* Trying to load a system sysdig-probe, if present
* Trying to load a dkms sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 2.6.32-39-pve
Found kernel config at /boot/config-2.6.32-39-pve
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.1.102-x86_64-2.6.32-39-pve-f9c010e053612af9f0cf3856537bef42.ko
Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
root@proxmox04:/tmp#
root@proxmox04:/tmp# cat /var/lib/dkms/sysdig/0.1.102/build/make.log
DKMS make.log for sysdig-0.1.102 for kernel 2.6.32-39-pve (x86_64)
Fri Aug 14 09:10:34 BST 2015
expr: syntax error
make: Entering directory `/usr/src/linux-headers-2.6.32-39-pve'
  LD      /var/lib/dkms/sysdig/0.1.102/build/built-in.o
  CC [M]  /var/lib/dkms/sysdig/0.1.102/build/main.o
/var/lib/dkms/sysdig/0.1.102/build/main.c: In function ‘ppm_ioctl’:
/var/lib/dkms/sysdig/0.1.102/build/main.c:782:3: error: implicit declaration of function ‘for_each_process’ [-Werror=implicit-function-declaration]
/var/lib/dkms/sysdig/0.1.102/build/main.c:782:23: error: expected ‘;’ before ‘{’ token
/var/lib/dkms/sysdig/0.1.102/build/main.c:751:27: warning: unused variable ‘t’ [-Wunused-variable]
cc1: some warnings being treated as errors
make[1]: *** [/var/lib/dkms/sysdig/0.1.102/build/main.o] Error 1
make: *** [_module_/var/lib/dkms/sysdig/0.1.102/build] Error 2
make: Leaving directory `/usr/src/linux-headers-2.6.32-39-pve'
root@proxmox04:/tmp#

I installed the kernel headers using apt-get install pve-headers-2.6.32-39-pve after apt-get -y install linux-headers-$(uname -r) failed (Unable to locate package linux-headers-2.6.32-39-pve)

What can I do to get this to work? Thanks!

@ton31337

This comment has been minimized.

Show comment
Hide comment
@ton31337

ton31337 Aug 14, 2015

Contributor

Looks like you don't have headers:

implicit declaration of function ‘for_each_process’

What does aptitute search linux-headers-* return?

Contributor

ton31337 commented Aug 14, 2015

Looks like you don't have headers:

implicit declaration of function ‘for_each_process’

What does aptitute search linux-headers-* return?

@rmoff

This comment has been minimized.

Show comment
Hide comment
@rmoff

rmoff Aug 14, 2015

root@proxmox04:/tmp# aptitude search linux-headers-*
v   linux-headers                                                                      -
v   linux-headers-2.6                                                                  -
p   linux-headers-2.6-amd64                                                            - Header files for Linux amd64 configuration (dummy package)
p   linux-headers-3.2.0-4-all                                                          - All header files for Linux 3.2 (meta-package)
p   linux-headers-3.2.0-4-all-amd64                                                    - All header files for Linux 3.2 (meta-package)
i A linux-headers-3.2.0-4-amd64                                                        - Header files for Linux 3.2.0-4-amd64
i A linux-headers-3.2.0-4-common                                                       - Common header files for Linux 3.2.0-4
p   linux-headers-3.2.0-4-common-rt                                                    - Common header files for Linux 3.2.0-4-rt
p   linux-headers-3.2.0-4-rt-amd64                                                     - Header files for Linux 3.2.0-4-rt-amd64
i A linux-headers-amd64                                                                - Header files for Linux amd64 configuration (meta-package)
p   linux-headers-rt-amd64                                                             - Header files for Linux rt-amd64 configuration (meta-package)

rmoff commented Aug 14, 2015

root@proxmox04:/tmp# aptitude search linux-headers-*
v   linux-headers                                                                      -
v   linux-headers-2.6                                                                  -
p   linux-headers-2.6-amd64                                                            - Header files for Linux amd64 configuration (dummy package)
p   linux-headers-3.2.0-4-all                                                          - All header files for Linux 3.2 (meta-package)
p   linux-headers-3.2.0-4-all-amd64                                                    - All header files for Linux 3.2 (meta-package)
i A linux-headers-3.2.0-4-amd64                                                        - Header files for Linux 3.2.0-4-amd64
i A linux-headers-3.2.0-4-common                                                       - Common header files for Linux 3.2.0-4
p   linux-headers-3.2.0-4-common-rt                                                    - Common header files for Linux 3.2.0-4-rt
p   linux-headers-3.2.0-4-rt-amd64                                                     - Header files for Linux 3.2.0-4-rt-amd64
i A linux-headers-amd64                                                                - Header files for Linux amd64 configuration (meta-package)
p   linux-headers-rt-amd64                                                             - Header files for Linux rt-amd64 configuration (meta-package)
@ton31337

This comment has been minimized.

Show comment
Hide comment
@ton31337

ton31337 Aug 14, 2015

Contributor

Try to install aptitude install linux-headers linux-headers-2.6

Contributor

ton31337 commented Aug 14, 2015

Try to install aptitude install linux-headers linux-headers-2.6

@rmoff

This comment has been minimized.

Show comment
Hide comment
@rmoff

rmoff Aug 14, 2015

No joy. After doing this:

root@proxmox04:~# aptitude install linux-headers linux-headers-2.6
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.

I still get the same make/build error as above.

rmoff commented Aug 14, 2015

No joy. After doing this:

root@proxmox04:~# aptitude install linux-headers linux-headers-2.6
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.

I still get the same make/build error as above.

@ton31337

This comment has been minimized.

Show comment
Hide comment
@ton31337

ton31337 Aug 14, 2015

Contributor

strange, did you do aptitude update before?

Contributor

ton31337 commented Aug 14, 2015

strange, did you do aptitude update before?

@rmoff

This comment has been minimized.

Show comment
Hide comment
@rmoff

rmoff Aug 14, 2015

Yep.
I also tried installing pve-headers-2.6.32-39-pve but that didn't help.

root@proxmox04:~# aptitude search pve-headers*
p   pve-headers-2.6.32-23-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-24-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-25-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-26-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-27-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-28-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-29-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-30-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-31-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-32-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-33-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-34-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-37-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-38-pve                          - The Proxmox PVE Kernel Headers
i   pve-headers-2.6.32-39-pve                          - The Proxmox PVE Kernel Headers
i   pve-headers-2.6.32-40-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-1-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-10-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-11-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-2-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-3-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-4-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-5-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-7-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-8-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-9-pve                           - The Proxmox PVE Kernel Headers

NB I updated the kernel, in case that was part of the problem.

root@proxmox04:~# uname -r
2.6.32-40-pve

rmoff commented Aug 14, 2015

Yep.
I also tried installing pve-headers-2.6.32-39-pve but that didn't help.

root@proxmox04:~# aptitude search pve-headers*
p   pve-headers-2.6.32-23-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-24-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-25-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-26-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-27-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-28-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-29-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-30-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-31-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-32-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-33-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-34-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-37-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-2.6.32-38-pve                          - The Proxmox PVE Kernel Headers
i   pve-headers-2.6.32-39-pve                          - The Proxmox PVE Kernel Headers
i   pve-headers-2.6.32-40-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-1-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-10-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-11-pve                          - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-2-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-3-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-4-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-5-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-7-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-8-pve                           - The Proxmox PVE Kernel Headers
p   pve-headers-3.10.0-9-pve                           - The Proxmox PVE Kernel Headers

NB I updated the kernel, in case that was part of the problem.

root@proxmox04:~# uname -r
2.6.32-40-pve
@gianlucaborello

This comment has been minimized.

Show comment
Hide comment
@gianlucaborello

gianlucaborello Aug 14, 2015

Member

The problem is not related to the missing kernel headers. If they were missing the compilation wouldn't even start. The problem is that your version of the kernel is patched in a way that doesn't provide the for_each_process function. We'll take a look but it might take a while since this is probably low priority as it is definitely not a mainstream distribution.

Thanks

Member

gianlucaborello commented Aug 14, 2015

The problem is not related to the missing kernel headers. If they were missing the compilation wouldn't even start. The problem is that your version of the kernel is patched in a way that doesn't provide the for_each_process function. We'll take a look but it might take a while since this is probably low priority as it is definitely not a mainstream distribution.

Thanks

@rmoff

This comment has been minimized.

Show comment
Hide comment
@rmoff

rmoff commented Aug 14, 2015

@gnosek

This comment has been minimized.

Show comment
Hide comment
@gnosek

gnosek Nov 13, 2015

Contributor

I hit the same issue on CloudLinux 6.7 (basically patched CentOS with probably-OpenVZ and some other bells and whistles). AIUI, CentOS 6 is supported, so I'd expect to have the for_each_process macro on CloudLinux too. Oh well.

Contributor

gnosek commented Nov 13, 2015

I hit the same issue on CloudLinux 6.7 (basically patched CentOS with probably-OpenVZ and some other bells and whistles). AIUI, CentOS 6 is supported, so I'd expect to have the for_each_process macro on CloudLinux too. Oh well.

@jfchevrette

This comment has been minimized.

Show comment
Hide comment
@jfchevrette

jfchevrette Jan 20, 2016

It looks like the Proxmox kernel is based off OpenVZ's kernel, which is also the case for Cloudlinux kernel.

The devs at Cloudlinux suggested the following to workaround as it appears the macros used by sysdig have been renamed in OpenVZ's kernel.

cd /var/lib/dkms/sysdig/0.6.0/source/
sed -i 's/for_each_process/for_each_process_all/g' *.[ch]
sed -i 's/while_each_thread/while_each_thread_all/g' *.[ch]
sed -i 's/do_each_thread/do_each_thread_all/g' *.[ch]

Then run the following to build and install the module

dkms build -m sysdig -v 0.6.0
dkms install --force -m sysdig -v 0.6.0

Worked for me and I can start sysdig/csysdig and trace stuff. YMMV.

Any chances the sysdig folks can see if and how this could be handled in sysdig's source?

jfchevrette commented Jan 20, 2016

It looks like the Proxmox kernel is based off OpenVZ's kernel, which is also the case for Cloudlinux kernel.

The devs at Cloudlinux suggested the following to workaround as it appears the macros used by sysdig have been renamed in OpenVZ's kernel.

cd /var/lib/dkms/sysdig/0.6.0/source/
sed -i 's/for_each_process/for_each_process_all/g' *.[ch]
sed -i 's/while_each_thread/while_each_thread_all/g' *.[ch]
sed -i 's/do_each_thread/do_each_thread_all/g' *.[ch]

Then run the following to build and install the module

dkms build -m sysdig -v 0.6.0
dkms install --force -m sysdig -v 0.6.0

Worked for me and I can start sysdig/csysdig and trace stuff. YMMV.

Any chances the sysdig folks can see if and how this could be handled in sysdig's source?

jamesbrink added a commit to jamesbrink/sysdig that referenced this issue May 7, 2016

Update to support CloudLinux,Proxmox,OpenVZ
This update adds support to a handful of Linux distributions running
modified kernels, such as CloudLinux,Proxmox,OpenVZ (to name a few).
This fixes #415, and fixes #568

sysdig-CLA-1.0-signed-off-by: James Brink brink.james@gmail.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment