feat(ci): verify libcontainer.so links and loads on glibc < 2.34 - #1513
Conversation
Issue falcosecurity#1500 reached a release because no CI job both built the plugin against an old glibc and loaded the result. build-linux builds inside debian:bullseye but never dlopens what it produced, and falco-tests dlopens inside falcosecurity/falco:master-debian, which is Debian 12 with glibc 2.36, where a missing DT_NEEDED on libresolv.so.2 cannot fail because those symbols moved into libc in glibc 2.34. Add both checks to build-linux, which already runs on bullseye (glibc 2.31), so neither needs a new job, container or matrix entry and both run for amd64 and arm64. The first asserts the DT_NEEDED entry is present, which fails fast and points at the cause. The second compiles a small dlopen harness and loads the built library with RTLD_NOW, which is the check that would have caught falcosecurity#1500 and will also catch a different symbol going missing on old glibc for an unrelated reason. Fixes falcosecurity#1512 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Manny Castillo <manuel.franklin.castillo@gmail.com>
leogr
left a comment
There was a problem hiding this comment.
Hey @ManuelFCastillo
Thanks for this! Putting the checks in build-linux is the right call, since that job already runs on bullseye and covers arm64 too. The readelf guard is correct, and your analysis of why CI missed #1500 matches what I see: falco-tests runs in falcosecurity/falco:master-debian, which is Debian 12 with glibc 2.36.
However, I ran the two new steps against the real libcontainer-amd64 artifact built from main (https://github.com/falcosecurity/plugins/actions/runs/33775589712) inside debian:bullseye. The readelf step passes, but the dlopen step fails with undefined symbol: pthread_mutex_trylock. As is, this would turn the next plugins/container/** PR red. Details and a minimal fix in the inline comments.
N.B. The workflow did not run on this PR, since it only triggers on plugins/container/** changes. That is why the checks here are green.
Thanks again 🙏
| return 0; | ||
| } | ||
| EOF | ||
| gcc -o /tmp/dlopen_check /tmp/dlopen_check.c -ldl |
There was a problem hiding this comment.
| gcc -o /tmp/dlopen_check /tmp/dlopen_check.c -ldl | |
| gcc -o /tmp/dlopen_check /tmp/dlopen_check.c -Wl,--no-as-needed -lpthread -ldl |
I ran this step against the real libcontainer-amd64 artifact from https://github.com/falcosecurity/plugins/actions/runs/33775589712 inside debian:bullseye (glibc 2.31), and it fails:
dlopen failed: ./libcontainer.so: undefined symbol: pthread_mutex_trylock
The .so references pthread_* and dl* symbols (I believe they come from the Go runtime in libworker.a), but its only DT_NEEDED entries are libresolv.so.2, libc.so.6, and the loader. On glibc < 2.34 those symbols live in libpthread.so.0 and libdl.so.2, so a bare probe cannot resolve them. It works when Falco loads the plugin since the Falco binary itself links libpthread and libdl (via libsinsp), and the plugin resolves them from the global scope. That is also why #1500 only surfaced __res_search: Falco does not link libresolv.
Linking the probe with -Wl,--no-as-needed -lpthread -ldl makes it load the plugin the way a Falco process does. The --no-as-needed part is needed since bullseye's gcc defaults to --as-needed, which drops libpthread from the probe (that is why a plain -pthread does not help). I verified that with this change the step passes on the same artifact, and it still fails on a .so calling res_search without -lresolv, so the #1500 case is still caught. RTLD_NOW can stay.
The proper fix would be linking libpthread and libdl explicitly in the plugin (e.g. Threads::Threads and ${CMAKE_DL_LIBS} next to resolv in go-worker.cmake), so the .so is self-contained and the strict probe can stay as is. I did a quick simulation with a static archive and the resulting .so gets its own DT_NEEDED entries and loads with the strict probe, but I haven't rebuilt the actual plugin. However, that is a plugin change, so I'd keep it for a separate PR.
There was a problem hiding this comment.
Applied, thanks. Good catch: I had only tested the guard against a .so built to be missing -lresolv, so I confirmed it caught the bug and never confirmed it passed a healthy build
Verified against the libcontainer-amd64 artifact from the run you linked, inside debian:bullseye: fails with undefined symbol: pthread_mutex_trylock as you described, loads cleanly with -Wl,--no-as-needed -lpthread -ldl, and still rejects a .so calling res_search without -lresolv
| # Issue #1500 shipped because no job both built against an old glibc and | ||
| # loaded the result: this one builds on bullseye but never dlopen'd it, | ||
| # and falco-tests dlopens inside Debian 12 (glibc 2.36), where a missing | ||
| # DT_NEEDED on libresolv cannot fail. This job is already on bullseye | ||
| # (glibc 2.31), so both checks cost nothing but the steps themselves. |
There was a problem hiding this comment.
Nit (non-blocking). Since the workflow only triggers on plugins/container/**, the two new steps did not run on this PR (the green checks come from ci.yaml and CodeQL). May you add the workflow file itself to both paths lists (pull_request and push), so that changes like this one exercise themselves? 👇
paths:
- "plugins/container/**"
- ".github/workflows/container-ci.yaml"That should also let the dlopen step run right here once the probe is fixed.
There was a problem hiding this comment.
Added to both pull_request and push. build-linux ran here as a result, and both new steps passed on amd64 and arm64
…rkflow The probe was linked with -ldl only, which fails on the real plugin: dlopen failed: ./libcontainer.so: undefined symbol: pthread_mutex_trylock libcontainer.so pulls pthread_* and dl* from the Go runtime but declares no DT_NEEDED for them; its only entries are libresolv.so.2, libc.so.6 and the loader. Under Falco those resolve from the global scope because the Falco binary links both via libsinsp, so a bare probe is stricter than the real loading environment and would have failed every healthy build. Link the probe with -Wl,--no-as-needed -lpthread -ldl so it loads the plugin the way a Falco process does. --no-as-needed is required because bullseye's gcc defaults to --as-needed and drops libpthread, which the probe's own code never calls. Verified by running the two steps verbatim inside debian:bullseye against the libcontainer-amd64 artifact from the main branch: both guards pass. Against a shared object calling res_search without -lresolv they still fail, so falcosecurity#1500 remains covered. Also adds the workflow to its own pull_request and push paths, so changes to these steps exercise themselves rather than reporting green from unrelated jobs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Manny Castillo <manuel.franklin.castillo@gmail.com>
leogr
left a comment
There was a problem hiding this comment.
LGTM, thanks for the quick turnaround! Both guards now pass against the real artifact on amd64 and arm64, and the workflow runs on its own changes too.
/approve
|
LGTM label has been added. DetailsGit tree hash: 515c43aa9b0ccaea8431ad017bbcf285154c5c14 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: leogr, ManuelFCastillo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Fixes #1512
Why CI missed #1500
No job both built against an old glibc and loaded the result:
build-linuxbuilds insidedebian:bullseye(glibc 2.31) but neverdlopens what it produced.falco-testsdoesdlopen, but insidefalcosecurity/falco:master-debian(Debian 12, glibc 2.36), where a missingDT_NEEDEDonlibresolv.so.2cannot fail, because those symbols moved into libc in glibc 2.34.Change
Both checks go in
build-linux, which already runs on bullseye, so neither needs a new job, container or matrix entry, and both run for amd64 and arm64.readelf -dasserts theDT_NEEDEDentry is present. Fails fast and names the cause.dlopen(..., RTLD_NOW)loads the built library on glibc 2.31. This is the check that would have caught container: libcontainer.so fails to load on glibc < 2.34 (undefined symbol: __res_search) — missing -lresolv on Linux #1500, and unlike the first it also catches a different symbol going missing on old glibc for an unrelated reason.RTLD_NOWrather thanRTLD_LAZYso every symbol is bound at load time; lazy binding would defer function resolution and could pass.readelfandgccboth come frombuild-essential, already installed by the job.Verification
Built a probe library calling
res_searchtwo ways indebian:bullseyeand ran both guards against each:The second case is #1500 reproduced: a shared object links fine without
-lresolvbecause undefined symbols are permitted at link time, then fails atdlopenon any host whereres_searchstill lives inlibresolv.Note
The issue also suggested loading on a glibc < 2.34 image as the stronger option. That is what step 2 does, and putting it in
build-linuxrather than a separate job means it runs against the artifact that job just built, rather than a rebuild./kind feature
/area plugins
🤖 Generated with Claude Code