CI: Verifier tests: Keep generated object files and logs on test failure #25862
Conversation
qmonnet
added
area/CI
qmonnet
requested review from
ti-mo,
tklauser,
julianwiedmann and
viktor-kurchenko
|
Tested on another PR, for example on this run. |
qmonnet
force-pushed
the
pr/ci-verifier-keep-objfiles
branch
from
6068539
to
8178f17
Compare
|
Let's give all .o's a unique name instead and include it as part of the name of the subtest. I think renaming is confusing, and it'd be nice to have all objs in the zip for debugging. |
|
@ti-mo: I don't mind adding all .o's to the artifacts. But that still means renaming them, right? Only we rename them all, and earlier, instead of doing it only on failure. We could try to generate them with the right name from the beginning, but given we're executing a call to |
qmonnet
force-pushed
the
pr/ci-verifier-keep-objfiles
branch
from
8178f17
to
cb94261
Compare
|
Updated to rename all object files regardless of exit status. I checked (locally) that all object files are preserved. @ti-mo does this correspond to what you had in mind? |
Yes, thanks! The make invocation is indeed what deterred me from doing the renaming at the time. We can start using Cilium's Go clang pipeline here at some point for more flexibility. In the meantime, this will do! |
|
Not sure why |
|
/test |
|
/test-runtime |
qmonnet
force-pushed
the
pr/ci-verifier-keep-objfiles
branch
from
cb94261
to
2ef102c
Compare
|
/test Job 'Cilium-PR-K8s-1.26-kernel-net-next' failed: Click to show.Test NameFailure OutputJenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next/505/ If it is a flake and a GitHub issue doesn't already exist to track it, comment Then please upload the Jenkins artifacts to that issue. |
The verifier tests include a run of "make -C bpf/ clean" prior to testing the programs in each BPF source files. While this sounds like a sane practice, this means that at the end of the run, we're left with only one object file under bpf/, the one used by the last tests (namely: bpf_sock.o). While the object files are trivial to rebuild locally, this prevents the CI workflow to find and upload the relevant object files for instances where the verifier's logs are not enough to debug. Let's run instead a single "make -C bpf/ clean" on startup. Signed-off-by: Quentin Monnet <quentin@isovalent.com>
For a given source file, we build the object multiple times, with a different set of build options. This results in object files being lost on failure. To address this, we rename the object files to include the iteration number, so that they're still present when we upload to the CI artifact. Signed-off-by: Quentin Monnet <quentin@isovalent.com>
In the tests-datapath-verifier workflow, all jobs generated from the matrix (for running the tests with different kernel versions) upload files to the same artifact on failure. This means that a job may overwrite the upload from a previous job, and in the end we only get the uploads from the last job to complete. If several jobs fail with different errors, then we lose useful debugging information. Let's have the jobs upload their files to separate artifacts, named after the kernel version in use. Signed-off-by: Quentin Monnet <quentin@isovalent.com>
qmonnet
force-pushed
the
pr/ci-verifier-keep-objfiles
branch
from
2ef102c
to
bda8d4d
Compare
|
/test Job 'Cilium-PR-K8s-1.26-kernel-net-next' hit: #25958 (89.87% similarity) |
qmonnet
added
the
ready-to-merge
The CI workflow for testing Cilium's programs against the verifier is supposed to upload its log files and the generated files on failures. However, we lose some of this data, because:
Multiple jobs (for different kernel versions) upload their log and object files to the same artifact, overwriting previous uploads in the process, so that we only keep the files uploaded by the last job to complete. We can address that by using different artifact names for the different jobs.
The test itself runs
make -C bpf/ cleanbefore testing the programs of each source file, thus removing the object files previously generated. Instead, we can clean up the directory just once before running the tests, not between two tests.At last, each source file is compiled multiple times with different sets of options, resulting in only the last version of this object file being kept. The solution implemented here consists in renaming these object files (on test failure) to make sure we keep them around.