[bpf-ci-bot] Flaky test: task_local_storage/sys_enter_exit

[kernel-patches-review-bot]

Summary

The task_local_storage/sys_enter_exit BPF selftest is flaky. It fails
intermittently across multiple independent patch series and on the
baseline for-next_test branch, causing false CI failures in BPF CI
(kernel-patches/bpf).

Failure Pattern

When the test fails, the values are always the same:

test_sys_enter_exit:FAIL:enter_cnt unexpected enter_cnt: actual 2 != expected 3
test_sys_enter_exit:PASS:exit_cnt 0 nsec
test_sys_enter_exit:FAIL:mismatch_cnt unexpected mismatch_cnt: actual 3 != expected 0

That is: enter_cnt=2, exit_cnt=3, mismatch_cnt=3.

Affected CI Runs (recent examples)

• for-next_test run 22314858282 (Feb 23): test_progs on x86_64 gcc-15
• series/1056598=>bpf-next run 22315036624 (Feb 23): test_progs_no_alu32
• series/1055783=>bpf-next run 22314940225 (Feb 23): test_progs_no_alu32
• to-test run 22246444389 (Feb 21): test_progs_cpuv4 on x86_64 llvm-21

The test passes on other runs with the same code (e.g., run 22326871125),
confirming this is a flaky failure, not a deterministic regression.

Root Cause Analysis

The test in prog_tests/task_local_storage.c sets target_pid before
calling task_local_storage__attach():

skel->bss->target_pid = sys_gettid();    // set pid before attach

err = task_local_storage__attach(skel);   // attach BPF programs

sys_gettid();
sys_gettid();

/* 3x syscalls: 1x attach and 2x gettid */
ASSERT_EQ(skel->bss->enter_cnt, 3, "enter_cnt");

The test assumes that exactly 1 syscall during the attach() call will
be observed by the BPF programs. Specifically, it expects the
bpf(BPF_LINK_CREATE) syscall that attaches on_exit (the second
program) to be caught by the already-attached on_enter program.

The skeleton attaches programs in ELF order: on_enter (tp_btf/sys_enter)
first, then on_exit (tp_btf/sys_exit). When on_enter is attached via
bpf(BPF_LINK_CREATE), it registers with the sys_enter tracepoint.
The subsequent bpf(BPF_LINK_CREATE) for on_exit triggers sys_enter,
which should fire on_enter.

However, this relies on the tracepoint registration from the first
BPF_LINK_CREATE being immediately visible to the tracepoint dispatch
code during the second BPF_LINK_CREATE. The tracepoint callback array is
published via rcu_assign_pointer() and read via rcu_dereference_raw()
(essentially READ_ONCE). While this should work on the same CPU due to
program ordering, the user thread can be migrated between the two syscalls.
On a different CPU, the new tracepoint callback may not yet be visible if
the RCU publication hasn't fully propagated—the rcu_dereference_raw lacks
acquire semantics.

When the on_enter callback is not visible during the second
BPF_LINK_CREATE:

• sys_enter fires but on_enter does NOT run → no enter counted
• on_exit is registered during the syscall
• sys_exit fires and on_exit DOES run → exit_cnt=1, creating a
  fresh task storage entry with value 0

This off-by-one between enter and exit counts cascades through all
subsequent syscalls, producing the observed enter_cnt=2, exit_cnt=3, mismatch_cnt=3 failure.

Proposed Fix

Move the target_pid assignment to after the attach() call. This
way, no syscalls during the attach phase match the BPF program's PID
filter (since target_pid is still 0), eliminating the dependency on
how many syscalls attach() generates and their tracepoint visibility:

pid_t pid;

pid = sys_gettid();              // get PID (programs not attached yet)
err = task_local_storage__attach(skel);  // attach (target_pid == 0, nothing counted)
skel->bss->target_pid = pid;    // set PID (memory write, no syscall)

sys_gettid();                    // counted: enter_cnt=1, exit_cnt=1
sys_gettid();                    // counted: enter_cnt=2, exit_cnt=2

ASSERT_EQ(skel->bss->enter_cnt, 2, "enter_cnt");
ASSERT_EQ(skel->bss->exit_cnt, 2, "exit_cnt");
ASSERT_EQ(skel->bss->mismatch_cnt, 0, "mismatch_cnt");

This makes the test deterministic: only the two explicit sys_gettid()
calls are counted, with no dependence on the internal syscall behavior
of task_local_storage__attach().

A patch implementing this fix is attached as
0001-selftests-bpf-fix-flaky-task_local_storage-sys_enter_exit-test.patch.

[kernel-patches-review-bot]

0001-selftests-bpf-fix-flaky-task_local_storage-sys_enter_exit-test.patch

From: BPF CI Agent <bpf-ci-agent@kernel-patches.github.io>
Date: Sun, 23 Feb 2026 22:00:00 +0000
Subject: [PATCH] selftests/bpf: fix flaky task_local_storage/sys_enter_exit test

The task_local_storage/sys_enter_exit selftest is flaky, failing
intermittently across multiple independent CI runs with the pattern:

  enter_cnt: actual 2 != expected 3
  mismatch_cnt: actual 3 != expected 0

The test sets target_pid before calling task_local_storage__attach(),
which means BPF programs observe internal syscalls generated during the
skeleton attach sequence. Specifically, it assumes that exactly one
syscall during attach() -- the bpf(BPF_LINK_CREATE) that attaches the
second program (on_exit) -- will be observed by the already-attached
first program (on_enter).

This assumption is fragile because the tracepoint callback registration
from the first BPF_LINK_CREATE may not be immediately visible to the
tracepoint dispatch code during the second BPF_LINK_CREATE, particularly
if the user thread is migrated to a different CPU between the two
syscalls. The tracepoint callback array is published via
rcu_assign_pointer() but read via rcu_dereference_raw() (READ_ONCE),
which lacks acquire semantics on architectures with weak memory ordering.

When the on_enter callback is not visible during the attach of on_exit,
sys_exit fires without a prior sys_enter event from on_enter, creating
an off-by-one misalignment that cascades through all subsequent syscalls.

Fix this by moving the target_pid assignment to after attach(). The PID
is obtained before attach via sys_gettid() (which does not trigger the
BPF programs since they are not yet attached), stored in a local
variable, and then written to target_pid after attach completes. This
eliminates any dependency on the number or visibility of syscalls
generated internally by the skeleton attach sequence.

Fixes: 1f87dcf116ad ("selftests/bpf: Add non-BPF_LSM test for task local storage")
Signed-off-by: BPF CI Agent <bpf-ci-agent@kernel-patches.github.io>
---
 .../selftests/bpf/prog_tests/task_local_storage.c   | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/task_local_storage.c b/tools/testing/selftests/bpf/prog_tests/task_local_storage.c
index 7bee33797c71..01edc759416b 100644
--- a/tools/testing/selftests/bpf/prog_tests/task_local_storage.c
+++ b/tools/testing/selftests/bpf/prog_tests/task_local_storage.c
@@ -26,23 +26,32 @@ static void test_sys_enter_exit(void)
 {
 	struct task_local_storage *skel;
 	int err;
+	pid_t pid;

 	skel = task_local_storage__open_and_load();
 	if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
 		return;

-	skel->bss->target_pid = sys_gettid();
+	pid = sys_gettid();

 	err = task_local_storage__attach(skel);
 	if (!ASSERT_OK(err, "skel_attach"))
 		goto out;

+	/* Set target_pid after attach so that syscalls made during
+	 * attach are not counted. This avoids depending on the
+	 * number of internally-generated syscalls in attach, which
+	 * varies depending on tracepoint registration visibility
+	 * at the time each BPF program is attached.
+	 */
+	skel->bss->target_pid = pid;
+
 	sys_gettid();
 	sys_gettid();

-	/* 3x syscalls: 1x attach and 2x gettid */
-	ASSERT_EQ(skel->bss->enter_cnt, 3, "enter_cnt");
-	ASSERT_EQ(skel->bss->exit_cnt, 3, "exit_cnt");
+	/* 2x gettid syscalls */
+	ASSERT_EQ(skel->bss->enter_cnt, 2, "enter_cnt");
+	ASSERT_EQ(skel->bss->exit_cnt, 2, "exit_cnt");
 	ASSERT_EQ(skel->bss->mismatch_cnt, 0, "mismatch_cnt");
 out:
 	task_local_storage__destroy(skel);
--
2.43.0

[theihor]

As far as I can tell, the problem in the test that AI identified is right. It's not guaranteed that task_local_storage__attach will trigger exactly one on_enter and on_exit BPF prog call. And the solution makes sense too.

I am not convinced by the AI's analysis regarding the RCU race due to CPU migration. It may be right and may be wrong, but that's irrelevant in the context of the test flakiness.

I created (with AI) a test case that reproduces the assert mismatch without a race, here it is:

static void test_sys_enter_exit(void)
{
	struct task_local_storage *skel;
	struct bpf_link *exit_link = NULL, *enter_link = NULL;
	pid_t pid;
	int err;

	skel = task_local_storage__open_and_load();
	if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
		return;

	/*
	 * Reproduce the flaky failure deterministically.
	 *
	 * The original test sets target_pid before attach() and assumes the
	 * BPF_LINK_CREATE for on_exit is the only syscall observed during
	 * attach() (enter+exit paired, no mismatch).  If on_exit fires at
	 * sys_exit for a syscall where on_enter didn't fire at sys_enter,
	 * exit_cnt permanently leads enter_cnt by 1, and every subsequent
	 * on_exit check fails (stored value is MAGIC+enter_cnt but on_exit
	 * expects MAGIC+exit_cnt).
	 *
	 * We trigger that exact imbalance without a race by controlling the
	 * attach order and target_pid visibility:
	 *
	 *  1. Attach on_exit with target_pid set (on_enter not yet installed):
	 *       BPF_LINK_CREATE syscall: sys_enter → on_enter absent, nothing;
	 *                                on_exit installed mid-syscall;
	 *                                sys_exit → on_exit fires, exit_cnt=1,
	 *                                ptr=0 (F_CREATE), 0 != MAGIC+1: mismatch=1
	 *       State: enter=0 exit=1 mismatch=1
	 *
	 *  2. Attach on_enter with target_pid=0 so on_exit also skips its
	 *     BPF_LINK_CREATE syscall; restore target_pid afterwards.
	 *       State unchanged: enter=0 exit=1 mismatch=1
	 *
	 *  3. Two sys_gettid() calls with both programs live:
	 *       gettid #1: enter=1, ptr=MAGIC+1; exit=2, MAGIC+1 != MAGIC+2 → mismatch=2
	 *       gettid #2: enter=2, ptr=MAGIC+2; exit=3, MAGIC+2 != MAGIC+3 → mismatch=3
	 *
	 * Result: enter_cnt=2, exit_cnt=3, mismatch_cnt=3.
	 * This is the exact failure pattern reported for the original test.
	 */

	bpf_program__set_autoattach(skel->progs.on_enter, false);
	bpf_program__set_autoattach(skel->progs.on_exit, false);

	err = task_local_storage__attach(skel); /* nothing auto-attaches */
	if (!ASSERT_OK(err, "skel_attach"))
		goto out;

	pid = sys_gettid();

	/* Step 1: attach on_exit while on_enter is absent → one unpaired sys_exit */
	skel->bss->target_pid = pid;
	exit_link = bpf_program__attach(skel->progs.on_exit);
	if (!ASSERT_OK_PTR(exit_link, "attach_on_exit"))
		goto out;

	/* Step 2: attach on_enter invisibly (target_pid=0), then restore */
	skel->bss->target_pid = 0;
	enter_link = bpf_program__attach(skel->progs.on_enter);
	if (!ASSERT_OK_PTR(enter_link, "attach_on_enter"))
		goto out;
	skel->bss->target_pid = pid;

	/* Step 3: two gettids, both programs live but permanently off by one */
	sys_gettid();
	sys_gettid();

	/* Original test's assertions — enter_cnt=2 (not 3) is the bug */
	ASSERT_EQ(skel->bss->enter_cnt, 3, "enter_cnt");
	ASSERT_EQ(skel->bss->exit_cnt, 3, "exit_cnt");
	ASSERT_EQ(skel->bss->mismatch_cnt, 0, "mismatch_cnt");

out:
	bpf_link__destroy(enter_link);
	bpf_link__destroy(exit_link);
	task_local_storage__destroy(skel);
}

Basically, the issue can be reproduced by manually controlling the attach order.

[theihor]

Submitted a patch upstream: https://lore.kernel.org/bpf/20260224015855.1481707-1-ihor.solodrai@linux.dev/