Can't use argdist to trace args which type is double

[Hankin-Liu]

[root@fedora tools]# ./argdist -C 'p:c:sqrt(double x):double:x'
error: /virtual/main.c:24:13: in function sqrt_probe0 i32 (%struct.pt_regs*): A call to built-in function '__floatundidf' is not supported.

Does this tool support float or double parameter trace?

[chenhengqi]

Very weird. Will investigate.

BTW, you should trace libm.so, not libc.so.

#if defined(BPF_LICENSE)
#error BPF_LICENSE cannot be specified through cflags
#endif
#if !defined(CONFIG_CC_STACKPROTECTOR)
#if defined(CONFIG_CC_STACKPROTECTOR_AUTO) \
    || defined(CONFIG_CC_STACKPROTECTOR_REGULAR) \
    || defined(CONFIG_CC_STACKPROTECTOR_STRONG)
#define CONFIG_CC_STACKPROTECTOR
#endif
#endif

struct __string_t { char s[80]; };

#include <uapi/linux/ptrace.h>
                
struct sqrt_hash0_key_t {
long v0;
};
BPF_HASH(sqrt_hash0, struct sqrt_hash0_key_t, u64);

                __attribute__((section(".bpf.fn.sqrt_probe0")))
int sqrt_probe0(struct pt_regs *ctx)
{
 double x = ctx->di;  <- This line causes the problem.
        u64 __pid_tgid = bpf_get_current_pid_tgid();
        u32 __pid      = __pid_tgid;        // lower 32 bits
        u32 __tgid     = __pid_tgid >> 32;  // upper 32 bits
        
        
        
        struct sqrt_hash0_key_t __key = {};
        __key.v0 = x;

        if (!(1)) return 0;
        ({ typeof(sqrt_hash0.key) _key = __key; typeof(sqrt_hash0.leaf) *_leaf = bpf_map_lookup_elem_(bpf_pseudo_fd(1, -1), &_key); if (_leaf) lock_xadd(_leaf, 1);else { typeof(sqrt_hash0.leaf) _zleaf; __builtin_memset(&_zleaf, 0, sizeof(_zleaf)); _zleaf += 1;bpf_map_update_elem_(bpf_pseudo_fd(1, -1), &_key, &_zleaf, BPF_NOEXIST); } });
        return 0;
}

#include <bcc/footer.h>

[yonghong-song]

double x = ctx->di; <- This line causes the problem.
Right, the above is the problem. The BPF instruction set does not have floating point instructions so the above code won't be able to pass compilation.

[chenhengqi]

@yonghong-song

Did not realize this. :)

So what is BTF_KIND_FLOAT for ?

[yonghong-song]

@yonghong-song

Did not realize this. :)

So what is BTF_KIND_FLOAT for ?

BTF_KIND_FLOAT is for vmlinux BTF (with arch x86_64, arm, etc.)

[Hankin-Liu]

can we use assembly code to trace double variables? As we know, gcc ABI defines call convention. For X86_64, gcc will use xmm register to store the floating parameters.

[chenhengqi]

How about using long x to capture the param and interpret it as double in user space ?

[Hankin-Liu]

How about using long x to capture the param and interpret it as double in user space ?

I do not think so. I think if double implicitly cast to long, value will be changed.

[chenhengqi]

Yeah, should be __u8[8] instead of long.

[chenhengqi]

I tried, but failed. The double is skipped.

[chenhengqi]

Test program:

static int foobar(int x, double y, int z) {
	return x;
}

int main() {
	int x = foobar(123, 456, 789);
	return x;
}

BPF program:

SEC("kprobe/foobar")
int handle_foobar(struct pt_regs *ctx)
{
	__u64 p1 = PT_REGS_PARM1_CORE(ctx);
	bpf_printk("%ld %d %x", p1, p1, p1);

	__u64 p2 = PT_REGS_PARM2_CORE(ctx);
	bpf_printk("%ld %d %x", p2, p2, p2);

	__u64 p3 = PT_REGS_PARM3_CORE(ctx);
	bpf_printk("%ld %d %x", p3, p3, p3);

	return 0;
}

Output:

           a.out-3974052 [005] d..31 5562323.950167: bpf_trace_printk: 123 123 7b
           a.out-3974052 [005] d..31 5562323.950183: bpf_trace_printk: 789 789 315
           a.out-3974052 [005] d..31 5562323.950184: bpf_trace_printk: 140731377347240 -1816040792 93c16aa8

[Hankin-Liu]

Test program:

static int foobar(int x, double y, int z) {
	return x;
}

int main() {
	int x = foobar(123, 456, 789);
	return x;
}

BPF program:

SEC("kprobe/foobar")
int handle_foobar(struct pt_regs *ctx)
{
	__u64 p1 = PT_REGS_PARM1_CORE(ctx);
	bpf_printk("%ld %d %x", p1, p1, p1);

	__u64 p2 = PT_REGS_PARM2_CORE(ctx);
	bpf_printk("%ld %d %x", p2, p2, p2);

	__u64 p3 = PT_REGS_PARM3_CORE(ctx);
	bpf_printk("%ld %d %x", p3, p3, p3);

	return 0;
}

Output:

           a.out-3974052 [005] d..31 5562323.950167: bpf_trace_printk: 123 123 7b
           a.out-3974052 [005] d..31 5562323.950183: bpf_trace_printk: 789 789 315
           a.out-3974052 [005] d..31 5562323.950184: bpf_trace_printk: 140731377347240 -1816040792 93c16aa8

I think maybe we can fetch double parameter from registers. But I am not familiar with LLVM call convention. Does someone knows LLVM call convention?

[yonghong-song]

The x86_64 calling convention is to use xmm registers for floating point numbers. But unfortunately, xmm registers are not preserved in pt_regs, so we cannot really get them. Not sure debugfs uprobe_events can handle this or not.

[Hankin-Liu]

The x86_64 calling convention is to use xmm registers for floating point numbers. But unfortunately, xmm registers are not preserved in pt_regs, so we cannot really get them. Not sure debugfs uprobe_events can handle this or not.

I am not meaning get double from pt_resg, I mean get double from registers directly. For example(in X86_64 architecture):

#include <iostream>
#include <thread>
#include <chrono>
#include <atomic>
#include <cmath>
#include <cstdlib>
using namespace std;

double v1, v2;
volatile double myuprobe_sum_double(double a, double b)
{
    __asm__ __volatile__("movups %%xmm0, %0":"=m"(v1)::"memory");
    __asm__ __volatile__("movups %%xmm1, %0":"=m"(v2)::"memory");
    printf("v1 = %lf, v2 = %lf\na = %lf, b = %lf\n", v1, v2, a, b);
	return a + b;
}

int main()
{
	srand((int)time(0));
	while(1) {
        double d = rand() + rand() / double(RAND_MAX);
        double f = rand() + rand() / double(RAND_MAX);
        auto e = myuprobe_sum_double(d, f);
		std::this_thread::sleep_for(std::chrono::seconds(5));
	}
	return 0;
}

// compile : g++ test1.cpp -o test
// output
//# ./test
v1 = 1952021791.245945, v2 = 1920042416.187097
a = 1952021791.245945, b = 1920042416.187097
v1 = 907279145.362380, v2 = 541307865.774999
a = 907279145.362380, b = 541307865.774999
v1 = 946671870.404972, v2 = 2034959625.277564
a = 946671870.404972, b = 2034959625.277564

[chenhengqi]

You can't do that in BPF.

[Hankin-Liu]

You can't do that in BPF.

Yeah, just a thinking. Is assembly code avaliable in BPF code? I know it must be different from INTEL or ARM assembly code.

[chenhengqi]

bcc/libbpf-tools/cpufreq.bpf.c

Lines 20 to 26 in 3a03901

	#define clamp_umax(VAR, UMAX) \
	asm volatile ( \
	"if %0 <= %[max] goto +1\n" \
	"%0 = %[max]\n" \
	: "+r"(VAR) \
	: [max]"i"(UMAX) \
	)

[yonghong-song]

Yes, bpf supports asm code. The assembly code has to be bpf insn similar to asm code generated with clang -target bpf -O2 -S test.c and movups %%xmm0, %0 is certainly not a bpf insn so it won't work.

The best possible way is to use a bpf helper and inside the helper you can use x86 asm code to retrieve the value. The helper needs to be generic so it can apply to different architectures.

[davemarchevsky]

@Hankin-Liu I noticed a similar issue in #3880. In that case, none of the arguments to my USDT were floats, but register pressure (presumably) resulted in the compiler choosing to use an xmm register to pass an integer.

After talking to @yonghong-song, unfortunately his 'use a bpf helper' suggestion:

The best possible way is to use a bpf helper and inside the helper you can use x86 asm code to retrieve the value. The helper needs to be generic so it can apply to different architectures.

is the best path forward. I'm working on a patch to expose such a helper, but we'll have to wait until it's included in a new kernel release to fix this problem permanently.

---

In my case, I considered a potential workaround similar to what you did in this comment: reading the register from userspace since I wasn't using the value in the BPF program, just putting it in a perf buffer and sending it to userspace. Such a workaround would be very flakey at best since xmm isn't preserved by the kernel and thus could be clobbered by another process, and different compiler settings could result in a different register being chosen.

I decided not to validate this workaround, just throwing it out there.