[Bug]: bpftrace cannot obtain ena_io_poll return value

[hengqiali]

Preliminary Actions

• I have searched the existing issues and didn't find a duplicate.
• I have followed the AWS official troubleshoot documentation.
• I have followed the driver readme and best practices.

Driver Type

Linux kernel driver for Elastic Network Adapter (ENA)

Driver Tag/Commit

6.1.66-91.160.amzn2023.x86_64

Custom Code

No

OS Platform and Distribution

6.1.66-91.160.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMI

Bug description

Try to get the return value using the following command:
bpftrace -e 'kretprobe: ena_io_poll {@[nsecs/1000000000]=count();}'

bpftrace cannot obtain the ena_io_poll return value. How should I get it?
Modify the code? I added noinline which doesn't seem to work.

Reproduction steps

Excute the following cmd:
bpftrace -e 'kretprobe: ena_io_poll {@[nsecs/1000000000]=count();}'

Expected Behavior

Get the corresponding return value.

Actual Behavior

Cannot get the corresponding return value.

Additional Data

# bpftrace -e 'kretprobe: ena_io_poll {@[nsecs/1000000000]=count();}'
Attaching 1 probe...
cannot attach kprobe, probe entry may not exist
ERROR: Error attaching probe: 'kretprobe:ena_io_poll'

Relevant log output

No response

Contact Details

No response

[davidarinzon]

Hi @Data-crea

Thanks for submitting the query, the command you've posted works well on Amazon Linux 2 and Ubuntu 22.04.
We will investigate to see why it doesn't work on Amazon Linux 2023 and will get back to you.
Stay tuned.

[amitbern-aws]

Hi @Data-crea,
We can confirm that this issue has been replicated in the Amazon Linux 2023 kernel (version 6.1).
Although the problem is not related to the ENA driver, we have internally assigned it to the Amazon Linux kernel team.

To mitigate this issue, you can:

1. Use Amazon Linux 2 (5.10)
2. Use kretfunc

sudo bpftrace -e 'kretfunc:ena:ena_io_poll {@[nsecs/1000000000]=count();}'

We will provide an update once we receive a resolution from the Amazon Linux team.

Thank you for bringing this to our attention.

[hengqiali]

Hi @amitbern-aws @davidarinzon

Hi @Data-crea, We can confirm that this issue has been replicated in the Amazon Linux 2023 kernel (version 6.1). Although the problem is not related to the ENA driver, we have internally assigned it to the Amazon Linux kernel team.

To mitigate this issue, you can:

1. Use Amazon Linux 2 (5.10)
2. Use kretfunc

sudo bpftrace -e 'kretfunc:ena:ena_io_poll {@[nsecs/1000000000]=count();}'

Thank you very much for your effective response! I will try this!

We will provide an update once we receive a resolution from the Amazon Linux team.

Thank you for bringing this to our attention.

[amitbern-aws]

Update from kernel team:

• The bug introduced in commit ("tracing/kprobes: Return EADDRNOTAVAIL when func matches several symbols") merged in upstream kernel v6.1.60 and broke the kprobe on kernel module symbols.
• The fix ("tracing/kprobes: Fix symbol counting logic by looking at modules as well") will be merged to v6.1.71.

Amazon Linux 2023 will take the fix in the coming weeks,
Will update once its available.

Thanks

[hengqiali]

I see it! Thanks again!