[Missing Driver]: falco_amazonlinux2_5.4.238-148.346.amzn2.x86_64_1.ko

[tigardis]

Describe the bug

falco-driver-loader * Looking for a falco module locally (kernel 5.4.238-148.346.amzn2.x86_64)
falco-driver-loader * Filename 'falco_amazonlinux2_5.4.238-148.346.amzn2.x86_64_1.ko' is composed of:
falco-driver-loader - driver name: falco
falco-driver-loader - target identifier: amazonlinux2
falco-driver-loader - kernel release: 5.4.238-148.346.amzn2.x86_64
falco-driver-loader - kernel version: 1
falco-driver-loader * Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_amazonlinux2_5.4.238-148.346.amzn2.x86_64_1.ko
falco-driver-loader curl: (22) The requested URL returned error: 404

How to reproduce it

1. Start/install Falco on Amazon Linux2 5.4.238-148.346

Expected behaviour

Startup succeeds after locating driver

Screenshots

Environment

• Falco version:

Below is from our Prod cluster - Falco doesn't start in our test cluster. I have tried deploying falco v3.1.3 but it does not solve the problem.

Tue Apr 11 20:33:07 2023: Falco version: 0.34.0 (x86_64)
Tue Apr 11 20:33:07 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Falco version: 0.34.0
Libs version: 0.10.3
Plugin API: 2.0.0
Engine: 15
Driver:
API version: 3.0.0
Schema version: 2.0.0
Default driver: 4.0.0+driver

• System info:

• Cloud provider or hardware configuration:
• OS:

• Kernel:

• Installation method:
  AWS EKS - Kubernetes v1.23.17 (Helm chart 3.0.0 or 3.1.3)

Additional context

[Andreagit97]

uhm thank you for reporting this, probably the kernel crawler still searching for this kernel @FedeDP

[FedeDP]

Hi!
We have 5.4.238-148.347: https://download.falco.org/driver/site/index.html?lib=4.0.0%2Bdriver&target=amazonlinux2&arch=x86_64&kind=kmod&search=falco_amazonlinux2_5.4.238-148.347.amzn2.x86_64_1.ko
I think you need to upgrade your kernel to latest version! We did never build 5.4.238-148.346 unfortunately; most probably it was immediately patched to .347 and since our pipeline runs weekly, we only discovered and built the latter.

[renilthomas]

The EKS latest AMI release has pinned 5.4.238-148.346.amzn2 for k8s v1.23 or below. Can the kernel crawler build based on a specific version?
At this stage, there are some critical vulnerabilities that is addressed in this new AMI but we have only two options:

1. Disable falco and upgrade AMI
2. Upgrade k8s - which obviously requires scheduling a maintenance and planning towards it.

[FedeDP]

Hi! It seems like we fail to crawl the kernel; i will look into this asap :)

[renilthomas]

Thank you @FedeDP 🙇

[FedeDP]

Mmh my guess is that the 5.4.238-148.346.amzn2 kernel is not present in any mirror anymore; as i said previously, we can only scrape 5.4.238-148.347.amzn2.
Are you able to share the url for the 5.4.238-148.346.amzn2 package?

[renilthomas]

I got this URL from the instance with the specific kernel version

https://amazonlinux-2-repos-us-west-2.s3.dualstack.us-west-2.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/2d43c5da4c1aee506efc1db66eca9864d37d1f92076bd0eb8b357c32189079bf/../../../../../../blobstore/6e296e6f7ea2f7f03d201ac21967f5c61bdb7904485492cc822b34d6892af4c8/kernel-5.4.238-148.346.amzn2.x86_64.rpm

[FedeDP]

For reference, this is what the crawler finds instead:
http://amazonlinux.us-west-2.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61/../../../../../../blobstore/fa5caa71652bca6f4a8aeff9356d18bd1785d8620080398768acbb53a7fb5756/kernel-devel-5.4.238-148.347.amzn2.x86_64.rpm

[renilthomas]

Yes i have seen that running kernel-crawler myself. Is it possible to build with that URL I shared? Thank you in advance!

[FedeDP]

The quickest fix, since you have the full URL, is to add a new config on test-infra for it by eg: copy/pasting the 347 one: https://github.com/falcosecurity/test-infra/blob/master/driverkit/config/4.0.0%2Bdriver/x86_64/amazonlinux2_5.4.238-148.347.amzn2.x86_64_1.yaml

If you open a PR (please add it for all 3 supported driver versions!) i will be happy to accept it :)

[renilthomas]

Brilliant, on it.. Thanks @FedeDP

[FedeDP]

In the meantime, i am trying to figure out what are we missing :)
Also, @EXONER4TED (gentle tag ahah) is the 👑 when finding weird kernel-crawler issues :D

[FedeDP]

Wait! i just notice: you are sharing the kernel URL, not the kernel-devel one.
Perhaps there is no kernel-devel package for it?

[renilthomas]

oh right, checking

[renilthomas]

You are right, there seems to be none for 346 😞

[FedeDP]

So, did they push an updated kernel without pushing updated headers? That would be really weird...

[renilthomas]

I am checking with AWS directly, yes if that is the case it is truly absurd.

[FedeDP]

I still don't get where does this kernel version come from; doing a

wget amazonlinux.us-west-2.amazonaws.com/2/extras/kernel-5.4/latest/x86_64/mirror.list

yelds a mirror.list file with

http://amazonlinux.us-east-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61

as a content. That is what kernel-crawler is discovering too.
There should not be any other mirror, as far as i could find, for kernel-5.4.

[eelkoniovb]

We also suffer from this problem. It's happened before, about two years ago I think.

I've already asked AWS Support to fix this. Indeed, there are no entries for the newest AMI in their package list. We build the falco module via the driverkit, but it fails as there are no package entries for this new AMI kernel version:

Step by step; when you retrieve this:

http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/latest/x86_64/mirror.list

..it returns this URL: http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61

When retrieving this as an sqlite database:
http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61/repodata/primary.sqlite.gz

...we get a database that holds no info for the latest AMI version, but several entries for the ones before:

% gunzip primary.sqlite.gz ; sqlite3 primary.sqlite
SQLite version 3.39.5 2022-10-14 20:58:05
Enter ".help" for usage hints.
sqlite> select rpm_sourcerpm from packages where rpm_sourcerpm like "%5.4.235-144.344%";
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
sqlite> select rpm_sourcerpm from packages where rpm_sourcerpm like "%5.4.238-148.346.amzn2.x86_64%";
sqlite>

My AWS support person has not yet understood this is the problem, or cannot find the people involved for this sqlite database. My guess is not the latter, unfortunately.

Cheers,
Eelko

[FedeDP]

Yep, our kernel-crawler does exactly the same thing; and indeed it cannot find this specific kernel.
To me, this seems like an aws internal issue; let's see.

[renilthomas]

My AWS case is already escalated to the internal ec2 team. Will keep you posted!

[tigardis]

Sure thing. @FedeDP we are using the latest available AWS verified AMI for 1.23 (ami-0559f43d314aa2e7e), and as others have mentioned upgrading K8s to 1.24 will require more planning. At this point I am most likely going to try downgrading to an earlier AMI since this version was missed due to pipeline configuration.

…

On Wed, Apr 12, 2023 at 3:57 AM Andrea Terzolo ***@***.***> wrote: uhm thank you for reporting this, probably the kernel crawler still searching for this kernel @FedeDP <https://github.com/FedeDP> — Reply to this email directly, view it on GitHub <#1093 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQ3DKIWEKCDMUVPY6FB3EQDXA2DCJANCNFSM6AAAAAAW2YSN7Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mpurusottamc]

We are also facing similar issue with latest AWS EKS AMI for 1.23. We have tried to downgrade the ami and it seems to be working.

@renilthomas Thanks for working with the ec2 team. Please share if you hear any updates.

[mpurusottamc]

@renilthomas AWS has updated the AMIs and the updated AMI uses kernel version 5.4.238-148.347.amzn2.x86_64. We did an initial testing and falco probe is getting downloaded properly and is working properly now.
cc: @FedeDP

[mpurusottamc]

Release details are here: https://github.com/awslabs/amazon-eks-ami/releases

[renilthomas]

That is correct, they responded to my issue awslabs/amazon-eks-ami#1266 (comment)
And I had also escalated it within AWS and last I heard earlier was they were working on the fix.
So, all good now @FedeDP

[FedeDP]

So, this was an issue on the AWS side right?
Great to hear it's getting fixed!:)

[ajinkya1986]

Hi All, Just wanted to update that we have tested falco versions 2.0.18,2.5.5,3.0.0 and 3.1.3 with k8s version 1.22 to 1.26. And the falco daemonset is working as expected with ebpf probe getting loaded and we are getting threats data as well

[FedeDP]

I am going to pin this issue since it caused multiple headaches :D

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[tigardis]

/close

…

On Tue, Jul 25, 2023 at 6:32 AM poiana ***@***.***> wrote: Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/falcosecurity/community. /lifecycle stale — Reply to this email directly, view it on GitHub <#1093 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQ3DKIUV6MJNDUEE7QBVTHLXR7DG7ANCNFSM6AAAAAAW2YSN7Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[poiana]

@tigardis: Closing this issue.

In response to this:

/close

On Tue, Jul 25, 2023 at 6:32 AM poiana @.***> wrote:

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually
close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

—
Reply to this email directly, view it on GitHub
#1093 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQ3DKIUV6MJNDUEE7QBVTHLXR7DG7ANCNFSM6AAAAAAW2YSN7Y
.
You are receiving this because you authored the thread.Message ID:
@.***>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.