Session view processor procfs #38799

mjwolf · 2024-04-10T05:38:19Z

Proposed commit message

Add a procfs backend to the auditbeat add_session_metadata processor. For systems that don't support ebpf, this procfs backend can be used instead to enrich auditbeat events with the data needed for session view.

However the backend is expected to be less reliable than the ebpf backend; if processes exit before their info is read from procfs, the info will not be available, and some processes will be not be enriched (particularly very short-lived processes).

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

In the auditbeat config file, add below lines to enable the add_session_metadata processor, with forced procfs backend.

- module: auditd
  # Load audit rules from separate files. Same format as audit.rules(7).
  processors:
    - add_session_metadata:
        backend: "procfs"

When running and sending data to a supported Kibana version, the session viewer should show events from auditbeat, powered by procfs.

Related issues

Use cases

Screenshots

Logs

This processor will enrich process events with additional infomation needed to enable session view in Kibana. This processor can be run on Linux systems, and will use eBPF to enrich auditd events for process exec and exit events. The additional fields that will be added are information on process parent, session leader and process group leader.

Calculate and append entry leader information to enriched processes.

…r_ebpf

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Remove the DB interface, as there will only be one implementation for it

…r_ebpf

Add a procfs provider to the add_session_view processor, which can be used to gather session metadata on systems where the ebpf implementation is not supported.

* Changed to use time.Duration in timeutils for process start NS * Used go-cmp library to compare ECS docs in tests

Fix linter warnings and upgrade go-libaudit to v2.5.0

…r_ebpf

ebpfevents library has been updated to use a single channel. Updated to use latest ebpfevents library and the single channel.

Use watcher, which provides singleton access for ebpfevents

Co-authored-by: Mattia Meleleo <melmat@tuta.io>

…r_ebpf

… into session_view_processor_ebpf

…r_ebpf

Remove possibe panics in program initialization, and handle unexpected events more gracefully.

…r_ebpf

…o session_view_processor_procfs

If any process ancestry is incomplete, retry scraping the info from proc. As procfs scraping can miss events, or not be updated when process re-parenting happens, if any inconsistancy in the DB is found, rescrape to update the data.

…r_procfs

elasticmachine · 2024-04-10T05:47:38Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-04-15T17:27:56.038+0000
Duration: 47 min 11 sec

Test stats 🧪

Test	Results
Failed	0
Passed	467
Skipped	60
Total	527

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

x-pack/auditbeat/processors/sessionmd/provider/procfs_provider/procfs_provider_test.go

elasticmachine · 2024-04-12T20:43:34Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

elasticmachine · 2024-04-12T20:43:34Z

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

…r_procfs

mergify · 2024-04-15T09:25:55Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b session_view_processor_procfs upstream/session_view_processor_procfs
git merge upstream/main
git push upstream session_view_processor_procfs

…r_procfs

mjwolf and others added 30 commits January 10, 2024 10:53

Add unit tests for add_session_metadata

d5a01bf

Calculate process entry leader

7cec455

Calculate and append entry leader information to enriched processes.

Merge remote-tracking branch 'origin/main' into session_view_processo…

7de480d

…r_ebpf

Add entry leader tests

782504c

Update CHANGELOG

6b7037b

Merge branch 'main' into session_view_processor_ebpf

0f610cb

Apply suggestions from code review

c688f5a

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Rework directory structure

f35b4c7

Remove DB interface

bf38e89

Remove the DB interface, as there will only be one implementation for it

Pass DB by reference in tests

910aba2

Rework entry leader tests

0cbb970

Reformat processor

06b7064

Only build tests on Linux

be57ad8

Add linux build directive to all files in processor

9ad7811

Merge remote-tracking branch 'origin/main' into session_view_processo…

f6aad7e

…r_ebpf

Add Procfs provider for add_session_view processor

3f0bdae

Add a procfs provider to the add_session_view processor, which can be used to gather session metadata on systems where the ebpf implementation is not supported.

Changes for PR feedback

3598b3c

* Changed to use time.Duration in timeutils for process start NS * Used go-cmp library to compare ECS docs in tests

Merge branch 'elastic:main' into session_view_processor_ebpf

743e8da

Add empty file to add_session_metadata package

17fdf1c

Fix linter warnings

1ab4752

Fix linter warnings and upgrade go-libaudit to v2.5.0

Merge branch 'main' into session_view_processor_ebpf

23e097c

Merge remote-tracking branch 'origin/main' into session_view_processo…

882f8a4

…r_ebpf

Use single channel from epbevents

145f627

ebpfevents library has been updated to use a single channel. Updated to use latest ebpfevents library and the single channel.

Use watcher for ebpf events

e9aea4d

Use watcher, which provides singleton access for ebpfevents

remove seccomp init

c001219

Update x-pack/auditbeat/internal/ebpf/watcher_linux.go

a5986dd

Co-authored-by: Mattia Meleleo <melmat@tuta.io>

Merge remote-tracking branch 'origin/main' into session_view_processo…

5658c76

…r_ebpf

Add seccomp policy modification

d5da140

Merge branch 'session_view_processor_ebpf' of github.com:mjwolf/beats…

ca81839

… into session_view_processor_ebpf

mjwolf added 7 commits March 11, 2024 11:20

Merge remote-tracking branch 'origin/main' into session_view_processo…

9a52b90

…r_ebpf

Remove possibilities of panics

36c8998

Remove possibe panics in program initialization, and handle unexpected events more gracefully.

Merge remote-tracking branch 'origin/main' into session_view_processo…

5f5f777

…r_ebpf

Merge remote-tracking branch 'origin/session_view_processor_ebpf' int…

b43e9bc

…o session_view_processor_procfs

Update procfs provider to work with other processor changes

afe10b2

Retry scraping process if ancestry incomplete

82572d2

If any process ancestry is incomplete, retry scraping the info from proc. As procfs scraping can miss events, or not be updated when process re-parenting happens, if any inconsistancy in the DB is found, rescrape to update the data.

Merge remote-tracking branch 'origin/main' into session_view_processo…

f2a538f

…r_procfs

mjwolf added enhancement backport-skip Skip notification from the automated backport with mergify labels Apr 10, 2024

mjwolf requested a review from a team as a code owner April 10, 2024 05:38

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 10, 2024

mergify bot assigned mjwolf Apr 10, 2024

Add changelog entry

67004c4

andrewkroh approved these changes Apr 12, 2024

View reviewed changes

x-pack/auditbeat/processors/sessionmd/provider/procfs_provider/procfs_provider_test.go Outdated Show resolved Hide resolved

x-pack/auditbeat/processors/sessionmd/provider/procfs_provider/procfs_provider_test.go Outdated Show resolved Hide resolved

andrewkroh added Team:Security-Linux Platform Linux Platform Team in Security Solution Team:Security-Deployment and Devices Deployment and Devices Team in Security Solution labels Apr 12, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 12, 2024

andrewkroh added Auditbeat needs_team Indicates that the issue/PR needs a Team:* label labels Apr 12, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 12, 2024

Merge remote-tracking branch 'origin/main' into session_view_processo…

9229f97

…r_procfs

mjwolf added 3 commits April 15, 2024 10:24

Move test cases in separate functions

16be56e

Merge remote-tracking branch 'origin/main' into session_view_processo…

4603027

…r_procfs

Merge branch 'main' into session_view_processor_procfs

db82ab7

mjwolf merged commit 1898d37 into elastic:main Apr 15, 2024
39 checks passed

mjwolf deleted the session_view_processor_procfs branch April 15, 2024 18:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Session view processor procfs #38799

Session view processor procfs #38799

mjwolf commented Apr 10, 2024 •

edited

elasticmachine commented Apr 10, 2024 •

edited

Build stats

Test stats 🧪

elasticmachine commented Apr 12, 2024

elasticmachine commented Apr 12, 2024

mergify bot commented Apr 15, 2024

Session view processor procfs #38799

Session view processor procfs #38799

Conversation

mjwolf commented Apr 10, 2024 • edited

Proposed commit message

Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Apr 10, 2024 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Apr 12, 2024

elasticmachine commented Apr 12, 2024

mergify bot commented Apr 15, 2024

mjwolf commented Apr 10, 2024 •

edited

elasticmachine commented Apr 10, 2024 •

edited