Osquerybeat #24456

aleksmaus · 2021-03-10T03:13:41Z

What does this PR do?

This PR is the first cut of the agent actions with osquerybeat. This allows agent to run osquerybeat, dispatch the ad-hoc queries received with "agent actions" and send the results back.
The osquerybeat was migrated from https://github.com/elastic/osquerybeat, the original POC we worked on awhile back.

Here is how the actions work with this implementation:

The agent starts the apps (osquerybeat in particular)
The osquerybeat receives the configuration from the agent
The osquerybeat parses configuration and sends the supported input types and osqueryd version back to the agent as a payload with the status update. Along with the payload it also sends the version of osqueryd binary that is shipped with the agent. The requirement for the product is to be able to surface this information to the end-users.
The osquerybeat also registers the action handlers with the rpc client for it's input types (in this case the input type is "osquery")
The agent received the input types in the status update payload and updates inputTypes set of the corresponding ApplicationState
When the new action is received from the Fleet Server, the Agent finds the matching ApplicationState for the input type and dispatches the action.
The result of the action is forwarded back to Fleet Server. The query result data is send directly to elasticsearch by osquerybeat.

Packaging:

The osquerybeat build downloads the official distro from https://osquery.io/downloads/official and caches it locally for incremental builds.
The official distro binaries are provided as:

.tar.gz for linux
.pkg for MacOS
.msi for Winderz

For osquerybeat package the osqueryd binary is extracted from the official distro(the only thing that we need) for the linux package. For windows and Mac the .pkg or .msi file is included into the package.
When osquerybeat if ran for the first time, depending on the platform it either unpacks the the osqueryd from the distro on that platform first or starts osqueryd as a child process right away.

Disclaimer:

This is my first attempt on digging through the agent code, so there is a possibility that the lines of responsibilities for different pieces of the code are not exactly where they intended to be as designed by the original contributors. Would need some feedback.
Also not sure I covered all the bases with CI build for osquerybeat to get it correctly published to artifactory etc.
Might need some advice this.
Fully expect to iterate on osquerybeat customization further after this.

Why is it important?

This allows the security assets management team to ship osquery integration in the upcoming release.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Prerequisites:

The osquerybeat-8.0.0-SNAPSHOT-[platform]-x86_64.tar.gz and corresponding .sha512 files needs to be dropped into the agent download folder, until the oquerybeat is published to artifactory.
The Osquery Elastic Managed integration needs to be added to the agent policy.

The agent needs to enrolled with the latest Fleet Server (under developement)

Related issues

Closes https://github.com/elastic/security-team/issues/875

Sample action document with the query:

Corresponding action result document:

Corresponding query result datastream:

elasticmachine · 2021-03-10T03:13:47Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2021-03-10T03:24:44Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24456 updated
Start Time: 2021-03-26T12:51:42.844+0000
Duration: 136 min 32 sec
Commit: a3de591

Test stats 🧪

Test	Results
Failed	0
Passed	46429
Skipped	5104
Total	51533

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	46429
Skipped	5104
Total	51533

urso · 2021-03-11T13:34:03Z

Is it possible to split the PR such that we can review the libbeat management changes independent of osquerybeat, and also independent of the changes to packaging?

aleksmaus · 2021-03-11T13:50:47Z

Is it possible to split the PR such that we can review the libbeat management changes independent of osquerybeat, and also independent of the changes to packaging?

Sure. I did this in the same PR initially to make it easier to change if I need to move more of the stuff into libbeat based on the feedback. Will create two new PRs referring to this one.

scunningham

Looks good. Couple minor security questions.

x-pack/elastic-agent/pkg/core/server/server.go

x-pack/osquerybeat/beater/osquerybeat.go

scunningham · 2021-03-15T15:16:12Z

x-pack/osquerybeat/beater/osquerybeat.go

+	// Create temp directory for socket and possibly other things
+	// The unix domain socker path is limited to 108 chars and would
+	// not always be able to create in subdirectory
+	tmpdir, removeTmpDir, err := createTempDir()


Try to use unguessable dir name with minimal permissions; eg. uuidv4.

x-pack/osquerybeat/beater/osquerybeat.go

scunningham · 2021-03-15T15:23:56Z

x-pack/osquerybeat/beater/scheduler.go

+
+func (s *Scheduler) isCancelled() bool {
+	select {
+	case <-s.ctx.Done():


can also do:

return s.ctx.Err() != nil

x-pack/osquerybeat/internal/distro/distro.go

x-pack/osquerybeat/internal/fetch/fetch.go

x-pack/osquerybeat/internal/osqueryd/osqueryd.go

Based on conversation on PR made the change so now the input type are not passed back to the agent. Instead they are derived by the agent from the application spec. This is the additional property that was added to the spec to declare accepted action_input_types: action_input_types: - osquery

* Try to use /var/run directory for a socket, fallback on platform specific temp dir if access denied (running as non-root) * Upgrade to build with osquery 4.7.0

v1v · 2021-03-23T10:47:28Z

x-pack/osquerybeat/Jenkinsfile.yml

@@ -0,0 +1,67 @@
+when:


Is it required ARM support? If so, the ARM stage is missing similar to the packaging stage.

This currently uses the official distro of osquery, and at the time this project started there were no ARM official distro. The linux arm distro is now available in beta https://osquery.io/blog/linux-arm64-beta-support, could add it later.

Another and possibly better option for GA is to build osquery for all the platforms from source and host in our artifactory. This would allow us to avoid packaging official distros .msi or .pkg for windows and mac into the distribution and build any flavor we need possibly earlier:
https://osquery.readthedocs.io/en/latest/development/building/

Anybody can assist with settings this up on our CI infrastructure?

One of the downsides of building osquery ourselves: we would have to figure out any issues for platforms they don't officially support, we would have to sign the binaries ourselves, possible extra efforts with testing.

Anybody can assist with settings this up on our CI infrastructure?

If the signing is required, the unified release process supports such a service, though we need to run the build in another CI. But if required, we can start a conversation with the team unified release team about where to host those binaries.

@aleksmaus I think we can skip arm64 for 7.13 since it's only in beta on osquery and add it later when it's out of beta.

yeah, that's the plan: to land what we have and then iterate.

v1v · 2021-03-23T16:02:38Z

Build&Test / metricbeat-goIntegTest / TestData – github.com/elastic/beats/v7/metricbeat/module/kafka/broker

As far as I see those are unrelated to these changes, though it was triggered as a consequence of changing some files that forced to run the whole pipeline similar to the one triggered in any of the branches.

v1v

+1 from the CI point of view, I cannot say anything else about the other changes

aleksmaus · 2021-03-23T16:11:39Z

Build&Test / metricbeat-goIntegTest / TestData – github.com/elastic/beats/v7/metricbeat/module/kafka/broker

As far as I see those are unrelated to these changes, though it was triggered as a consequence of changing some files that forced to run the whole pipeline similar to the one triggered in any of the branches.

So for initial release we are going to rely on the official osquery binaries it looks like. The official binaries support 64bit windows/linux/mac. They are properly signed for MacOS for example.
There are no official builds for 32bit binaries and there is a beta release for linux ARM 64bit.

Not sure if there is a way to have a different agent specs that would not include osquerybeat for these unsupported platforms and architectures for the agent for now. Or if there is a better way to handle it.
@blakerouse @urso ?

blakerouse · 2021-03-23T17:20:52Z

@aleksmaus From the build of the osquerybeat itself I do not know. When it comes to Agent you can limit where an Application (aka. osquerybeat) can run based on a EQL condition.

See https://github.com/elastic/beats/blob/master/x-pack/elastic-agent/spec/endpoint.yml#L73 (endpoint currently doesn't support arm64)

aleksmaus · 2021-03-23T18:40:02Z

just did a quick test the osqueryd amd64 runs on Apple ARM architecture. So this PR should just work as is if we enable ARM. For linux ARM we can pick up a beta ARM distro. Easy to update after the initial merge.

aleksmaus · 2021-03-25T15:10:43Z

[2021-03-25T13:09:36.127Z] {"log.level":"error","@timestamp":"2021-03-25T13:09:33.905Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-03-25T13:09:33Z: type: 'ERROR': sub_type: 'FAILED' message: Application: endpoint-security--8.0.0-SNAPSHOT[fca78797-9936-4851-b91c-a75ba2f104a2]: State changed to FAILED: 2 errors occurred:\n\t* package '/opt/Elastic/Agent/data/elastic-agent-cfa785/downloads/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz' not found: open /opt/Elastic/Agent/data/elastic-agent-cfa785/downloads/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz: no such file or directory\n\t* fetching package failed: Get \"https://artifacts.elastic.co/downloads/endpoint-dev/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz\": x509: certificate signed by unknown authority\n\n","ecs.version":"1.6.0"}

@v1v? @blakerouse ?

The PR was up to date with the latest master as of yesterday.

v1v · 2021-03-25T15:34:59Z

@aleksmaus

what do you need from my side?

aleksmaus · 2021-03-25T15:39:10Z

@aleksmaus

what do you need from my side?

any advice how to make it green? :-)
it seems to be failing on

downloads/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz: no such file or directory

doesn't look related to the osquerybeat anyhow

urso · 2021-03-26T16:33:58Z

The PR can be merged already. The E2E test failures (all on metricbeat) are unfortunate, but currently no blocker.

* Osquerybeat with Agent actions supported * Revert grpc upgrade in this PR back to what it was before v1.29.1 * Make check happy * Check in forgotten spec file, regenerate the spec * Some regenerated after clean build fields * Agent Actions: Part 1 of Osquerybeat with Agent actions * Rollback some mods upgrade. Address some code review feedback * Address code review feedback * Add missing copyright header * Address more of the code review feedback * Remove input types from payload communicated back to the agent * Change the way the inputs are tied to the applications Based on conversation on PR made the change so now the input type are not passed back to the agent. Instead they are derived by the agent from the application spec. This is the additional property that was added to the spec to declare accepted action_input_types: action_input_types: - osquery * Address code review feedback * Try to use /var/run directory for a socket, fallback on platform specific temp dir if access denied (running as non-root) * Upgrade to build with osquery 4.7.0 * Update CI scripts to get osquerybeat building * Exclude arm64 from running osquery for now (cherry picked from commit 591685e)

Cherry-pick of PR #24456 to 7.x branch. Original message: ## What does this PR do? This PR is the first cut of the agent actions with osquerybeat. This allows agent to run osquerybeat, dispatch the ad-hoc queries received with "agent actions" and send the results back. The osquerybeat was migrated from https://github.com/elastic/osquerybeat, the original POC we worked on awhile back. Here is how the actions work with this implementation: 1. The agent starts the apps (osquerybeat in particular) 2. The osquerybeat receives the configuration from the agent 3. The osquerybeat parses configuration and sends the supported input types and osqueryd version back to the agent as a payload with the status update. Along with the payload it also sends the version of osqueryd binary that is shipped with the agent. The requirement for the product is to be able to surface this information to the end-users. 4. The osquerybeat also registers the action handlers with the rpc client for it's input types (in this case the input type is "osquery") 5. The agent received the input types in the status update payload and updates inputTypes set of the corresponding ApplicationState 6. When the new action is received from the Fleet Server, the Agent finds the matching ApplicationState for the input type and dispatches the action. 7. The result of the action is forwarded back to Fleet Server. The query result data is send directly to elasticsearch by osquerybeat. #### Packaging: The osquerybeat build downloads the official distro from https://osquery.io/downloads/official and caches it locally for incremental builds. The official distro binaries are provided as: * .tar.gz for linux * .pkg for MacOS * .msi for Winderz For osquerybeat package the osqueryd binary is extracted from the official distro(the only thing that we need) for the linux package. For windows and Mac the .pkg or .msi file is included into the package. When osquerybeat if ran for the first time, depending on the platform it either unpacks the the osqueryd from the distro on that platform first or starts osqueryd as a child process right away. #### Disclaimer: This is my first attempt on digging through the agent code, so there is a possibility that the lines of responsibilities for different pieces of the code are not exactly where they intended to be as designed by the original contributors. Would need some feedback. Also not sure I covered all the bases with CI build for osquerybeat to get it correctly published to artifactory etc. Might need some advice this. Fully expect to iterate on osquerybeat customization further after this.

Osquerybeat with Agent actions supported

372ffca

aleksmaus added the enhancement label Mar 10, 2021

aleksmaus requested review from ph, ruflin, scunningham, blakerouse and james-elastic March 10, 2021 03:13

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Ingest Management labels Mar 10, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 10, 2021

ruflin requested a review from urso March 10, 2021 13:16

aleksmaus added 5 commits March 10, 2021 08:42

Revert grpc upgrade in this PR back to what it was before v1.29.1

f19c269

Make check happy

d587960

Check in forgotten spec file, regenerate the spec

be4841e

Merge branch 'master' into osquerybeat

16fe722

Some regenerated after clean build fields

43e12de

Agent Actions: Part 1 of Osquerybeat with Agent actions

a501a06

aleksmaus mentioned this pull request Mar 11, 2021

Agent Actions: Part 1 of Osquerybeat with Agent actions #24507

Merged

6 tasks

Rollback some mods upgrade. Address some code review feedback

daf1586

scunningham suggested changes Mar 15, 2021

View reviewed changes

aleksmaus added 4 commits March 15, 2021 12:09

Address code review feedback

7410707

Add missing copyright header

6a2efbb

Address more of the code review feedback

2f82268

Merge branch 'feature/osquerybeat_agent_actions' into osquerybeat

28114df

ph added Team:Elastic-Agent Label for the Agent team and removed Team:Ingest Management labels Mar 15, 2021

aleksmaus added 5 commits March 16, 2021 21:47

Merge branch 'master' into osquerybeat

7d3f333

Merge branch 'master' into osquerybeat

c2f27f9

Address code review feedback

271b143

* Try to use /var/run directory for a socket, fallback on platform specific temp dir if access denied (running as non-root) * Upgrade to build with osquery 4.7.0

Update CI scripts to get osquerybeat building

e769b9b

aleksmaus requested a review from a team as a code owner March 23, 2021 01:39

botelastic bot added the Team:Automation Label for the Observability productivity team label Mar 23, 2021

v1v reviewed Mar 23, 2021

View reviewed changes

scunningham approved these changes Mar 23, 2021

View reviewed changes

v1v approved these changes Mar 23, 2021

View reviewed changes

Exclude arm64 from running osquery for now

71c6dd3

Merge branch 'master' into osquerybeat

cfa7854

aleksmaus changed the title ~~Osquerybeat with Agent actions~~ Osquerybeat Mar 25, 2021

Merge branch 'master' into osquerybeat

d523dad

Merge branch 'master' into osquerybeat

a3de591

james-elastic approved these changes Mar 26, 2021

View reviewed changes

aleksmaus merged commit 591685e into elastic:master Mar 26, 2021

aleksmaus mentioned this pull request Apr 6, 2021

Cherry-pick #24456 to 7.x: Osquerybeat #24947

Merged

6 tasks

aleksmaus added the v7.13.0 label Apr 6, 2021

andresrc mentioned this pull request Jul 27, 2021

No changelog for 8.0.0-alpha1 #27063

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Osquerybeat #24456

Osquerybeat #24456

aleksmaus commented Mar 10, 2021

elasticmachine commented Mar 10, 2021

elasticmachine commented Mar 10, 2021 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

urso commented Mar 11, 2021

aleksmaus commented Mar 11, 2021

scunningham left a comment

scunningham Mar 15, 2021

scunningham Mar 15, 2021

v1v Mar 23, 2021

aleksmaus Mar 23, 2021 •

edited

v1v Mar 23, 2021

james-elastic Mar 23, 2021

aleksmaus Mar 23, 2021

v1v commented Mar 23, 2021

v1v left a comment

aleksmaus commented Mar 23, 2021 •

edited

blakerouse commented Mar 23, 2021

aleksmaus commented Mar 23, 2021

aleksmaus commented Mar 25, 2021

v1v commented Mar 25, 2021

aleksmaus commented Mar 25, 2021

urso commented Mar 26, 2021

Osquerybeat #24456

Osquerybeat #24456

Conversation

aleksmaus commented Mar 10, 2021

What does this PR do?

Packaging:

Disclaimer:

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Mar 10, 2021

elasticmachine commented Mar 10, 2021 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

urso commented Mar 11, 2021

aleksmaus commented Mar 11, 2021

scunningham left a comment

Choose a reason for hiding this comment

scunningham Mar 15, 2021

Choose a reason for hiding this comment

scunningham Mar 15, 2021

Choose a reason for hiding this comment

v1v Mar 23, 2021

Choose a reason for hiding this comment

aleksmaus Mar 23, 2021 • edited

Choose a reason for hiding this comment

v1v Mar 23, 2021

Choose a reason for hiding this comment

james-elastic Mar 23, 2021

Choose a reason for hiding this comment

aleksmaus Mar 23, 2021

Choose a reason for hiding this comment

v1v commented Mar 23, 2021

v1v left a comment

Choose a reason for hiding this comment

aleksmaus commented Mar 23, 2021 • edited

blakerouse commented Mar 23, 2021

aleksmaus commented Mar 23, 2021

aleksmaus commented Mar 25, 2021

v1v commented Mar 25, 2021

aleksmaus commented Mar 25, 2021

urso commented Mar 26, 2021

elasticmachine commented Mar 10, 2021 •

edited by jenkins-beats-ci bot

aleksmaus Mar 23, 2021 •

edited

aleksmaus commented Mar 23, 2021 •

edited