Ship audit activity to external destinations

[zhumo]

Problem

As a Fleet admin who is expanding my Fleet user base to other teams/groups/departments in my organization, I want to aggregate users' Fleet activity in an external destination built for storage and analytics purposes, so that I can analyze user activity in case of issues or set up monitoring to detect non-standard usage.

Business Case

This enables Fleet to be used by a more diverse user base. It reduces the hesitancy on the part of the Fleet admin to give more access more broadly.

Analytics

• number of Fleet installations with audit logs turned on

Requirements

• The audit log destination is essentially treated as analogous to the existing osquery_result_log_plugin option.
• There is a configuration activity_audit_log_plugin which accepts the same set of options as osquery_result_log_plugin.
• For each of the external destinations, there is a key equivalent to osquery's result_stream called activity_audit_stream, which pushes the data to a certain stream in the external destination.
• The audit log will support the same log streaming options as osquery result and status logs: filesystem, firehose, kinesis, lambda, pubsub, kafkarest, and stdout.
• This is a premium-only feature.

Documentation

• Document the format of the various audit data types (this will be tackled on a separate issue: Document audit log events generated by Fleet #9061)
• Document how to turn on/off the audit logs.

Analytics

• In usage statistics, count number of Fleet installations with this feature turned on.

--

Related

• Notes and video from call: https://docs.google.com/document/d/1C_CgbtqImxe2mvnr46yJtTTSzbqaom-7KTGoCPbyztQ/edit?usp=sharing. Relevant time period: 7:00 - 13:00

PR: #9001

Child issues

Platform team

• Implement streaming for activities #8975
• Usage statistics: count number of installations with audit log feature turned on #9060

Documentation

• See the documentation adde in the PR.

[lucasmrod]

Hi folks!

Let's define what we mean by "audit activities/logs".

Is it the activities we track in the "Activity" section?

Asking because the video mentions Fleet's stdout and stderr when talking about "audit logs".
For instance, I'm running Fleet locally and I created a test policy and fleet's stdout was empty, and stderr contained the following:

level=info ts=2022-12-08T17:13:12.987772Z component=redis mode=standalone
level=info ts=2022-12-08T17:13:13.013995Z msg="started cron schedules: automations, cleanups_then_aggregation, integrations, usage_statistics, vulnerabilities"
ts=2022-12-08T17:13:13.030212Z transport=https address=0.0.0.0:8080 msg=listening
level=info ts=2022-12-08T17:13:15.360995Z component=http path=/api/latest/fleet/device/a4900dc2-762b-48b6-acce-d829feefe636/desktop internal="authentication error: invalid device authentication token" err=": Authentication required"
level=error ts=2022-12-08T17:13:19.819087Z component=http user=unauthenticated method=POST uri=/api/fleet/orbit/enroll took=260.064779ms hardware_uuid=0a084d56-6f5f-fcef-f17d-dbca1c9e706b err="enroll host failed: maximum number of hosts reached: 1"
level=info ts=2022-12-08T17:13:20.363647Z component=http path=/api/latest/fleet/device/a4900dc2-762b-48b6-acce-d829feefe636/desktop internal="authentication error: invalid device authentication token" err=": Authentication required"
level=info ts=2022-12-08T17:13:25.361938Z component=http path=/api/latest/fleet/device/a4900dc2-762b-48b6-acce-d829feefe636/desktop internal="authentication error: invalid device authentication token" err=": Authentication required"
level=error ts=2022-12-08T17:13:28.860183Z cron=cleanups_then_aggregation schedule=cleanups_then_aggregation msg="unlock failed" err="context canceled"
level=error ts=2022-12-08T17:13:28.8602Z cron=integrations schedule=integrations msg="unlock failed" err="context canceled"
level=error ts=2022-12-08T17:13:28.860232Z cron=vulnerabilities schedule=vulnerabilities msg="unlock failed" err="context canceled"
[...]

which doesn't show the policy creation activity. (Fleet's stderr contains all sorts of logs that the user may not be interested in.)

[lucasmrod]

We clarified this on a call and what we want to stream to an external destination are the activities (as shown in the screenshot).

[lucasmrod]

@zhumo This only requires code changes in #platform. Shall I edit the description to remove the remaining TODOs?

[zhumo]

Yes please go ahead!

[lucasmrod]

@zhumo To not cause issues (and/or degrade performance) with admin operations that will generate activities (ideally all), we'll stream the activity externally on a separate asynchronous job.
Is a delay, of, say, 5 minutes on the external streaming acceptable? (In other words: The activity will be visible in the UI instantly as usual, but the streaming to the external destination may take up to 5 minutes.)

@chiiph

[zhumo]

That’s fine but please document.

On Thu, Dec 15, 2022 at 8:54 AM Lucas Manuel Rodriguez < ***@***.***> wrote: @zhumo <https://github.com/zhumo> To not cause issues (and/or degrade performance) with admin operations that will generate activities (ideally all), we'll stream the activity externally on a separate asynchronous job. Is a delay, of, say, 5 minutes on the external streaming acceptable? (In other words: The activity will be visible in the UI instantly as usual, but the streaming to the external destination may take up to 5 minutes.) @chiiph <https://github.com/chiiph> — Reply to this email directly, view it on GitHub <#8416 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABI4PYVXSG7PF2NTFE5OW3TWNMPH7ANCNFSM6AAAAAARL43P2U> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Mo Zhu* Head of Product fleetdm.com <http://www.fleetdm.com> | @fleetctl <https://twitter.com/fleetctl>

[zhumo]

#9001

[noahtalerman]

Re opening this issue because the feature hasn't been released.

[fleet-release]

/docs/configuration/config.md)

Auditing activity

• Easily done
• Stream to clouds
• Graphs to analyze

[fleet-release]

/docs/content/admin/configuration.md#activity-audit-log-plugin) for more details.

A distant stream,
Audit logs, insights gleaned,
Fleet usage soars.