[jamf_compliance_reporter] JAMF Compliance Reporter. #3210

vinit-chauhan · 2022-04-27T14:51:35Z

What does this PR do?

Generated the skeleton of the JAMF Compliance Reporter integration package.
Added a data stream.
Added data collection logic to the data stream.
Added the ingest pipeline for the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
Added dashboards and visualizations.
Added test for pipeline for the data stream.
Added system test cases for the data stream.

NOTE: This integration has been tested using a mock server and with sample events only.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.17.0 || ^8.0.0).

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/jamf_compliance_reporter directory.
Run the following command to run tests.

elastic-package test

Screenshots

elasticmachine · 2022-04-27T14:52:19Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-04-27T15:05:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-05T12:42:16.015+0000
Duration: 17 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	11
Skipped	0
Total	11

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-04-27T15:06:09Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`47/47`)	💚 3.263
Classes	100.0% (`47/47`)	💚 3.263
Methods	100.0% (`156/156`)	💚 10.97
Lines	99.297% (`2542/2560`)	👍 9.419
Conditionals	100.0% (`0/0`)	💚

r00tu53r

I've only looked at audit data stream. Looking good have a few comments around mapping to ECS fields etc.

In general since this is for macos. The main pipeline could also set host.os.type to macos.

packages/jamf_compliance_reporter/_dev/build/docs/README.md

packages/jamf_compliance_reporter/changelog.yml

packages/jamf_compliance_reporter/manifest.yml

packages/jamf_compliance_reporter/data_stream/audit/manifest.yml

packages/jamf_compliance_reporter/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

vinit-chauhan · 2022-04-29T05:41:49Z

Hey @r00tu53r, We are looking into the field mapping changes which you suggested. We will get back to you with the updates.

andrewkroh · 2022-05-06T20:57:42Z

Missing some event.* fields.

jamf_compliance_reporter/audit Verify sample_event.json:
[0] field "event.outcome" is undefined
[1] field "event.code" is undefined

Duplicate field definitions.

packages/jamf_compliance_reporter/data_stream/app_metrics
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/ecs.yml:17 - host.id : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/agent.yml:105 - host.id : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/agent.yml:153 - host.os.version : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/ecs.yml:23 - host.os.version : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/ecs.yml:15 - host.hostname : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/agent.yml:100 - host.hostname : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/ecs.yml:19 - host.mac : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/app_metrics/fields/agent.yml:114 - host.mac : [Duplicate field (2 times).]

packages/jamf_compliance_reporter/data_stream/audit
packages/jamf_compliance_reporter/data_stream/audit/fields/agent.yml:100 - host.hostname : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/ecs.yml:15 - host.hostname : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/agent.yml:153 - host.os.version : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/ecs.yml:25 - host.os.version : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/ecs.yml:17 - host.id : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/agent.yml:105 - host.id : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/ecs.yml:19 - host.ip : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/agent.yml:110 - host.ip : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/ecs.yml:21 - host.mac : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/audit/fields/agent.yml:114 - host.mac : [Duplicate field (2 times).]

packages/jamf_compliance_reporter/data_stream/event
packages/jamf_compliance_reporter/data_stream/event/fields/agent.yml:100 - host.hostname : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/ecs.yml:21 - host.hostname : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/ecs.yml:31 - host.os.version : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/agent.yml:153 - host.os.version : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/agent.yml:110 - host.ip : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/ecs.yml:25 - host.ip : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/agent.yml:114 - host.mac : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/ecs.yml:27 - host.mac : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/agent.yml:105 - host.id : [Duplicate field (2 times).]
packages/jamf_compliance_reporter/data_stream/event/fields/ecs.yml:23 - host.id : [Duplicate field (2 times).]

Please make sure the MAC addresses are valid. See my comment at #3232 (comment).

jamf_compliance_reporter/event test-pipeline-xprotect-event-log.log:
[0] parsing field value failed: field "host.mac"'s value, 38-X9-X8-15-5X-82, does not match the expected pattern: ^[A-F0-9]{2}(-[A-F0-9]{2}){5,}$

andrewkroh

Can you please update the dashboards to embed the visualizations into the dashboards.

andrewkroh · 2022-05-06T21:00:15Z

...s/jamf_compliance_reporter/data_stream/app_metrics/elasticsearch/ingest_pipeline/default.yml

+      ignore_missing: true
+  - script:
+      lang: painless
+      if: ctx?.json?.app_metric_info?.cpu_percentage != null


Suggested change

if: ctx?.json?.app_metric_info?.cpu_percentage != null

if: ctx.json?.app_metric_info?.cpu_percentage != null

ctx is always going to be present. Can you apply this change everywhere.

…lastic/integrations into package_jamf_compliance_reporter

r00tu53r

Sorry for taking long to come back to this. Thank you for making the earlier suggested changes. Its looking good.
I see a few more cases where I have comments around ECS and duplication of data.

In general avoid duplicate storage. If a value is stored in ECS then no need to store it in the field group.
Convert access_mode to octal representation so users can search 755 or 666 etc. instead of 33261.
All the syscall audit events seem to have header, host_info, identity, return, subject sections. Perhaps the pipeline can be arranged to take advantage of this ?
The subject group fields can be mapped to ECS as shown below.

subject
{
   "audit_id"|"user_id" --> process.real_user.id
   "audit_user_name"|"user_name" --> process.real_user.name
   "effective_group_id": process.group.id
   "effective_group_name": --> process.group.name
   "effective_user_id": --> process.user.id
   "effective_user_name":  --> process.user.name
   "group_id": -> process.real_group.id
   "group_name": --> process.real_group.name
}

5.I saw instances of PID (Process ID) in some pipelines which can be mapped to process.pid or equivalent.

r00tu53r · 2022-05-23T05:13:38Z

packages/jamf_compliance_reporter/_dev/build/docs/README.md

+The [Jamf Compliance Reporter](https://docs.jamf.com/compliance-reporter/documentation/Compliance_Reporter_Overview.html) Integration collects and parses data received from Jamf Compliance Reporter using TLS or HTTP Endpoint.  
+
+## Requirements
+- Enable the Integration with the TLS or HTTP Endpoint input.


The integration involves making configuration changes on two sides Elastic and Jamf. That could be made clearer in the documentation. The doc could state what needs to be done and then point to the exact documentation on Jamf so the user can view the how part. You could refer to this doc for the headings etc. that have been used in other packages too.

r00tu53r · 2022-05-23T05:20:12Z

packages/jamf_compliance_reporter/_dev/build/docs/README.md

+
+## Setup Steps
+
+- After validating settings, you can use a configuration profile in Jamf Pro to deploy certificates to endpoints in production.


It might be a good idea to take the user through a series of steps and point them to the exact Jamf link / page for each step that needs to be performed in that order. Also I see the Remote Endpoint Logging Preference Keys page here shows a list of configurations. I presume that user must choose generic TLS Logging endpoint ? If so can that be explicitly specified in the steps.

r00tu53r · 2022-05-23T06:02:49Z

...s/jamf_compliance_reporter/data_stream/app_metrics/elasticsearch/ingest_pipeline/default.yml

+      ignore_missing: true
+  - date:
+      field: json.header.time_seconds_epoch
+      target_field: '@timestamp'


nit: @timestamp is the default target for the date processor.

r00tu53r · 2022-05-23T06:14:54Z

...s/jamf_compliance_reporter/data_stream/app_metrics/elasticsearch/ingest_pipeline/default.yml

+  - convert:
+      field: json.app_metric_info.cpu_percentage
+      type: double
+      target_field: jamf_compliance_reporter.app_metrics.app_metric_info.cpu_percentage
+      ignore_failure: true 
+  - convert:
+      field: json.app_metric_info.cpu_time_seconds
+      type: double
+      target_field: jamf_compliance_reporter.app_metrics.app_metric_info.cpu_time_seconds
+      ignore_failure: true
+  - convert:
+      field: json.app_metric_info.interrupt_wakeups
+      type: long
+      target_field: jamf_compliance_reporter.app_metrics.app_metric_info.interrupt_wakeups
+      ignore_failure: true
+  - convert:
+      field: json.app_metric_info.platform_idle_wakeups
+      type: long
+      target_field: jamf_compliance_reporter.app_metrics.app_metric_info.platform_idle_wakeups
+      ignore_failure: true
+  - convert:
+      field: json.app_metric_info.resident_memory_size_mb
+      type: double
+      target_field: jamf_compliance_reporter.app_metrics.app_metric_info.resident_memory_size_mb
+      ignore_failure: true
+  - convert:
+      field: json.app_metric_info.virtual_memory_size_mb
+      type: double
+      target_field: jamf_compliance_reporter.app_metrics.app_metric_info.virtual_memory_size_mb
+      ignore_failure: true


I wonder if it is worth making app_metric_info a flattened field and storing the values as-is. Unless there is a plan to raise an alert for configured CPU thresholds.

r00tu53r · 2022-05-23T06:18:39Z

...s/jamf_compliance_reporter/data_stream/app_metrics/elasticsearch/ingest_pipeline/default.yml

+      field: json.header.event_name
+      target_field: jamf_compliance_reporter.header.event_name
+      ignore_missing: true
+  - set:


Could be renamed instead of set as it gets duplicated.

r00tu53r · 2022-05-23T08:21:13Z

packages/jamf_compliance_reporter/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      ignore_failure: true
+  - rename:
+      field: json.host_info.host_uuid
+      target_field: host.id


Saving serial_number in host.id might be more useful as its more likely for users to query by serial number than UUID

r00tu53r · 2022-05-23T08:24:04Z

...mpliance_reporter/data_stream/audit/elasticsearch/ingest_pipeline/pipeline_aue_ssauthint.yml

+  - remove:
+      field: json.arguments.known_UID_
+      ignore_missing: true


Is this required ? The json object is being removed in the end during cleanup.

r00tu53r · 2022-05-23T08:26:55Z

...f_compliance_reporter/data_stream/audit/elasticsearch/ingest_pipeline/pipeline_aue_wait4.yml

+processors:
+  - convert:
+      field: json.arguments.pid
+      target_field: jamf_compliance_reporter.audit.arguments.pid


Can go into process.pid

r00tu53r · 2022-05-23T08:27:36Z

...eporter/data_stream/audit/elasticsearch/ingest_pipeline/pipeline_exec_chain_child_object.yml

+      ignore_missing: true
+  - convert:
+      field: json.exec_chain_child.parent_pid
+      target_field: jamf_compliance_reporter.audit.exec_chain_child.parent.pid


can go into process.parent.pid

r00tu53r · 2022-05-23T08:31:43Z

...pliance_reporter/data_stream/audit/elasticsearch/ingest_pipeline/pipeline_process_object.yml

@@ -0,0 +1,130 @@
+---


Many of the fields here jamf_compliance_reporter.audit.process.* can go into process.*, process.user.*, process.group.* etc.

vinit-chauhan · 2022-06-02T05:53:13Z

Hey @r00tu53r, Sorry for the delayed response. We are working on the changes.

r00tu53r

Thank you for making the changes. LGTM

packages/jamf_compliance_reporter/_dev/build/build.yml

packages/jamf_compliance_reporter/_dev/build/docs/README.md

packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/default.yml

...es/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_audit.yml

...f_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_app_metrics.yml

...iance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_license_info_event.yml

...compliance_reporter/data_stream/log/_dev/test/pipeline/test-pipeline-audit.log-expected.json

...liance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_unified_log_event.yml

packages/jamf_compliance_reporter/data_stream/log/fields/fields.yml

andrewkroh · 2022-07-06T23:41:24Z

packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+processors:
+  - set:
+      field: ecs.version
+      value: '8.2.0'


This should match 8.3.0 in the build manifest.

Initial commit for JAMF Compliance Reporter.

974c8b3

vinit-chauhan added the Team:Security-External Integrations label Apr 27, 2022

vinit-chauhan added the New Integration label Apr 27, 2022

Updated changelog entry.

9e6390c

vinit-chauhan force-pushed the package_jamf_compliance_reporter branch from 14814bf to 9e6390c Compare April 27, 2022 15:54

andrewkroh added the Integration:JAMF_Compliance_Reporter label Apr 27, 2022

andrewkroh requested a review from a team April 27, 2022 17:46

r00tu53r reviewed Apr 28, 2022

View reviewed changes

vinit-chauhan changed the title ~~[JAMF Compliance Reporter] Initial commit for JAMF Compliance Reporter.~~ [jamf_compliance_reporter] Initial commit for JAMF Compliance Reporter. Apr 29, 2022

Changes as per comments.

b0ec674

vinit-chauhan requested a review from r00tu53r May 5, 2022 14:51

andrewkroh reviewed May 6, 2022

View reviewed changes

vinit-chauhan and others added 4 commits May 12, 2022 12:52

Updated as per comments.

05800f8

Merge branch 'main' into package_jamf_compliance_reporter

b44e51f

Merge branch 'package_jamf_compliance_reporter' of github.com:vinit-e…

edfb36a

…lastic/integrations into package_jamf_compliance_reporter

Update changelog entry.

ef768bf

jamiehynds mentioned this pull request May 20, 2022

Support for MacOS Unified Logging #6589

Open

r00tu53r reviewed May 23, 2022

View reviewed changes

Update as per comment.

40d1d7a

vinit-chauhan marked this pull request as draft June 9, 2022 08:33

Merged all the data streams to one.

9d38af2

vinit-chauhan marked this pull request as ready for review June 10, 2022 15:17

vinit-chauhan marked this pull request as draft June 13, 2022 04:39

Merged two data streams.

5c447d6

vinit-chauhan marked this pull request as ready for review June 16, 2022 05:04

vinit-chauhan requested review from r00tu53r and andrewkroh June 16, 2022 05:04

r00tu53r approved these changes Jun 16, 2022

View reviewed changes

efd6 reviewed Jul 5, 2022

View reviewed changes

Update as per review comments

3af5ddc

andrewkroh approved these changes Jul 6, 2022

View reviewed changes

andrewkroh merged commit 35d3583 into elastic:main Jul 6, 2022

epixa changed the title ~~[jamf_compliance_reporter] Initial commit for JAMF Compliance Reporter.~~ [jamf_compliance_reporter] JAMF Compliance Reporter. Jul 12, 2022

andrewkroh mentioned this pull request Jul 13, 2022

[production] Promote packages from snapshot (cef-2.2.0, cisco_asa-2.5.1, cisco_secure_endpoint-2.5.1, github-1.2.0, httpjson-1.4.1, jamf_compliance_reporter-0.1.0, suricata-2.3.0) elastic/package-storage#4866

Merged

vinit-chauhan mentioned this pull request Sep 29, 2022

[lastpass] Initial Release for the LastPass #4254

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jamf_compliance_reporter] JAMF Compliance Reporter. #3210

[jamf_compliance_reporter] JAMF Compliance Reporter. #3210

vinit-chauhan commented Apr 27, 2022 •

edited

Loading

elasticmachine commented Apr 27, 2022

elasticmachine commented Apr 27, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 27, 2022 •

edited

Loading

r00tu53r left a comment •

edited

Loading

vinit-chauhan commented Apr 29, 2022

andrewkroh commented May 6, 2022

andrewkroh left a comment

andrewkroh May 6, 2022

r00tu53r left a comment •

edited

Loading

r00tu53r May 23, 2022

r00tu53r May 23, 2022

r00tu53r May 23, 2022 •

edited

Loading

r00tu53r May 23, 2022

r00tu53r May 23, 2022

r00tu53r May 23, 2022

r00tu53r May 23, 2022

r00tu53r May 23, 2022

r00tu53r May 23, 2022

r00tu53r May 23, 2022

vinit-chauhan commented Jun 2, 2022

r00tu53r left a comment

andrewkroh Jul 6, 2022

	if: ctx?.json?.app_metric_info?.cpu_percentage != null
	if: ctx.json?.app_metric_info?.cpu_percentage != null


		## Setup Steps

		- After validating settings, you can use a configuration profile in Jamf Pro to deploy certificates to endpoints in production.

[jamf_compliance_reporter] JAMF Compliance Reporter. #3210

[jamf_compliance_reporter] JAMF Compliance Reporter. #3210

Conversation

vinit-chauhan commented Apr 27, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Screenshots

elasticmachine commented Apr 27, 2022

elasticmachine commented Apr 27, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Apr 27, 2022 • edited Loading

🌐 Coverage report

r00tu53r left a comment • edited Loading

Choose a reason for hiding this comment

vinit-chauhan commented Apr 29, 2022

andrewkroh commented May 6, 2022

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

r00tu53r left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

r00tu53r May 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinit-chauhan commented Jun 2, 2022

r00tu53r left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinit-chauhan commented Apr 27, 2022 •

edited

Loading

elasticmachine commented Apr 27, 2022 •

edited

Loading

elasticmachine commented Apr 27, 2022 •

edited

Loading

r00tu53r left a comment •

edited

Loading

r00tu53r left a comment •

edited

Loading

r00tu53r May 23, 2022 •

edited

Loading