Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Qualys VMDR] Initial release for the Qualys VMDR #6872

Merged
merged 6 commits into from
Aug 10, 2023

Conversation

piyush-elastic
Copy link
Contributor

@piyush-elastic piyush-elastic commented Jul 7, 2023

What does this PR do?

  • Updated data collection logic for the Asset Host Detection and Knowledge Base data streams.
  • Updated the ingest pipeline for the Asset Host Detection and Knowledge Base streams.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
  • Added dashboards and visualizations.
  • Updated test for pipeline for the Asset Host Detection and Knowledge Base streams.
  • Updated system test cases for the Asset Host Detection and Knowledge Base streams.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

  • Change follows the contributing guidelines
  • Supported versions of the monitoring target is documented
  • Supported operating systems are documented (if applicable)
  • Integration or System tests exist
  • Documentation exists
  • Fields follow ECS and naming conventions
  • At least a manual test with ES / Kibana / Agent has been performed.
  • Required Kibana version set to: ^8.9.0

New Package

  • Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

  • Dashboards exists
  • Screenshots added or updated
  • Datastream filters added to visualizations

Log dataset changes

  • Pipeline tests exist (if applicable)
  • Generated output for at least 1 log file exists
  • Sample event (sample_event.json) exists

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/qualys_vmdr directory.
  • Run the following command to run tests.

elastic-package test

Screenshots

OverviewPage
IntegrationPage

Automated Test

Run test suite for the package
Run system tests for the package
2023/07/12 16:10:19 DEBUG Running system tests for data stream
2023/07/12 16:10:19 DEBUG running test with configuration 'default'
2023/07/12 16:10:19 DEBUG setting up service...
2023/07/12 16:10:19 DEBUG setting up service using Docker Compose service deployer
2023/07/12 16:10:19 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/12 16:10:20 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/12 16:10:20 DEBUG output command: /usr/bin/docker network inspect elastic-package-stack_default
2023/07/12 16:10:20 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service up --build -d
Creating network "elastic-package-service_default" with the default driver
Creating elastic-package-service_qualys_vmdr_1 ... done
2023/07/12 16:10:21 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service ps -q
2023/07/12 16:10:22 DEBUG Wait for healthy containers: 53639a06e0101425669ce52bd12f23fa83ba6a771c35b0bf45fb995f04979bae
2023/07/12 16:10:22 DEBUG output command: /usr/bin/docker inspect 53639a06e0101425669ce52bd12f23fa83ba6a771c35b0bf45fb995f04979bae
2023/07/12 16:10:22 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.10.0","Labels":{"BRANCH_NAME":"v0.10.0","GIT_SHA":"2a076c9b1acdf1c35b5f5c2f8c23904c7c2c441a","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-30_11:29","com.docker.compose.config-hash":"02b9ebf16cfdc59b25afdaa8ae7549b4ab8ac8c20554cbb7f0dc94304c805da2","com.docker.compose.container-number":"1","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"/root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"/root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker","com.docker.compose.service":"qualys_vmdr","com.docker.compose.version":"1.29.2"}},"ID":"53639a06e0101425669ce52bd12f23fa83ba6a771c35b0bf45fb995f04979bae","State":{"Status":"running","ExitCode":0,"Health":null}}
2023/07/12 16:10:22 DEBUG run command: /usr/bin/docker network connect elastic-package-stack_default elastic-package-service_qualys_vmdr_1
2023/07/12 16:10:22 DEBUG adding service container elastic-package-service_qualys_vmdr_1 internal ports to context
2023/07/12 16:10:22 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service config
2023/07/12 16:10:23 DEBUG creating test policy...
2023/07/12 16:10:23 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies
2023/07/12 16:10:26 DEBUG adding package data stream to test policy...
2023/07/12 16:10:26 DEBUG POST https://127.0.0.1:5601/api/fleet/package_policies
2023/07/12 16:10:29 DEBUG deleting old data in data stream...
2023/07/12 16:10:29 DEBUG found 0 hits in logs-qualys_vmdr.asset_host_detection-ep data stream: index_not_found_exception: no such index [logs-qualys_vmdr.asset_host_detection-ep] Status=404
2023/07/12 16:10:29 DEBUG GET https://127.0.0.1:5601/api/fleet/agents
2023/07/12 16:10:29 DEBUG filter agents using criteria: NamePrefix=docker-fleet-agent
2023/07/12 16:10:29 DEBUG found 1 enrolled agent(s)
2023/07/12 16:10:29 DEBUG GET https://127.0.0.1:5601/api/fleet/agent_policies/8158a2f0-20a0-11ee-bfcb-cb1eaabbb138
2023/07/12 16:10:29 DEBUG assigning package data stream to agent...
2023/07/12 16:10:29 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f/reassign
2023/07/12 16:10:31 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:32 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"8158a2f0-20a0-11ee-bfcb-cb1eaabbb138","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:32 DEBUG Wait until the policy (ID: 8158a2f0-20a0-11ee-bfcb-cb1eaabbb138, revision: 2) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:34 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:34 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"8158a2f0-20a0-11ee-bfcb-cb1eaabbb138","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:34 DEBUG Wait until the policy (ID: 8158a2f0-20a0-11ee-bfcb-cb1eaabbb138, revision: 2) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:36 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:36 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"8158a2f0-20a0-11ee-bfcb-cb1eaabbb138","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:36 DEBUG Wait until the policy (ID: 8158a2f0-20a0-11ee-bfcb-cb1eaabbb138, revision: 2) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:38 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:38 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"8158a2f0-20a0-11ee-bfcb-cb1eaabbb138","policy_revision":2,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:38 DEBUG Policy revision assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:38 DEBUG checking for expected data in data stream...
2023/07/12 16:10:38 DEBUG found 0 hits in logs-qualys_vmdr.asset_host_detection-ep data stream
2023/07/12 16:10:39 DEBUG found 1 hits in logs-qualys_vmdr.asset_host_detection-ep data stream
2023/07/12 16:10:39 DEBUG assert hit count expected 1, observed 1
2023/07/12 16:10:39 DEBUG reassigning original policy back to agent...
2023/07/12 16:10:39 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f/reassign
2023/07/12 16:10:40 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:41 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:41 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 7) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:43 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:43 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:43 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 7) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:45 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:45 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:45 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 7) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:47 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:10:47 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","policy_revision":7,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:10:47 DEBUG Policy revision assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:10:47 DEBUG deleting test policy...
2023/07/12 16:10:47 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies/delete
2023/07/12 16:10:50 DEBUG tearing down service...
2023/07/12 16:10:50 DEBUG tearing down service using Docker Compose runner
2023/07/12 16:10:50 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/12 16:10:50 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/12 16:10:50 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service logs
2023/07/12 16:10:51  INFO Write container logs to file: /root/integration/integrations/build/container-logs/qualys_vmdr-1689158451459870095.log
2023/07/12 16:10:51 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service down --volumes
Stopping elastic-package-service_qualys_vmdr_1 ... done
Removing elastic-package-service_qualys_vmdr_1 ... done
Removing network elastic-package-service_default
2023/07/12 16:10:52 DEBUG deleting data in data stream...
2023/07/12 16:10:52 DEBUG Running system tests for data stream
2023/07/12 16:10:52 DEBUG running test with configuration 'default'
2023/07/12 16:10:52 DEBUG setting up service...
2023/07/12 16:10:52 DEBUG setting up service using Docker Compose service deployer
2023/07/12 16:10:52 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/12 16:10:53 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/12 16:10:53 DEBUG output command: /usr/bin/docker network inspect elastic-package-stack_default
2023/07/12 16:10:53 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service up --build -d
Creating network "elastic-package-service_default" with the default driver
Creating elastic-package-service_qualys_vmdr_1 ... done
2023/07/12 16:10:54 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service ps -q
2023/07/12 16:10:55 DEBUG Wait for healthy containers: b14eda53b571bde0758edf30ac977e52979ea06c29636cf1b8844ef614171df1
2023/07/12 16:10:55 DEBUG output command: /usr/bin/docker inspect b14eda53b571bde0758edf30ac977e52979ea06c29636cf1b8844ef614171df1
2023/07/12 16:10:55 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.10.0","Labels":{"BRANCH_NAME":"v0.10.0","GIT_SHA":"2a076c9b1acdf1c35b5f5c2f8c23904c7c2c441a","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-30_11:29","com.docker.compose.config-hash":"02b9ebf16cfdc59b25afdaa8ae7549b4ab8ac8c20554cbb7f0dc94304c805da2","com.docker.compose.container-number":"1","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"/root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"/root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker","com.docker.compose.service":"qualys_vmdr","com.docker.compose.version":"1.29.2"}},"ID":"b14eda53b571bde0758edf30ac977e52979ea06c29636cf1b8844ef614171df1","State":{"Status":"running","ExitCode":0,"Health":null}}
2023/07/12 16:10:55 DEBUG run command: /usr/bin/docker network connect elastic-package-stack_default elastic-package-service_qualys_vmdr_1
2023/07/12 16:10:55 DEBUG adding service container elastic-package-service_qualys_vmdr_1 internal ports to context
2023/07/12 16:10:55 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service config
2023/07/12 16:10:55 DEBUG creating test policy...
2023/07/12 16:10:55 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies
2023/07/12 16:10:59 DEBUG adding package data stream to test policy...
2023/07/12 16:10:59 DEBUG POST https://127.0.0.1:5601/api/fleet/package_policies
2023/07/12 16:11:02 DEBUG deleting old data in data stream...
2023/07/12 16:11:02 DEBUG found 0 hits in logs-qualys_vmdr.knowledge_base-ep data stream: index_not_found_exception: no such index [logs-qualys_vmdr.knowledge_base-ep] Status=404
2023/07/12 16:11:02 DEBUG GET https://127.0.0.1:5601/api/fleet/agents
2023/07/12 16:11:02 DEBUG filter agents using criteria: NamePrefix=docker-fleet-agent
2023/07/12 16:11:02 DEBUG found 1 enrolled agent(s)
2023/07/12 16:11:02 DEBUG GET https://127.0.0.1:5601/api/fleet/agent_policies/94d94ff0-20a0-11ee-bfcb-cb1eaabbb138
2023/07/12 16:11:02 DEBUG assigning package data stream to agent...
2023/07/12 16:11:02 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f/reassign
2023/07/12 16:11:03 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:04 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"94d94ff0-20a0-11ee-bfcb-cb1eaabbb138","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:04 DEBUG Wait until the policy (ID: 94d94ff0-20a0-11ee-bfcb-cb1eaabbb138, revision: 2) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:06 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:06 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"94d94ff0-20a0-11ee-bfcb-cb1eaabbb138","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:06 DEBUG Wait until the policy (ID: 94d94ff0-20a0-11ee-bfcb-cb1eaabbb138, revision: 2) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:08 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:08 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"94d94ff0-20a0-11ee-bfcb-cb1eaabbb138","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:08 DEBUG Wait until the policy (ID: 94d94ff0-20a0-11ee-bfcb-cb1eaabbb138, revision: 2) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:10 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:10 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"94d94ff0-20a0-11ee-bfcb-cb1eaabbb138","policy_revision":2,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:10 DEBUG Policy revision assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:10 DEBUG checking for expected data in data stream...
2023/07/12 16:11:10 DEBUG found 0 hits in logs-qualys_vmdr.knowledge_base-ep data stream
2023/07/12 16:11:11 DEBUG found 1 hits in logs-qualys_vmdr.knowledge_base-ep data stream
2023/07/12 16:11:11 DEBUG assert hit count expected 1, observed 1
2023/07/12 16:11:11 DEBUG reassigning original policy back to agent...
2023/07/12 16:11:11 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f/reassign
2023/07/12 16:11:13 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:13 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:13 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 7) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:15 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:15 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:15 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 7) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:17 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:17 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:17 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 7) is assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:19 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/faeff951-d3ef-4066-8a14-00f33ab1ae6f
2023/07/12 16:11:19 DEBUG Agent data: {"id":"faeff951-d3ef-4066-8a14-00f33ab1ae6f","policy_id":"elastic-agent-managed-ep","policy_revision":7,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2023/07/12 16:11:19 DEBUG Policy revision assigned to the agent (ID: faeff951-d3ef-4066-8a14-00f33ab1ae6f)...
2023/07/12 16:11:19 DEBUG deleting test policy...
2023/07/12 16:11:19 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies/delete
2023/07/12 16:11:22 DEBUG tearing down service...
2023/07/12 16:11:22 DEBUG tearing down service using Docker Compose runner
2023/07/12 16:11:22 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/07/12 16:11:22 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1
2023/07/12 16:11:22 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service logs
2023/07/12 16:11:23  INFO Write container logs to file: /root/integration/integrations/build/container-logs/qualys_vmdr-1689158483525738014.log
2023/07/12 16:11:23 DEBUG running command: /usr/local/bin/docker-compose -f /root/integration/integrations/packages/qualys_vmdr/_dev/deploy/docker/docker-compose.yml -p elastic-package-service down --volumes
Stopping elastic-package-service_qualys_vmdr_1 ... done
Removing elastic-package-service_qualys_vmdr_1 ... done
Removing network elastic-package-service_default
2023/07/12 16:11:24 DEBUG deleting data in data stream...
--- Test results for package: qualys_vmdr - START ---
╭─────────────┬──────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE     │ DATA STREAM          │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────────┼──────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ qualys_vmdr │ asset_host_detection │ system    │ default   │ PASS   │ 19.423088285s │
│ qualys_vmdr │ knowledge_base       │ system    │ default   │ PASS   │ 18.805989189s │
╰─────────────┴──────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: qualys_vmdr - END   ---
Done
Run asset tests for the package
2023/07/12 16:11:24 DEBUG installing package...
2023/07/12 16:11:24 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages/qualys_vmdr-0.1.0
--- Test results for package: qualys_vmdr - START ---
╭─────────────┬──────────────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM          │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├─────────────┼──────────────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ qualys_vmdr │                      │ asset     │ dashboard qualys_vmdr-017c0220-1001-11ee-b28e-615808a979fd is loaded  │ PASS   │      1.703µs │
│ qualys_vmdr │                      │ asset     │ dashboard qualys_vmdr-686c4470-11b6-11ee-a722-91244a8ae892 is loaded  │ PASS   │        102ns │
│ qualys_vmdr │                      │ asset     │ search qualys_vmdr-4119cae0-100e-11ee-b28e-615808a979fd is loaded     │ PASS   │        214ns │
│ qualys_vmdr │                      │ asset     │ search qualys_vmdr-fc0b5150-125e-11ee-a722-91244a8ae892 is loaded     │ PASS   │        110ns │
│ qualys_vmdr │ asset_host_detection │ asset     │ index_template logs-qualys_vmdr.asset_host_detection is loaded        │ PASS   │        221ns │
│ qualys_vmdr │ asset_host_detection │ asset     │ ingest_pipeline logs-qualys_vmdr.asset_host_detection-0.1.0 is loaded │ PASS   │         83ns │
│ qualys_vmdr │ knowledge_base       │ asset     │ index_template logs-qualys_vmdr.knowledge_base is loaded              │ PASS   │        226ns │
│ qualys_vmdr │ knowledge_base       │ asset     │ ingest_pipeline logs-qualys_vmdr.knowledge_base-0.1.0 is loaded       │ PASS   │        145ns │
╰─────────────┴──────────────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_vmdr - END   ---
Done
Run pipeline tests for the package
--- Test results for package: qualys_vmdr - START ---
╭─────────────┬──────────────────────┬───────────┬───────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM          │ TEST TYPE │ TEST NAME                     │ RESULT │ TIME ELAPSED │
├─────────────┼──────────────────────┼───────────┼───────────────────────────────┼────────┼──────────────┤
│ qualys_vmdr │ asset_host_detection │ pipeline  │ test-asset-host-detection.log │ PASS   │  21.872271ms │
│ qualys_vmdr │ knowledge_base       │ pipeline  │ test-knowledge-base.log       │ PASS   │   8.199736ms │
╰─────────────┴──────────────────────┴───────────┴───────────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_vmdr - END   ---
Done
Run static tests for the package
--- Test results for package: qualys_vmdr - START ---
╭─────────────┬──────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM          │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────────┼──────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ qualys_vmdr │ asset_host_detection │ static    │ Verify sample_event.json │ PASS   │  86.727011ms │
│ qualys_vmdr │ knowledge_base       │ static    │ Verify sample_event.json │ PASS   │  82.162652ms │
╰─────────────┴──────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_vmdr - END   ---
Done

@elasticmachine
Copy link

elasticmachine commented Jul 7, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-08-03T10:37:05.892+0000

  • Duration: 16 min 45 sec

Test stats 🧪

Test Results
Failed 0
Passed 14
Skipped 0
Total 14

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jul 7, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (2/2) 💚
Files 100.0% (2/2) 💚
Classes 100.0% (2/2) 💚
Methods 100.0% (26/26) 💚 75.0
Lines 95.9% (2105/2195) 👎 -4.1
Conditionals 100.0% (0/0) 💚

@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial review: CEL LGTM

Do we have a test where there is only a single element in the response list?

@piyush-elastic
Copy link
Contributor Author

Do we have a test where there is only a single element in the response list?

  • We have tested considering batch size = 1 on Kibana and it's working fine. But if we consider single element while running system test , it will not work because we are having map function.

@efd6
Copy link
Contributor

efd6 commented Jul 11, 2023

But if we consider single element while running system test , it will not work because we are having map function.

Why do you think this? It should be making an array of a single element and that can be mapped over. If this is not the case this is not safe.

@piyush-elastic piyush-elastic requested a review from efd6 July 13, 2023 12:10
packages/qualys_vmdr/_dev/deploy/docker/files/config.yml Outdated Show resolved Hide resolved
packages/qualys_vmdr/_dev/deploy/docker/files/config.yml Outdated Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do Qualys provide better docs?

"dataset": "qualys_vmdr.asset_host_detection",
"ingested": "2023-07-12T07:41:28Z",
"kind": "alert",
"original": "{\"DETECTION_LIST\":{\"DETECTION\":[{\"FIRST_FOUND_DATETIME\":\"2023-06-28T06:04:26Z\",\"IS_DISABLED\":\"0\",\"IS_IGNORED\":\"0\",\"LAST_FOUND_DATETIME\":\"2023-07-03T06:23:47Z\",\"LAST_PROCESSED_DATETIME\":\"2023-07-03T06:25:17Z\",\"LAST_TEST_DATETIME\":\"2023-07-03T06:23:47Z\",\"LAST_UPDATE_DATETIME\":\"2023-07-03T06:25:17Z\",\"QID\":\"91681\",\"RESULTS\":\"\",\"SEVERITY\":\"5\",\"SSL\":\"0\",\"STATUS\":\"Active\",\"TIMES_FOUND\":\"11\",\"TYPE\":\"Confirmed\"}]},\"DNS\":\"\",\"DNS_DATA\":{\"DOMAIN\":\"\",\"FQDN\":\"\",\"HOSTNAME\":\"\"},\"ID\":\"12048633\",\"IP\":\"10.50.2.111\",\"LAST_PC_SCANNED_DATE\":\"2023-06-28T09:58:12Z\",\"LAST_SCAN_DATETIME\":\"2023-07-03T06:25:17Z\",\"LAST_VM_SCANNED_DATE\":\"2023-07-03T06:23:47Z\",\"LAST_VM_SCANNED_DURATION\":\"1113\",\"NETBIOS\":\"\",\"OS\":\"\",\"TRACKING_METHOD\":\"IP\"}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design question. Do we want to retain the original XML here rather than the JSON? We can do this, though it obviously adds network cost and a small amount of complexity to the CEL code/ingest preamble.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do not have any processor for ingest pipeline which will decode the XML. So, we have directly ingested json format inside pipeline test. Since sample_event.json takes event.original directly from ingest pipeline, it sets it in the json format.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is where the additional complexity comes from. The body send to ingest would not just be the JSON, it would be {"json": <JSON message>, "xml": <original XML>}. Entirely doable, but I wondering if it is worth doing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it could be considered a bit different.

  1. If we want to have event.original, it needs to be the original body before any sort of decode.
  2. If that is not plausible, we should not have the option to store event.original, as it would go against what the field stands for (also for compliance reasons). In certain cases we have rather added support for this in the input itself (http_endpoint is one example).

If we feel that there is added complexity, we can go ahead and implement the initial version without event.original, while we discuss the best approach to add it in later.

Copy link
Contributor Author

@piyush-elastic piyush-elastic Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@P1llus, currently we do have event.orignal but we are considering JSON object (creates immediate after decoding actual XML). Is that okay? because if we remove event.original then preserve original event functionality will not work.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efd6 as Marius is on PTO, could you review Piyush's comment here? It's the last outstanding point, so if we're ok with this approach we can merge the PR now vs waiting for Marius to return.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marius' view was that we could add the event.original handling later. I'm OK with this too.

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still LGTM

@efd6 efd6 merged commit 6a26eec into elastic:main Aug 10, 2023
4 checks passed
@elasticmachine
Copy link

Package qualys_vmdr - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=qualys_vmdr

@jamiehynds
Copy link

@SpencerLN @clement-fouque Qualys integration now available if you'd like to test and provide any feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants