Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Osquerybeat: Result values type translation #25012

Merged
merged 7 commits into from
Apr 14, 2021

Conversation

aleksmaus
Copy link
Member

What does this PR do?

Translates Osquery results values to appropriate type according to the column type information of the query.
Utilizes the GetQueryColumns osquery go client API, caches the types information per query in LRU cache.

Why is it important?

Primarily allows us to handle better the numeric values that were strings by default.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Can test with standalone Osquerybeat config, example:

osquerybeat:
  inputs:
    - type: osquery
      streams:
        - id: "E169F085-AC8B-48AF-9355-D2977030CE24"
          query: "select * from temperature_sensors"
          interval: 1m        

Or running with agent and fleet server.

Related issues

Related issues

Screenshots

osquery mapping:
Screen Shot 2021-04-10 at 4 27 22 PM

collected osquery data with types converted appropriately with osquerybeat
Screen Shot 2021-04-10 at 4 25 11 PM

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 10, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 10, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25012 updated

  • Start Time: 2021-04-14T12:34:56.937+0000

  • Duration: 65 min 18 sec

  • Commit: 0404152

Test stats 🧪

Test Results
Failed 0
Passed 46994
Skipped 5134
Total 52128

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 10

Expand to view the steps failures

auditbeat-packaging-arm-arm - mage package
  • Took 4 min 37 sec . View more details on here
  • Description: mage package
filebeat-packaging-arm-arm - mage package
  • Took 8 min 47 sec . View more details on here
  • Description: mage package
heartbeat-packaging-arm-arm - mage package
  • Took 7 min 16 sec . View more details on here
  • Description: mage package
journalbeat-packaging-arm-arm - mage package
  • Took 11 min 8 sec . View more details on here
  • Description: mage package
metricbeat-packaging-arm-arm - mage package
  • Took 12 min 13 sec . View more details on here
  • Description: mage package
packetbeat-packaging-arm-arm - mage package
  • Took 7 min 21 sec . View more details on here
  • Description: mage package
x-pack/elastic-agent-packaging-arm-arm - mage package
  • Took 22 min 35 sec . View more details on here
  • Description: mage package
x-pack/heartbeat-packaging-arm-arm - mage package
  • Took 8 min 13 sec . View more details on here
  • Description: mage package
x-pack/metricbeat-packaging-arm-arm - mage package
  • Took 6 min 26 sec . View more details on here
  • Description: mage package
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-14T13:39:37.183Z] 10.13s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_136_oracle
[2021-04-14T13:39:37.183Z] 9.98s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_153_gcp
[2021-04-14T13:39:37.183Z] 9.89s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_135_oracle
[2021-04-14T13:39:37.183Z] 9.88s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_164_cisco
[2021-04-14T13:39:37.183Z] 9.88s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_004_cyberark
[2021-04-14T13:39:37.183Z] ======================= 323 passed in 1608.10s (0:26:48) =======================
[2021-04-14T13:39:37.183Z] >> python test: Integration Testing Complete
[2021-04-14T13:39:40.961Z] Cleaning up /var/lib/jenkins/workspace/PR-25012-7-2561faab-a7f1-4089-8b25-a11689f26347
[2021-04-14T13:39:40.961Z] Client: Docker Engine - Community
[2021-04-14T13:39:40.961Z]  Version:           20.10.3
[2021-04-14T13:39:40.962Z]  API version:       1.41
[2021-04-14T13:39:40.962Z]  Go version:        go1.13.15
[2021-04-14T13:39:40.962Z]  Git commit:        48d30b5
[2021-04-14T13:39:40.962Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-14T13:39:40.962Z]  OS/Arch:           linux/amd64
[2021-04-14T13:39:40.962Z]  Context:           default
[2021-04-14T13:39:40.962Z]  Experimental:      true
[2021-04-14T13:39:40.962Z] 
[2021-04-14T13:39:40.962Z] Server: Docker Engine - Community
[2021-04-14T13:39:40.962Z]  Engine:
[2021-04-14T13:39:40.962Z]   Version:          20.10.3
[2021-04-14T13:39:40.962Z]   API version:      1.41 (minimum version 1.12)
[2021-04-14T13:39:40.962Z]   Go version:       go1.13.15
[2021-04-14T13:39:40.962Z]   Git commit:       46229ca
[2021-04-14T13:39:40.962Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-14T13:39:40.962Z]   OS/Arch:          linux/amd64
[2021-04-14T13:39:40.962Z]   Experimental:     false
[2021-04-14T13:39:40.962Z]  containerd:
[2021-04-14T13:39:40.962Z]   Version:          1.4.4
[2021-04-14T13:39:40.962Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-14T13:39:40.962Z]  runc:
[2021-04-14T13:39:40.962Z]   Version:          1.0.0-rc93
[2021-04-14T13:39:40.962Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-14T13:39:40.962Z]  docker-init:
[2021-04-14T13:39:40.962Z]   Version:          0.19.0
[2021-04-14T13:39:40.962Z]   GitCommit:        de40ad0
[2021-04-14T13:39:40.962Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-14T13:39:40.962Z] Unable to find image 'alpine:3.4' locally
[2021-04-14T13:39:41.918Z] 3.4: Pulling from library/alpine
[2021-04-14T13:39:41.918Z] c1e54eec4b57: Pulling fs layer
[2021-04-14T13:39:42.186Z] c1e54eec4b57: Download complete
[2021-04-14T13:39:42.186Z] c1e54eec4b57: Pull complete
[2021-04-14T13:39:42.455Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-14T13:39:42.455Z] Status: Downloaded newer image for alpine:3.4
[2021-04-14T13:39:44.420Z] Change permissions with write access of all files inside the specific folder
[2021-04-14T13:39:45.856Z] Running in /var/lib/jenkins/workspace/PR-25012-7-2561faab-a7f1-4089-8b25-a11689f26347/src/github.com/elastic/beats/build
[2021-04-14T13:39:46.161Z] + rm -rf ve
[2021-04-14T13:39:46.161Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-14T13:39:46.502Z] + python .ci/scripts/pre_archive_test.py
[2021-04-14T13:39:48.433Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-04-14T13:39:48.457Z] Running in /var/lib/jenkins/workspace/PR-25012-7-2561faab-a7f1-4089-8b25-a11689f26347/src/github.com/elastic/beats/build
[2021-04-14T13:39:48.475Z] Recording test results
[2021-04-14T13:39:51.870Z] [Checks API] No suitable checks publisher found.
[2021-04-14T13:39:52.246Z] + go clean -modcache
[2021-04-14T13:39:55.907Z] Cleaning up /var/lib/jenkins/workspace/PR-25012-7-2561faab-a7f1-4089-8b25-a11689f26347
[2021-04-14T13:39:55.907Z] Client: Docker Engine - Community
[2021-04-14T13:39:55.907Z]  Version:           20.10.3
[2021-04-14T13:39:55.907Z]  API version:       1.41
[2021-04-14T13:39:55.907Z]  Go version:        go1.13.15
[2021-04-14T13:39:55.907Z]  Git commit:        48d30b5
[2021-04-14T13:39:55.907Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-14T13:39:55.907Z]  OS/Arch:           linux/amd64
[2021-04-14T13:39:55.907Z]  Context:           default
[2021-04-14T13:39:55.907Z]  Experimental:      true
[2021-04-14T13:39:55.907Z] 
[2021-04-14T13:39:55.907Z] Server: Docker Engine - Community
[2021-04-14T13:39:55.907Z]  Engine:
[2021-04-14T13:39:55.907Z]   Version:          20.10.3
[2021-04-14T13:39:55.907Z]   API version:      1.41 (minimum version 1.12)
[2021-04-14T13:39:55.907Z]   Go version:       go1.13.15
[2021-04-14T13:39:55.907Z]   Git commit:       46229ca
[2021-04-14T13:39:55.907Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-14T13:39:55.907Z]   OS/Arch:          linux/amd64
[2021-04-14T13:39:55.907Z]   Experimental:     false
[2021-04-14T13:39:55.907Z]  containerd:
[2021-04-14T13:39:55.907Z]   Version:          1.4.4
[2021-04-14T13:39:55.907Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-14T13:39:55.907Z]  runc:
[2021-04-14T13:39:55.907Z]   Version:          1.0.0-rc93
[2021-04-14T13:39:55.908Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-14T13:39:55.908Z]  docker-init:
[2021-04-14T13:39:55.908Z]   Version:          0.19.0
[2021-04-14T13:39:55.908Z]   GitCommit:        de40ad0
[2021-04-14T13:39:55.908Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-14T13:40:02.548Z] Change permissions with write access of all files inside the specific folder
[2021-04-14T13:40:02.870Z] Running in /var/lib/jenkins/workspace/PR-25012-7-2561faab-a7f1-4089-8b25-a11689f26347
[2021-04-14T13:40:08.331Z] + gsutil --version
[2021-04-14T13:40:09.837Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-14T13:40:10.159Z] + gcloud auth activate-service-account --key-file ****
[2021-04-14T13:40:10.739Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-14T13:40:11.073Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LWJ1aWxkMDQwNDE1MjE2N2VkOWVkNjIyZWVjNzBlMWRhMjMzZDEwZWVhOTU4OQ gs://beats-ci-temp/ci/cache/
[2021-04-14T13:40:13.228Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-14T13:40:13.303Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25012/src/github.com/elastic/beats
[2021-04-14T13:40:13.772Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25012
[2021-04-14T13:40:13.826Z] [INFO] getVaultSecret: Getting secrets
[2021-04-14T13:40:13.937Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-14T13:40:14.920Z] + chmod 755 generate-build-data.sh
[2021-04-14T13:40:14.920Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25012/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25012/runs/7 FAILURE 3917710
[2021-04-14T13:40:15.171Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25012/runs/7/steps/?limit=10000 -o steps-info.json
[2021-04-14T13:40:25.048Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25012/runs/7/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 46994
Skipped 5134
Total 52128

@aleksmaus aleksmaus added the Team:Elastic-Agent Label for the Agent team label Apr 10, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 10, 2021
@mergify
Copy link
Contributor

mergify bot commented Apr 10, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b osquery/result_type_translation upstream/osquery/result_type_translation
git merge upstream/master
git push upstream osquery/result_type_translation

… the first time the osquery integration is activated
@aleksmaus
Copy link
Member Author

the integration and kibana are updated to handle keyword/long keyword/double multifields correctly

Screen Shot 2021-04-13 at 11 40 37 AM

Screen Shot 2021-04-13 at 11 40 25 AM

Screen Shot 2021-04-13 at 11 42 15 AM


if c.log == nil {
c.log = logp.NewLogger(logTag)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit; new code should not use NewLogger. The logger should be assumed to be a dependency that must be passed in.

@aleksmaus aleksmaus added the backport-v7.13.0 Automated backport with mergify label Apr 14, 2021
@urso urso merged commit bcf6c92 into elastic:master Apr 14, 2021
mergify bot pushed a commit that referenced this pull request Apr 14, 2021
Translates Osquery results values to appropriate type according to the column type information of the query.
Utilizes the GetQueryColumns osquery go client API, caches the types information per query in LRU cache.

(cherry picked from commit bcf6c92)

# Conflicts:
#	NOTICE.txt
#	go.mod
urso pushed a commit that referenced this pull request Apr 14, 2021
Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co>
v1v added a commit to v1v/beats that referenced this pull request Apr 15, 2021
* upstream/master:
  packer cache support for the 7.x and 7.latestMinor branches (elastic#25091)
  Remove EventFetcher and EventsFetcher interface (elastic#25093)
  Update go-structform to 0.0.8 (elastic#25051)
  Update copy_fields.asciidoc (elastic#25053)
  [elastic-agent] ensure container is backwards compatible (elastic#25092)
  Add --fleet-server-service-token. Rename --fleet-server to --fleet-server-es. (elastic#25083)
  Add cgroup.cpuacct percentages (elastic#25057)
  Add tests for truncated and symlinked files in filestream input (elastic#24425)
  Fix panic when Hearbeat monitor initialization fails twice (elastic#25073)
  [Filebeat][httpjson] Change append transform to initiate new fields as a slice (elastic#25074)
  Osquerybeat: Result values type translation (elastic#25012)
  Update Osquerybeat spec to get it downloading from the correct artifactory path (elastic#25076)
  Fix changelog (elastic#25079)
  Strip Azure EventHub connection string in debug logs (elastic#25066)
  Change googlecloud to gcp in field names (elastic#25038)
  Bump stack version to 7.12.0 for testing (elastic#24957)
  packer-cache: cache the existing docker images on ARM and some more (elastic#25068)
  Disable logstash TestFetch flaky test (elastic#25044)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants