Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for cloudfoundry logs collection using consumer v1 API #19125

Merged
merged 18 commits into from
Jun 16, 2020

Conversation

jsoriano
Copy link
Member

@jsoriano jsoriano commented Jun 11, 2020

What does this PR do?

Add support for collection of logs from cloud foundry using consumer v1 API. v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding version: v2 to the configuration, v1 is used by default as is in principle more reliable. version is ignored by now by Metricbeat, that continues using v2. Support for v1 in Metricbeat will be implemented in a future change.

This change adds some additional dependencies for the doppler consumer (noaa).

Why is it important?

We already support the v2 API, but we saw that this API is not reliable enough (see #18202).

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Review FIXMEs in code.
  • Test with PCF

How to test this PR locally

  • Run filebeat with the cloudfoundry input and check that it collects events. Try these combinations of the version option:
    • No version option (what would use v1)
    • version: v1.
    • version: v2.
    • For other values of version it should fail to start.

Related issues

@jsoriano jsoriano added enhancement in progress Pull request is currently in progress. [zube]: In Progress Team:Platforms Label for the Integrations - Platforms team labels Jun 11, 2020
@jsoriano jsoriano self-assigned this Jun 11, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 11, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 11, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-06-15T22:36:14.836+0000

  • Duration: 17 min 14 sec

Steps errors

Expand to view the steps failures

  • Name: Make check
    • Description: make check

    • Duration: 10 min 44 sec

    • Start Time: 2020-06-15T22:43:18.091+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-15T22:53:06.539Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-15T22:53:06.539Z] Stage "Kubernetes" skipped due to earlier failure(s)
[2020-06-15T22:53:06.629Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.630Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.631Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-15T22:53:06.632Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.633Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.634Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.635Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.636Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:06.637Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-15T22:53:07.288Z] Failed in branch Elastic Agent x-pack
[2020-06-15T22:53:07.288Z] Failed in branch Elastic Agent x-pack Windows
[2020-06-15T22:53:07.289Z] Failed in branch Elastic Agent Mac OS X
[2020-06-15T22:53:07.289Z] Failed in branch Filebeat oss
[2020-06-15T22:53:07.290Z] Failed in branch Filebeat x-pack
[2020-06-15T22:53:07.290Z] Failed in branch Filebeat Mac OS X
[2020-06-15T22:53:07.291Z] Failed in branch Filebeat x-pack Mac OS X
[2020-06-15T22:53:07.292Z] Failed in branch Filebeat Windows
[2020-06-15T22:53:07.292Z] Failed in branch Filebeat x-pack Windows
[2020-06-15T22:53:07.293Z] Failed in branch Auditbeat oss Linux
[2020-06-15T22:53:07.293Z] Failed in branch Auditbeat crosscompile
[2020-06-15T22:53:07.294Z] Failed in branch Auditbeat oss Mac OS X
[2020-06-15T22:53:07.295Z] Failed in branch Auditbeat oss Windows
[2020-06-15T22:53:07.295Z] Failed in branch Auditbeat x-pack
[2020-06-15T22:53:07.296Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-06-15T22:53:07.296Z] Failed in branch Auditbeat x-pack Windows
[2020-06-15T22:53:07.297Z] Failed in branch Libbeat x-pack
[2020-06-15T22:53:07.298Z] Failed in branch Metricbeat OSS Unit tests
[2020-06-15T22:53:07.298Z] Failed in branch Metricbeat OSS Integration tests
[2020-06-15T22:53:07.299Z] Failed in branch Metricbeat Python integration tests
[2020-06-15T22:53:07.300Z] Failed in branch Metricbeat crosscompile
[2020-06-15T22:53:07.301Z] Failed in branch Metricbeat Mac OS X
[2020-06-15T22:53:07.301Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-06-15T22:53:07.302Z] Failed in branch Metricbeat Windows
[2020-06-15T22:53:07.302Z] Failed in branch Metricbeat x-pack Windows
[2020-06-15T22:53:07.303Z] Failed in branch Winlogbeat Windows x-pack
[2020-06-15T22:53:07.304Z] Failed in branch Kubernetes
[2020-06-15T22:53:07.610Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.611Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.612Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-15T22:53:07.613Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.614Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.615Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-15T22:53:07.689Z] Failed in branch Packetbeat
[2020-06-15T22:53:07.690Z] Failed in branch dockerlogbeat
[2020-06-15T22:53:07.691Z] Failed in branch Journalbeat
[2020-06-15T22:53:07.917Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.919Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.920Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-15T22:53:07.921Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-15T22:53:07.971Z] Failed in branch Metricbeat x-pack
[2020-06-15T22:53:07.971Z] Failed in branch Winlogbeat
[2020-06-15T22:53:08.182Z] Failed in branch Heartbeat
[2020-06-15T22:53:08.183Z] Failed in branch Libbeat
[2020-06-15T22:53:08.184Z] Failed in branch Functionbeat
[2020-06-15T22:53:08.184Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-15T22:53:08.276Z] Failed in branch Generators
[2020-06-15T22:53:08.418Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19125/src/github.com/elastic/beats
[2020-06-15T22:53:08.731Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-15T22:53:08.742Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19125/src/github.com/elastic/beats/Lint
[2020-06-15T22:53:09.121Z] + cat
[2020-06-15T22:53:09.121Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-15T22:53:09.121Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-15T22:53:15.698Z] runbld>>> runbld started
[2020-06-15T22:53:15.698Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-15T22:53:17.088Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19125' in order of occurrence in the config (last value wins).
[2020-06-15T22:53:18.473Z] runbld>>> Debug logging enabled.
[2020-06-15T22:53:18.473Z] runbld>>> Storing result
[2020-06-15T22:53:18.473Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-15T22:53:18.473Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200615225318-BD8BEA11
[2020-06-15T22:53:18.473Z] runbld>>> Adding system facts.
[2020-06-15T22:53:19.412Z] runbld>>> Adding vcs info for the latest commit:  f00081f12a9e37dfd2d5cb989c20bec605ea7fa0
[2020-06-15T22:53:19.672Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-15T22:53:19.672Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-15T22:53:19.672Z] Processing JUnit reports with runbld...
[2020-06-15T22:53:19.672Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-15T22:53:19.931Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-15T22:53:19.931Z] runbld>>> DURATION: 10ms
[2020-06-15T22:53:19.931Z] runbld>>> STDOUT: 40 bytes
[2020-06-15T22:53:19.931Z] runbld>>> STDERR: 49 bytes
[2020-06-15T22:53:19.931Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-15T22:53:19.931Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19125/src/github.com/elastic/beats
[2020-06-15T22:53:21.310Z] runbld>>> Storing build metadata: 
[2020-06-15T22:53:21.310Z] runbld>>> Adding test report.
[2020-06-15T22:53:21.310Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19125/src/github.com/elastic/beats
[2020-06-15T22:53:22.248Z] runbld>>> Found 0 test output files
[2020-06-15T22:53:22.248Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-06-15T22:53:22.507Z] runbld>>> Storing result
[2020-06-15T22:53:22.767Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-15T22:53:22.767Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200615225318-BD8BEA11
[2020-06-15T22:53:22.767Z] runbld>>> Email notification disabled by environment variable.
[2020-06-15T22:53:22.767Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-15T22:53:28.389Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19125
[2020-06-15T22:53:28.493Z] [INFO] getVaultSecret: Getting secrets
[2020-06-15T22:53:28.564Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-15T22:53:29.425Z] + chmod 755 generate-build-data.sh
[2020-06-15T22:53:29.425Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19125/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19125/runs/9 FAILURE 1034289
[2020-06-15T22:53:29.425Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19125/runs/9/steps/?limit=10000 -o steps-info.json
[2020-06-15T22:53:29.676Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19125/runs/9/tests/?status=FAILED -o tests-errors.json

@jsoriano jsoriano added needs_backport PR is waiting to be backported to other branches. v7.9.0 labels Jun 11, 2020
@jsoriano jsoriano marked this pull request as ready for review June 12, 2020 14:49
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@jsoriano jsoriano added [zube]: In Review review test-plan Add this PR to be manual test plan and removed [zube]: In Progress in progress Pull request is currently in progress. labels Jun 12, 2020
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, like the ability to switch between v1 and v2.

@jsoriano jsoriano merged commit 3ebceec into elastic:master Jun 16, 2020
@jsoriano jsoriano deleted the cloudfoundry-v1 branch June 16, 2020 08:05
@jsoriano jsoriano removed the needs_backport PR is waiting to be backported to other branches. label Jun 16, 2020
jsoriano added a commit to jsoriano/beats that referenced this pull request Jun 16, 2020
…lastic#19125)

Add support for collection of logs from cloud foundry using consumer v1 API.
v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses
the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding `version: v2` to the
configuration, v1 is used by default as is in principle more reliable. `version`
is ignored by now by Metricbeat.

(cherry picked from commit 3ebceec)
jsoriano added a commit that referenced this pull request Jun 16, 2020
…19125) (#19212)

Add support for collection of logs from cloud foundry using consumer v1 API.
v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses
the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding `version: v2` to the
configuration, v1 is used by default as is in principle more reliable. `version`
is ignored by now by Metricbeat.

(cherry picked from commit 3ebceec)
@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Jul 14, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…lastic#19125)

Add support for collection of logs from cloud foundry using consumer v1 API.
v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses
the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding `version: v2` to the
configuration, v1 is used by default as is in principle more reliable. `version`
is ignored by now by Metricbeat.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement review Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filebeat : Cloud foundry input dropping logs
4 participants