Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new parser to filestream input: container #26115

Merged
merged 2 commits into from
Jun 9, 2021

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Jun 2, 2021

What does this PR do?

This PR adds support for a new parser named container. This is the reader that powers the container input behind the scenes. Now it is exposed as a parser.

Example configuration for reading container logs with the filesteam input:

type: filestream
paths:
  - /path/to/containers/*/*.log
parsers:
  - container: ~

Limitations

The PR does not provide feature parity with the container input because of the lack of support for separating the states of stdout and strerr streams. It is coming in a follow-up PR.

Why is it important?

It is a step toward supporting reading container logs from every input that supports parsers option.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 2, 2021
@kvch kvch added the Team:Elastic-Agent Label for the Agent team label Jun 2, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 2, 2021
@kvch kvch force-pushed the feature-filestream-add-docker-json branch from 5849da6 to 20fc060 Compare June 2, 2021 14:20
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 2, 2021

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: Pull request #26115 updated

  • Start Time: 2021-06-02T14:20:54.163+0000

  • Duration: 183 min 56 sec

  • Commit: 20fc060150e0a1b3e6c56eac124032afcf18003f

Test stats 🧪

Test Results
Failed 1
Passed 47381
Skipped 5253
Total 52635

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Extended / metricbeat-windows-10-windows-10 / test_processors – metricbeat.tests.system.test_lightmodules.Test
    Expand to view the error details

     beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds. 
    

    Expand to view the stacktrace

     self = <test_lightmodules.Test testMethod=test_processors>
    
        def test_processors(self):
            shutil.copytree(
                os.path.join(self.beat_path, "mb/testing/testdata/lightmodules"),
                os.path.join(self.working_dir, "module"),
            )
        
            with http_test_server() as server:
                self.render_config_template(modules=[{
                    "name": "test",
                    "metricsets": ["json"],
                    "namespace": "test",
                    # Hard-coding 'localhost' because hostname in server.server_name doesn't always work.
                    "hosts": [f"localhost:{server.server_port}"],
                }])
        
                proc = self.start_beat()
        
    >           self.wait_until(lambda: self.output_lines() > 0)
    
    tests\system\test_lightmodules.py:32: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <test_lightmodules.Test testMethod=test_processors>
    cond = <function Test.test_processors.<locals>.<lambda> at 0x000001FE01C54B80>
    max_timeout = 10, poll_interval = 0.1, name = 'cond'
    
        def wait_until(self, cond, max_timeout=10, poll_interval=0.1, name="cond"):
            """
            Waits until the cond function returns true,
            or until the max_timeout is reached. Calls the cond
            function every poll_interval seconds.
        
            If the max_timeout is reached before cond() returns
            true, an exception is raised.
            """
            start = datetime.now()
            while not cond():
                if datetime.now() - start > timedelta(seconds=max_timeout):
    >               raise TimeoutError("Timeout waiting for '{}' to be true. ".format(name) +
                                       "Waited {} seconds.".format(max_timeout))
    E               beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.
    
    ..\libbeat\tests\system\beat\beat.py:362: TimeoutError 
    

Steps errors 1

Expand to view the steps failures

Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

Log output

Expand to view the last 100 lines of log output

[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:14:07] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:14:17] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:14:25] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:14:35] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:14:45] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:14:57] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:15:07] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:15:17] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:15:25] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:15:35] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:15:47] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:15:55] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:16:05] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:16:15] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:16:25] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:16:37] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:16:47] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:16:55] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:17:07] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:17:15] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:17:27] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:17:35] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:17:45] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.140Z] 127.0.0.1 - - [02/Jun/2021 17:17:55] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:18:05] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:18:15] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:18:27] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:18:37] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:18:47] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:18:57] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:19:05] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:19:15] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:19:25] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:19:35] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:19:47] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:19:55] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:20:05] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:20:17] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:20:25] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:20:37] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:20:47] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:20:57] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:21:07] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:21:15] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:21:27] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:21:37] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:21:45] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:21:57] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:22:07] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:22:15] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:22:25] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:36.141Z] 127.0.0.1 - - [02/Jun/2021 17:22:35] "GET / HTTP/1.1" 200 -
[2021-06-02T17:22:49.139Z] Cancelling nested steps due to timeout
[2021-06-02T17:22:49.142Z] Sending interrupt signal to process
[2021-06-02T17:22:49.562Z] Sending interrupt signal to process
[2021-06-02T17:22:51.013Z] ^CTerminate batch job (Y/N)? 
[2021-06-02T17:22:51.018Z] ^Cscript returned exit code -1073741510
[2021-06-02T17:22:51.392Z] 
[2021-06-02T17:22:51.392Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-02T17:22:53.606Z] 
[2021-06-02T17:22:53.606Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-02T17:22:54.983Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-06-02T17:22:54.983Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-06-02T17:22:55.000Z] Running in C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats\build
[2021-06-02T17:22:55.019Z] Recording test results
[2021-06-02T17:22:57.938Z] [Checks API] No suitable checks publisher found.
[2021-06-02T17:22:58.256Z] 
[2021-06-02T17:22:58.256Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats\build>tar --version  1>NUL 
[2021-06-02T17:22:58.592Z] 
[2021-06-02T17:22:58.592Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats\build>tar --exclude=test-build-artifacts-metricbeat-windows-10-windows-10-tgz -czf test-build-artifacts-metricbeat-windows-10-windows-10-tgz . 
[2021-06-02T17:23:02.279Z] 
[2021-06-02T17:23:02.279Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats\build>gsutil --version  1>NUL 
[2021-06-02T17:23:14.503Z] Masking supported pattern matches of %FILE_CREDENTIAL%
[2021-06-02T17:23:15.077Z] 
[2021-06-02T17:23:15.077Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats\build>gcloud auth activate-service-account --key-file **** 
[2021-06-02T17:23:16.017Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-02T17:23:20.025Z] [INFO] system-tests='build\metricbeat\build\system-tests'. If no empty then let's create a tarball
[2021-06-02T17:23:20.390Z] 
[2021-06-02T17:23:20.390Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>tar --version  1>NUL 
[2021-06-02T17:23:20.724Z] 
[2021-06-02T17:23:20.724Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>tar --exclude=metricbeat--system-tests-windows-tgz -czf metricbeat--system-tests-windows-tgz build\metricbeat\build\system-tests 
[2021-06-02T17:23:22.994Z] 
[2021-06-02T17:23:22.994Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>gsutil --version  1>NUL 
[2021-06-02T17:23:24.925Z] Masking supported pattern matches of %FILE_CREDENTIAL%
[2021-06-02T17:23:25.257Z] 
[2021-06-02T17:23:25.257Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>gcloud auth activate-service-account --key-file **** 
[2021-06-02T17:23:26.190Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-02T17:23:29.200Z] 
[2021-06-02T17:23:29.200Z] C:\Users\jenkins\workspace\PR-26115-3-08a783d7-1655-422f-8224-a8c2eb8bba25\src\github.com\elastic\beats>go clean -modcache 
[2021-06-02T17:23:49.139Z] Body did not finish within grace period; terminating with extreme prejudice
[2021-06-02T17:23:49.300Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-02T17:23:49.362Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-02T17:23:49.470Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26115/src/github.com/elastic/beats
[2021-06-02T17:23:49.805Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26115
[2021-06-02T17:23:49.865Z] [INFO] getVaultSecret: Getting secrets
[2021-06-02T17:23:49.914Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-02T17:23:50.623Z] + chmod 755 generate-build-data.sh
[2021-06-02T17:23:50.623Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/3 ABORTED 10976200
[2021-06-02T17:23:50.623Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/3/steps/?limit=10000 -o steps-info.json
[2021-06-02T17:24:00.499Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/3/tests/?status=FAILED -o tests-errors.json

@kvch
Copy link
Contributor Author

kvch commented Jun 4, 2021

Failing tests are unrelated.

@mergify
Copy link
Contributor

mergify bot commented Jun 7, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-filestream-add-docker-json upstream/feature-filestream-add-docker-json
git merge upstream/master
git push upstream feature-filestream-add-docker-json

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@kvch kvch force-pushed the feature-filestream-add-docker-json branch from 20fc060 to 9777cf4 Compare June 9, 2021 09:11
@kvch kvch merged commit e2449af into elastic:master Jun 9, 2021
@kvch kvch added the backport-v7.14.0 Automated backport with mergify label Jun 9, 2021
mergify bot pushed a commit that referenced this pull request Jun 9, 2021
## What does this PR do?

This PR adds support for a new parser named `container`. This is the reader that powers the `container` input behind the scenes. Now it is exposed as a parser.

Example configuration for reading container logs with the `filesteam` input:

```yaml
type: filestream
paths:
  - /path/to/containers/*/*.log
parsers:
  - container: ~
```

### Limitations

The PR does not provide feature parity with the `container` input because of the lack of support for separating the states of stdout and strerr streams. It is coming in a follow-up PR.

## Why is it important?

It is a step toward supporting reading container logs from every input that supports `parsers` option.

(cherry picked from commit e2449af)
@elasticmachine
Copy link
Collaborator

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26115 updated

  • Start Time: 2021-06-09T09:12:21.511+0000

  • Duration: 125 min 10 sec

  • Commit: 9777cf4

Test stats 🧪

Test Results
Failed 0
Passed 47446
Skipped 5256
Total 52702

Trends 🧪

Image of Build Times

Image of Tests

Log output

Expand to view the last 100 lines of log output

[2021-06-09T11:16:13.327Z]  Go version:        go1.13.15
[2021-06-09T11:16:13.327Z]  Git commit:        48d30b5
[2021-06-09T11:16:13.327Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-06-09T11:16:13.327Z]  OS/Arch:           linux/amd64
[2021-06-09T11:16:13.328Z]  Context:           default
[2021-06-09T11:16:13.328Z]  Experimental:      true
[2021-06-09T11:16:13.328Z] 
[2021-06-09T11:16:13.328Z] Server: Docker Engine - Community
[2021-06-09T11:16:13.328Z]  Engine:
[2021-06-09T11:16:13.328Z]   Version:          20.10.3
[2021-06-09T11:16:13.328Z]   API version:      1.41 (minimum version 1.12)
[2021-06-09T11:16:13.328Z]   Go version:       go1.13.15
[2021-06-09T11:16:13.328Z]   Git commit:       46229ca
[2021-06-09T11:16:13.328Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-06-09T11:16:13.328Z]   OS/Arch:          linux/amd64
[2021-06-09T11:16:13.328Z]   Experimental:     false
[2021-06-09T11:16:13.328Z]  containerd:
[2021-06-09T11:16:13.328Z]   Version:          1.4.3
[2021-06-09T11:16:13.328Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-06-09T11:16:13.328Z]  runc:
[2021-06-09T11:16:13.328Z]   Version:          1.0.0-rc92
[2021-06-09T11:16:13.328Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-06-09T11:16:13.328Z]  docker-init:
[2021-06-09T11:16:13.328Z]   Version:          0.19.0
[2021-06-09T11:16:13.328Z]   GitCommit:        de40ad0
[2021-06-09T11:16:13.328Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-06-09T11:16:13.328Z] Unable to find image 'alpine:3.4' locally
[2021-06-09T11:16:14.266Z] 3.4: Pulling from library/alpine
[2021-06-09T11:16:14.266Z] c1e54eec4b57: Pulling fs layer
[2021-06-09T11:16:14.836Z] c1e54eec4b57: Verifying Checksum
[2021-06-09T11:16:15.096Z] c1e54eec4b57: Pull complete
[2021-06-09T11:16:15.096Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-06-09T11:16:15.096Z] Status: Downloaded newer image for alpine:3.4
[2021-06-09T11:16:17.017Z] Change permissions with write access of all files inside the specific folder
[2021-06-09T11:16:17.611Z] Running in /var/lib/jenkins/workspace/PR-26115-4-afc55b15-78a9-4e6e-be46-71c1136115f4/src/github.com/elastic/beats/build
[2021-06-09T11:16:17.907Z] + rm -rf ve
[2021-06-09T11:16:17.908Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-06-09T11:16:18.227Z] + python .ci/scripts/pre_archive_test.py
[2021-06-09T11:16:28.249Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2021-06-09T11:16:28.249Z] Copy ./x-pack/elastic-agent/build/golang-crossbuild into build/x-pack/elastic-agent/build/golang-crossbuild
[2021-06-09T11:16:28.249Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
[2021-06-09T11:16:28.249Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build
[2021-06-09T11:16:28.249Z] Copy ./x-pack/metricbeat/build into build/x-pack/metricbeat/build
[2021-06-09T11:16:28.249Z] Copy ./x-pack/metricbeat/build/golang-crossbuild into build/x-pack/metricbeat/build/golang-crossbuild
[2021-06-09T11:16:28.249Z] Copy ./x-pack/heartbeat/build into build/x-pack/heartbeat/build
[2021-06-09T11:16:28.249Z] Copy ./x-pack/heartbeat/build/golang-crossbuild into build/x-pack/heartbeat/build/golang-crossbuild
[2021-06-09T11:16:28.249Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-06-09T11:16:28.249Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-06-09T11:16:28.267Z] Running in /var/lib/jenkins/workspace/PR-26115-4-afc55b15-78a9-4e6e-be46-71c1136115f4/src/github.com/elastic/beats/build
[2021-06-09T11:16:28.284Z] Recording test results
[2021-06-09T11:16:31.002Z] None of the test reports contained any result
[2021-06-09T11:16:31.010Z] [Checks API] No suitable checks publisher found.
[2021-06-09T11:16:31.360Z] + go clean -modcache
[2021-06-09T11:16:34.208Z] Cleaning up /var/lib/jenkins/workspace/PR-26115-4-afc55b15-78a9-4e6e-be46-71c1136115f4
[2021-06-09T11:16:34.208Z] Client: Docker Engine - Community
[2021-06-09T11:16:34.208Z]  Version:           20.10.3
[2021-06-09T11:16:34.208Z]  API version:       1.41
[2021-06-09T11:16:34.208Z]  Go version:        go1.13.15
[2021-06-09T11:16:34.208Z]  Git commit:        48d30b5
[2021-06-09T11:16:34.208Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-06-09T11:16:34.208Z]  OS/Arch:           linux/amd64
[2021-06-09T11:16:34.208Z]  Context:           default
[2021-06-09T11:16:34.208Z]  Experimental:      true
[2021-06-09T11:16:34.208Z] 
[2021-06-09T11:16:34.208Z] Server: Docker Engine - Community
[2021-06-09T11:16:34.208Z]  Engine:
[2021-06-09T11:16:34.208Z]   Version:          20.10.3
[2021-06-09T11:16:34.208Z]   API version:      1.41 (minimum version 1.12)
[2021-06-09T11:16:34.208Z]   Go version:       go1.13.15
[2021-06-09T11:16:34.208Z]   Git commit:       46229ca
[2021-06-09T11:16:34.208Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-06-09T11:16:34.208Z]   OS/Arch:          linux/amd64
[2021-06-09T11:16:34.208Z]   Experimental:     false
[2021-06-09T11:16:34.208Z]  containerd:
[2021-06-09T11:16:34.208Z]   Version:          1.4.3
[2021-06-09T11:16:34.208Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-06-09T11:16:34.208Z]  runc:
[2021-06-09T11:16:34.208Z]   Version:          1.0.0-rc92
[2021-06-09T11:16:34.208Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-06-09T11:16:34.208Z]  docker-init:
[2021-06-09T11:16:34.208Z]   Version:          0.19.0
[2021-06-09T11:16:34.208Z]   GitCommit:        de40ad0
[2021-06-09T11:16:34.208Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-06-09T11:17:20.902Z] Change permissions with write access of all files inside the specific folder
[2021-06-09T11:17:20.931Z] Running in /var/lib/jenkins/workspace/PR-26115-4-afc55b15-78a9-4e6e-be46-71c1136115f4
[2021-06-09T11:17:25.163Z] + gsutil --version
[2021-06-09T11:17:26.575Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-09T11:17:26.887Z] + gcloud auth activate-service-account --key-file ****
[2021-06-09T11:17:27.453Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-09T11:17:27.764Z] + gsutil -m -q cp -a public-read eC1wYWNrL2VsYXN0aWMtYWdlbnQtcGFja2FnaW5nLWxpbnV4OTc3N2NmNGI3ZWM3YTczMDMwNmM2MzZlNDczZWM5MWY5ODkzMjI4NQ gs://beats-ci-temp/ci/cache/
[2021-06-09T11:17:29.264Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-09T11:17:29.336Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26115/src/github.com/elastic/beats
[2021-06-09T11:17:29.825Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26115
[2021-06-09T11:17:29.879Z] [INFO] getVaultSecret: Getting secrets
[2021-06-09T11:17:29.928Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-09T11:17:31.657Z] + chmod 755 generate-build-data.sh
[2021-06-09T11:17:31.657Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/4 FAILURE 7509881
[2021-06-09T11:17:31.657Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/4/steps/?limit=10000 -o steps-info.json
[2021-06-09T11:17:43.753Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/4/tests/?status=FAILED -o tests-errors.json
[2021-06-09T11:17:45.096Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26115/runs/4/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 47446
Skipped 5256
Total 52702

kvch added a commit to kvch/beats that referenced this pull request Jun 9, 2021
## What does this PR do?

This PR adds support for a new parser named `container`. This is the reader that powers the `container` input behind the scenes. Now it is exposed as a parser.

Example configuration for reading container logs with the `filesteam` input:

```yaml
type: filestream
paths:
  - /path/to/containers/*/*.log
parsers:
  - container: ~
```

### Limitations

The PR does not provide feature parity with the `container` input because of the lack of support for separating the states of stdout and strerr streams. It is coming in a follow-up PR.

## Why is it important?

It is a step toward supporting reading container logs from every input that supports `parsers` option.

(cherry picked from commit e2449af)
kvch added a commit that referenced this pull request Jun 10, 2021
## What does this PR do?

This PR adds support for a new parser named `container`. This is the reader that powers the `container` input behind the scenes. Now it is exposed as a parser.

Example configuration for reading container logs with the `filesteam` input:

```yaml
type: filestream
paths:
  - /path/to/containers/*/*.log
parsers:
  - container: ~
```

### Limitations

The PR does not provide feature parity with the `container` input because of the lack of support for separating the states of stdout and strerr streams. It is coming in a follow-up PR.

## Why is it important?

It is a step toward supporting reading container logs from every input that supports `parsers` option.

(cherry picked from commit e2449af)

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 21, 2021
* master: (26 commits)
  Report total and free CPU for vSphere virtual machines (elastic#26167)
  [filebeat] Add preserve_original_event option to o365audit input (elastic#26273)
  Change xml processor names in script processor to match convention (elastic#26263)
  [Oracle] Fixing default values for paths in config template (elastic#26276)
  Add more ECS fields to logs (elastic#25998)
  [Heartbeat] Fix broken invocation of synth package (elastic#26228)
  rename sqs file name (elastic#26227)
  Populate the agent action result if there is no matching action handlers (elastic#26152)
  Add ISO8601 as supported timestamp type (elastic#25564)
  Move Filebeat azure module to GA (elastic#26168)
  Filebeat azure module pipeline fixes and changes (elastic#26148)
  libbeat: monitor version (elastic#26214)
  Add new parser to filestream input: container (elastic#26115)
  [Metricbeat] Add state_statefulset replicas.ready (elastic#26088)
  Disable test processors system test for windows 10 (elastic#26216)
  Fix startup with failing configuration (elastic#26126)
  Remove 32 bits version of Elastic Agent. (elastic#25708)
  Chane fleetmode detection to ony use management.enabled (elastic#26180)
  Make `filestream` input GA (elastic#26127)
  libbeat/idxmgmt/ilm: fix alias creation (elastic#26146)
  ...
This was referenced Mar 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.14.0 Automated backport with mergify Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants