Enhance traefik integration to also handle JSON-formatted access logs #770
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ycombinator
changed the title
17 tasks
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪 |
ycombinator
force-pushed
the
migrate-traefik-format-json
branch
2 times, most recently
from
9d567da
to
4538ac7
Compare
|
@mtojek I could use a second pair of eyes on why the system tests are failing for this PR. The symptom is that no documents can be found in the data stream: I am able to reproduce this locally. While the system test is still running, I checked that the Elastic Agent container has the log files mounted as expected: I also checked the policy in the Fleet UI and it looks right too: I also checked that the ingest pipelines are loaded as expected: {
"logs-traefik.access-0.0.1" : { ... },
"logs-traefik.access-0.0.1-format-json" : { ... },
"logs-traefik.access-0.0.1-format-common" : { ... }
}Note that pipeline tests are passing. |
|
I looked into filebeat's logs and found this one (file: |
|
Thanks for looking into it, @mtojek, appreciate it! |
andrewkroh
reviewed
Mar 22, 2021
| - append: | ||
| field: related.ip | ||
| value: "{{source.ip}}" | ||
| if: "ctx?.source?.ip != null" |
There was a problem hiding this comment.
I recommend adding allow_duplicates: false to the append processors for "related" fields.
There was a problem hiding this comment.
Just curious, why not do this for all append processors, e.g. the ones for event.category or event.type?
There was a problem hiding this comment.
Yes, in most cases it probably makes sense to deduplicate the append.
| - append: | ||
| field: related.user | ||
| value: "{{user.name}}" | ||
| if: "ctx?.user?.name != null && ctx.user.name != '-'" | ||
| on_failure: |
There was a problem hiding this comment.
It appears we have all the fields populated to support the community_id processor. Can you add that in?
edit: We'd need to set a static network.transport: tcp to make it work. But I think this makes sense for an http proxy.
| field: destination.address | ||
| copy_from: destination.ip | ||
| if: "ctx?.destination?.ip != null" | ||
| - rename: |
There was a problem hiding this comment.
RequestAddr looks like it could be used to populate url.domain.
andrewkroh
approved these changes
Mar 23, 2021
ycombinator
force-pushed
the
migrate-traefik-format-json
branch
from
1e2f800
to
deb8179
Compare
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪 |
eyalkraft
pushed a commit
to build-security/integrations
that referenced
this pull request
Mar 30, 2022
…elastic#770) * Migrating traefik module * Formatting package files * Removing invalid path field * Adding categories * Formatting tweaks * Adding pipeline test files * Adding YAML header * Adding system tests * Renaming pipeline test case files * Fixing pipeline tests * Adding sample event for health data set * Adding system test for access data stream * Adding README * Starting to handle JSON formatted logs * Adding ARG to Dockerfile for log format * Adding sample JSON logs * Running elastic-package format * Removing host field from sample event * Fix docker compose file * Splitting in commonlog and json format pipelines * Making pipeline test pass * Updating README.md * Address TODOs in pipeline * Specify services in system test configs * Refactoring out common processors into common pipeline * Add @timestamp field * Adding service to health data stream system test * Adding CHANGELOG entries * Parsing out event.duration * Regenerating sample events * Updating README * Add allow_duplicates: false for related.* fields' append processors * Adding community_id processor * Populating url.domain * Set allow_duplicates: false for other append processors * Regenerating README
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR enhances the
traefikintegration to parse JSON-formatted access logs. Prior to this PR thetraefikintegration could only parse commonlog-formatted access logs.Checklist
changelog.ymlfile.Related issues