-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Filebeat] Add timezone config option to decode_cef and syslog input #27727
Merged
andrewkroh
merged 3 commits into
elastic:master
from
andrewkroh:feature/fb/decode-cef-timezone
Sep 7, 2021
Merged
[Filebeat] Add timezone config option to decode_cef and syslog input #27727
andrewkroh
merged 3 commits into
elastic:master
from
andrewkroh:feature/fb/decode-cef-timezone
Sep 7, 2021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewkroh
added
enhancement
Filebeat
Filebeat
Team:Security-External Integrations
labels
Sep 3, 2021
botelastic
bot
added
needs_team
Indicates that the issue/PR needs a Team:* label
and removed
needs_team
Indicates that the issue/PR needs a Team:* label
labels
Sep 3, 2021
This pull request doesn't have a |
Collaborator
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
andrewkroh
force-pushed
the
feature/fb/decode-cef-timezone
branch
3 times, most recently
from
September 3, 2021 14:00
7e55931
to
e62fc7c
Compare
CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the time zone to be specified when a timestamp does not contain an offset or zone. CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15 Note that the CEF module receives messages using the syslog input. The syslog input does not have a configurable time zone and always assumes timestamps without time zones are given in the machine's local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module. This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in `time/tzdata` package. The `timestamp` processor was updated. Fixes elastic#27232
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
andrewkroh
force-pushed
the
feature/fb/decode-cef-timezone
branch
from
September 3, 2021 14:23
e62fc7c
to
c47f3c9
Compare
run tests |
andrewkroh
changed the title
[Filebeat] Add timezone config option to decode_cef
[Filebeat] Add timezone config option to decode_cef and syslog input
Sep 3, 2021
kvch
reviewed
Sep 7, 2021
kvch
approved these changes
Sep 7, 2021
mergify bot
pushed a commit
that referenced
this pull request
Sep 7, 2021
…27727) CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the time zone to be specified when a timestamp does not contain an offset or zone. CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15 Note that the CEF module receives messages using the syslog input. The syslog input does not have a configurable time zone and always assumes timestamps without time zones are given in the machine's local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module. This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in `time/tzdata` package. The `timestamp` processor was updated. While I was adding the a timezone config type I made the syslog input's timezone configurable too. Fixes #27232 (cherry picked from commit b3497ca)
jarpy
pushed a commit
that referenced
this pull request
Sep 9, 2021
…27727) CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the time zone to be specified when a timestamp does not contain an offset or zone. CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15 Note that the CEF module receives messages using the syslog input. The syslog input does not have a configurable time zone and always assumes timestamps without time zones are given in the machine's local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module. This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in `time/tzdata` package. The `timestamp` processor was updated. While I was adding the a timezone config type I made the syslog input's timezone configurable too. Fixes #27232 (cherry picked from commit b3497ca)
mdelapenya
added a commit
to mdelapenya/beats
that referenced
this pull request
Sep 9, 2021
* master: (39 commits) [Heartbeat] Move JSON tests from python->go (elastic#27816) docs: simplify permissions for Dockerfile COPY (elastic#27754) Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789) [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774) warn log entry and no validation failure when both queue_url and buck… (elastic#27612) libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178) [Heartbeat] Add httpcommon options to ZipURL (elastic#27699) Add a header round tripper option to httpcommon (elastic#27509) [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779) Rename dashboards according to module.yml files for master (elastic#27749) Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726) Accept syslog dates with leading 0 (elastic#27775) [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727) [Filebeat] Threatintel compatibility updates (elastic#27323) Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707) [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491) Fix flakyness of TestFilestreamEmptyLine (elastic#27705) [Filebeat] kafka v2 using parsers (elastic#27335) Update Kafka version parsing / supported range (elastic#27720) Update Sarama to 1.29.1 (elastic#27717) ...
andrewkroh
added a commit
that referenced
this pull request
Sep 9, 2021
…27727) (#27780) CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the time zone to be specified when a timestamp does not contain an offset or zone. CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15 Note that the CEF module receives messages using the syslog input. The syslog input does not have a configurable time zone and always assumes timestamps without time zones are given in the machine's local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module. This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in `time/tzdata` package. The `timestamp` processor was updated. While I was adding the a timezone config type I made the syslog input's timezone configurable too. Fixes #27232 (cherry picked from commit b3497ca) Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
4 tasks
andrewkroh
added a commit
to andrewkroh/integrations
that referenced
this pull request
Sep 14, 2021
Expose the `timezone` config option for the `decode_cef` processor. It is an IANA time zone or time offset (e.g. `+0200`) to use when interpreting timestamps without a time zone in the CEF message. Relates: elastic/beats#27727
andrewkroh
added a commit
to elastic/integrations
that referenced
this pull request
Sep 27, 2021
Expose the `timezone` config option for the `decode_cef` processor. It is an IANA time zone or time offset (e.g. `+0200`) to use when interpreting timestamps without a time zone in the CEF message. Relates: elastic/beats#27727
Icedroid
pushed a commit
to Icedroid/beats
that referenced
this pull request
Nov 1, 2021
…lastic#27727) CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the time zone to be specified when a timestamp does not contain an offset or zone. CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15 Note that the CEF module receives messages using the syslog input. The syslog input does not have a configurable time zone and always assumes timestamps without time zones are given in the machine's local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module. This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in `time/tzdata` package. The `timestamp` processor was updated. While I was adding the a timezone config type I made the syslog input's timezone configurable too. Fixes elastic#27232
eyalkraft
pushed a commit
to build-security/integrations
that referenced
this pull request
Mar 30, 2022
Expose the `timezone` config option for the `decode_cef` processor. It is an IANA time zone or time offset (e.g. `+0200`) to use when interpreting timestamps without a time zone in the CEF message. Relates: elastic/beats#27727
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
CEF messages that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a
timezone
option to the decode_cef processor and cef module to allow thetime zone to be specified when a timestamp does not contain an offset or zone.
This also replaces the deprecated
import "4d63.com/tz"
with Go's relatively new built-intime/tzdata
package. Thetimestamp
processor was updated.While I was adding the a timezone config type I made the syslog input's timezone configurable too.
Fixes #27232
Why is it important?
Timestamps were being interpreted incorrectly.
Checklist
- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
Related issues
Logs
2021-09-03T10:08:32.659-0400 DEBUG [processors] processors/processor.go:120 Generated new processors: rename=[{From:message To:event.original}], decode_cef={"Field":"event.original","TargetField":"cef","IgnoreMissing":false,"IgnoreFailure":false,"ID":"","ECS":true,"Timezone":"America/New_York"}, community_id=[target=network.community_id, fields=[source_ip=source.ip, source_port=source.port, destination_ip=destination.ip, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0], add_fields={"ecs":{"version":"1.11.0"}}
Manual test