-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LOG-3883: Implement pruning fields from log messages #2348
Conversation
@Clee2691: This pull request references LOG-3883 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.8.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
/hold |
/retest |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Clee2691, jcantrill The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/hold cancel |
# Prune keys not in `notIn` list | ||
new_object = {} | ||
for_each(notIn) -> |_index, pathSeg| { | ||
val = get(., pathSeg) ?? null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we use exist
function instead of get
https://vector.dev/docs/reference/vrl/functions/#exists ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am using get
here because I broke up the field path into an array of path segments. The get
function expects an array instead of a path. I could not use something like:
paths = [.log, .message]
and iterate through that because for_each
would only extract the values and not preserve the path for the exists
function.
/hold cancel |
/retest |
/lgtm |
@Clee2691: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Description
This PR adds the prune filter feature. This feature allows admins to prune fields from log records to reduce the overall size of a record.
This feature extends the filter API by adding the fields
prune
along within
andnotIn
.The fields,
in
andnotIn
, expect an array of dot-delimited path expressions which should contain alphanumeric and underscores. Anything outside of that will have to be quoted..kubernetes."foo-bar/baz"
.log_type
An example prune filter spec is as follows:
/cc @cahartma @vparfonov
/assign @jcantrill
Links