Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for user-defined operator aliases #3067

Merged
merged 7 commits into from Apr 12, 2023

Conversation

jachris
Copy link
Contributor

@jachris jachris commented Apr 12, 2023

This PR adds support for user-defined operator aliases that can be defined in the config. For example:

vast:
  operators:
    anonymize_urls: |
      replace net.url="xxx"
    aggregate_flows: |
       summarize 
         pkts_toserver=sum(flow.pkts_toserver),
         pkts_toclient=sum(flow.pkts_toclient),
         bytes_toserver=sum(flow.bytes_toserver),
         bytes_toclient=sum(flow.bytes_toclient),
         start=min(flow.start),
         end=max(flow.end)
       by
         timestamp,
         src_ip,
         dest_ip
       resolution
         10 mins 

This allows composing more complex pipelines, for instance from vast | anonymize_urls | aggregate_flows | write json to stdout. The PR only affects the new pipelines, which will eventually be used everywhere. The new pipeline parser also looks for definitions in vast.pipelines instead of vast.operators. However, this behavior is immediately considered deprecated and only available for compatibility.

@jachris jachris added the feature New functionality label Apr 12, 2023
@jachris jachris changed the title [WIP] Add support for user-defined operator aliases Add support for user-defined operator aliases Apr 12, 2023
@jachris jachris marked this pull request as ready for review April 12, 2023 14:29
@jachris jachris force-pushed the topic/user-defined-operators branch from b7ba09d to 29a5a41 Compare April 12, 2023 14:30
@jachris jachris force-pushed the topic/user-defined-operators branch from 29a5a41 to 9462121 Compare April 12, 2023 14:34
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a very small feature that is incredibly useful to have. We discussed in Slack that I'll write the documentation for this in an upcoming PR.

libvast/builtins/endpoints/query.cpp Outdated Show resolved Hide resolved
@jachris jachris force-pushed the topic/user-defined-operators branch from 77687be to 46e81d5 Compare April 12, 2023 14:55
@jachris jachris enabled auto-merge April 12, 2023 14:55
@jachris jachris merged commit d2b0abe into main Apr 12, 2023
38 of 39 checks passed
@jachris jachris deleted the topic/user-defined-operators branch April 12, 2023 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
2 participants