Implement a first basic pipeline string parser for the currently existing pipeline operators #2877

Dakostu · 2023-01-19T13:22:32Z

This PR adds the ability to parse & execute the currently implemented pipeline operators in the command line, via vast export <pipeline>.

Fixes tenzir/issues#6

TODO:

unit tests
integration tests
syntax cleanup; we can maybe "globalize" e.g. extractor rules
if slated for 3.0 release: changelog item saying this is experimental & one-liner in docu
~~- [ ] if slated for 3.1 release: changelog item & docu~~

libvast/builtins/operators/extend.cpp

libvast/builtins/operators/pseudonymize.cpp

libvast/builtins/operators/replace.cpp

libvast/builtins/operators/summarize.cpp

libvast/src/plugin.cpp

libvast/include/vast/concept/parseable/vast/option_set.hpp

patszt · 2023-01-25T11:35:58Z

Thanks for the good test coverage. I think the only code related issue is Dominik's suggestion about the return value from the option_set parse method.

dominiklohmann

Mostly minor things for the docs and some inline notes in the code. I didn't review the parsers in-depth. This is a great piece of work, and I'm really looking forward to use this every day.

changelog/unreleased/features/2877--pipeline-string-parsing.md

libvast/src/pipeline.cpp

libvast/src/system/spawn_exporter.cpp

libvast/include/vast/concept/parseable/vast/option_set.hpp

web/docs/understand/query-language/operators/replace.md

web/docs/understand/query-language/operators/select.md

web/docs/understand/query-language/operators/summarize.md

web/docs/understand/query-language/pipelines.md

patszt

Just a minor comment with regards to initialization. Great work!

libvast/include/vast/concept/parseable/vast/option_set.hpp

tobim · 2023-01-26T12:52:58Z

@Dakostu would you mind squashing your commits into a smaller number of logical units? The PR consists of 78 commits now and many of them are small fixups. I understand it is already approved, but doing such a final pass will make it easier to navigate history should it become necessary.

Implement string parsing for pseudonymize pipeline operator Implement string parsing for extend pipeline operator Implement string parsing for replace pipeline operator Implement string parsing for rename pipeline operator Allow aggregate pipelines from VAST nodes (quick hacky way to allow summarize operator) Implement parsing for summarize pipeline operator Implement aggregation function name assignment in summarize operator

Add pipeline parsing unit tests Add first batch of pipeline string parsing unit tests Add unit test 'pipeline string parsing - options - operator with wrong short form option' Implement unit test 'pipeline string parsing - options - operator with wrong long form option' Implement simple pipeline parsers in new file parseable/vast/pipeline.hpp Add unit test 'option set - missing option value'

Add integration test 'Pipeline operator parsing only summarize' Add 'identity' to pipeline integration test Add integration test 'Pipeline operator parsing after expression' Add integration test 'Pipeline operator summarize after expression' Add -b option to integration tests that need blocking import

…arsing validation

…tion set parsing unit tests

Remove unnecessary inclusion

Specify that pipelines can come after expressions in pipeline Add empty lines around example headline in docs Elaborate on pipeline operator syntax and examples in docs Add "for" to changelog entry Co-authored-by: Dominik Lohmann <mail@dominiklohmann.de> Specify that the syntax is subject to change in changelog entry Co-authored-by: Dominik Lohmann <mail@dominiklohmann.de>

…e_pipelines::yes

Dakostu · 2023-01-26T14:45:48Z

@Dakostu would you mind squashing your commits into a smaller number of logical units? The PR consists of 78 commits now and many of them are small fixups. I understand it is already approved, but doing such a final pass will make it easier to navigate history should it become necessary.

I squashed the number down to 38.

…nd plugins use it for parsing input strings

dominiklohmann

I only looked at the last commit fixing the handling of the query language plugin. All comments I've left are trivial to address, so feel free to just resolve them after.

libvast/include/vast/system/spawn_arguments.hpp

libvast/src/system/spawn_counter.cpp

libvast/src/system/spawn_pivoter.cpp

libvast/src/system/make_source.cpp

…ing for inappropriate components

dominiklohmann and others added 3 commits January 18, 2023 15:20

Expose pipelines in export command (WIP)

f805eaa

Debug

74ba54b

Implement drop and select

e206ed6

Dakostu added the feature New functionality label Jan 19, 2023

Dakostu requested a review from a team January 19, 2023 13:22

dominiklohmann requested review from patszt and removed request for a team January 23, 2023 08:33

Dakostu marked this pull request as ready for review January 24, 2023 09:57