Implement transforms for VAST #1517

lava · 2021-04-01T19:59:21Z

📔 Description

This adds an MVP implementation of "transforms" to VAST.

📝 Checklist

All user-facing changes have changelog entries.
The changes are reflected on docs.tenzir.com/vast, if necessary.
The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

Look at the commit overview first: Roughly the first half consists of related bugfixes and refactorings I encountered while writing the main feature, the transforms implementation itself is split by category.

libvast/vast/system/spawn_transformer.hpp

libvast/src/system/spawn_transformer.cpp

libvast/src/transform.cpp

libvast/vast/system/actors.hpp

libvast/src/system/importer.cpp

lava · 2021-04-30T09:43:54Z

libvast/vast/transform_step.hpp

+//   [...]
+// };
+
+#if VAST_ENABLE_ARROW > 0


This can be rewritten as #if VAST_ENABLE_ARROW

When doing this please grep for VAST_ENABLE_ARROW > 0 generally. There are a few more places.

libvast/vast/system/source.hpp

lava · 2021-04-30T09:51:06Z

libvast/vast/system/source.hpp

+#include "vast/system/transformer.hpp"
+#include "vast/table_slice.hpp"
+#include "vast/table_slice_builder_factory.hpp"
+#include "vast/type_set.hpp"


Unnecessary includes

libvast/vast/system/importer.hpp

tobim

Fixing the Banners.

libvast/src/transform.cpp

libvast/src/transform_step.cpp

libvast/src/transform_steps/delete.cpp

libvast/src/transform_steps/replace.cpp

libvast/test/system/transformer.cpp

libvast/test/transform.cpp

libvast/vast/system/transformer.hpp

libvast/vast/transform.hpp

libvast/src/transform.cpp

libvast/src/system/spawn_importer.cpp

tobim

Accepting this now, but please add integration tests in a follow-up PR.

vast/vast.cpp

libvast/vast/system/spawn_importer.hpp

libvast/vast/system/source.hpp

libvast/vast/system/make_transforms.hpp

libvast/vast/system/make_source.hpp

libvast/vast/system/importer.hpp

libvast/vast/system/actors.hpp

libvast/vast/plugin.hpp

libvast/src/system/transformer.cpp

dominiklohmann added the feature New functionality label Apr 6, 2021

dominiklohmann mentioned this pull request Apr 9, 2021

Enable creation of table slices from record batches #1542

Closed

3 tasks

lava force-pushed the story/ch24236/transformer-actor branch 2 times, most recently from ad75a50 to 9cf7f9f Compare April 16, 2021 21:26

lava commented Apr 19, 2021

View reviewed changes

libvast/vast/system/spawn_transformer.hpp Outdated Show resolved Hide resolved

lava commented Apr 19, 2021

View reviewed changes

libvast/src/system/spawn_transformer.cpp Outdated Show resolved Hide resolved

lava commented Apr 19, 2021

View reviewed changes

libvast/src/transform.cpp Outdated Show resolved Hide resolved

lava commented Apr 19, 2021

View reviewed changes

libvast/vast/system/actors.hpp Outdated Show resolved Hide resolved

dominiklohmann mentioned this pull request Apr 21, 2021

Multiple node shutdown fixes #1563

Merged

2 tasks

dominiklohmann added the blocked Blocked by an (external) issue label Apr 23, 2021

lava force-pushed the story/ch24236/transformer-actor branch from 6849e13 to e2899fe Compare April 27, 2021 18:37

lava marked this pull request as ready for review April 27, 2021 18:37

lava changed the title ~~WIP: Implement transforms for VAST~~ Implement transforms for VAST Apr 27, 2021

lava force-pushed the story/ch24236/transformer-actor branch 2 times, most recently from a13b422 to 0402fbb Compare April 28, 2021 16:55

dominiklohmann removed the blocked Blocked by an (external) issue label Apr 29, 2021

lava force-pushed the story/ch24236/transformer-actor branch from 0402fbb to 2c3b673 Compare April 29, 2021 11:52

dominiklohmann mentioned this pull request Apr 29, 2021

Supported nested records in make_arrow_schema #1605

Closed

3 tasks

lava force-pushed the story/ch24236/transformer-actor branch 3 times, most recently from ccdaad3 to 2aa8fcf Compare April 29, 2021 16:15

lava commented Apr 30, 2021

View reviewed changes

libvast/src/system/importer.cpp Show resolved Hide resolved

lava commented Apr 30, 2021

View reviewed changes

libvast/vast/system/source.hpp Show resolved Hide resolved

lava commented Apr 30, 2021

View reviewed changes

libvast/vast/system/importer.hpp Show resolved Hide resolved

tobim requested changes Apr 30, 2021

View reviewed changes

lava commented Apr 30, 2021

View reviewed changes

libvast/src/transform.cpp Show resolved Hide resolved

lava commented Apr 30, 2021

View reviewed changes

libvast/src/transform.cpp Show resolved Hide resolved

lava commented Apr 30, 2021

View reviewed changes

libvast/src/system/spawn_importer.cpp Outdated Show resolved Hide resolved

lava added 12 commits May 7, 2021 16:21

Use correct actor type when spawning importer

2525ee6

Adjust assertions in for arrow table slices

d135ee5

Add a function to get the type from vast::data

0ac7a97

Add transforms to VAST

9f1ae24

Add built-in transform steps

970fef4

Add unit tests for transforms

c13dffd

Transform plugin example

15f13ca

Add CHANGELOG entry

78e39b5

Add new 'framed<t>' type for caf streaming

770373d

Remove overloaded stream handler

3969202

Use framed<> for source -> transformer communication

4f3edbb

Add workaround for lifetimes of temporary arrow table slices

5c28955

lava force-pushed the story/ch24236/transformer-actor branch from 77a039e to 5c28955 Compare May 7, 2021 14:22

lava added 3 commits May 7, 2021 16:48

Add checks to disable input transformations

5a2aa48

Address review comments

4563224

Add missing copyright banners

36089ab

lava force-pushed the story/ch24236/transformer-actor branch from 4e80dec to 36089ab Compare May 7, 2021 14:48

tobim approved these changes May 7, 2021

View reviewed changes

lava merged commit 3beb84b into master May 9, 2021

lava deleted the story/ch24236/transformer-actor branch May 9, 2021 23:49