Skip to content

Commit

Permalink
Add documentation for import batching options
Browse files Browse the repository at this point in the history
  • Loading branch information
dominiklohmann committed Sep 15, 2020
1 parent 71d19fd commit 74d3c3e
Showing 1 changed file with 18 additions and 0 deletions.
18 changes: 18 additions & 0 deletions doc/cli/vast-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,24 @@ The above command signals the running node to ingest (i.e., to archive and index
for later export) all Suricata events from the Eve JSON file passed via standard
input.

The import command batches parsed events in table slices. To control the
batching, the following options are available:

- `import.batch-encoding`: Controls the encoding of table slices. Available
options are `msgpack` (row-based) and `arrow` (column-based).
- `import.batch-size`: Sets an upper bound for the size of every table slice.
A table slice is the unit that all components work on, causing this to be a
high impact tuning parameter. Decreasing the table slice size causes reduced
latency up to the point where messaging overhead becomes important, and
increasing it may improve overall performance. Note that this setting does not
mean that events are forwarded to the index and archive immediately upon
exceeding the given number of events, as the table slices themselves are
buffered again. Setting this option to 0 causes the table slice size to be
unbounded, leaving control of the batch size to other parameters.
- `import.batch-timeout`: Sets a timeout for forwarding buffered table slices to
the index and archive, and can cause the `import.batch-size` parameter to be
underrun.

An optional filter expression allows for importing the relevant subset of
information only. For example, a user might want to import Suricata Eve JSON,
but skip over all events of type `suricata.stats`.
Expand Down

0 comments on commit 74d3c3e

Please sign in to comment.