New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement partition-local stores #1720

Merged

lava merged 15 commits into master from story/ch25303/partition-local-store

Jun 30, 2021

Member

lava commented Jun 15, 2021 •

edited

Loading

Add new functions to erase data from segments
Add a method to create an empty chunk
Add a new 'plugins::find()' function
Add new plugin type 'store_plugin'
Add the ability to use partition-local stores

📔 Description

This adds the ability to use separate store backends for each partition in VAST.

A new "partition-local" backend is provided that just provides a simplified version of the current segment store which is restricted to a single segment.

📝 Checklist

All user-facing changes have changelog entries.
The changes are reflected on docs.tenzir.com/vast, if necessary.
The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

lava force-pushed the story/ch25303/partition-local-store branch from 6fe7aeb to c44846e Compare

June 15, 2021 12:06

lava commented

View reviewed changes

libvast/src/system/index.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/index.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/index.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/global_segment_store.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/global_segment_store.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/index.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/index.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/local_segment_store.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/local_segment_store.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/local_segment_store.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/local_segment_store.cpp Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/partition.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/local_segment_store.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/src/system/spawn_node.cpp Outdated Show resolved Hide resolved

lava commented

View reviewed changes

libvast/vast/system/partition.hpp Outdated Show resolved Hide resolved

dominiklohmann added blocked feature labels

lava force-pushed the story/ch25303/partition-local-store branch 4 times, most recently from f364407 to 7c92d71 Compare

June 21, 2021 10:20

lava added 6 commits

June 22, 2021 09:50


          Add new functions to erase data from segments

1c19df4


          Add a method to create an empty chunk

d190d5e


          Add a new 'plugins::find()' function

566a681

This allows to search a plugin by name and type.


          Add new plugin type 'store_plugin'

08e1b03


          Add the ability to use partition-local stores

18d66f7

This enables VAST to read and interpret partitions that
use custom store backends per partition.

In order to have VAST write partitions with local stores,
the new config option `vast.partition-local-stores` has
to be used.


          Add special cases for global archive

7ca9ae3

lava force-pushed the story/ch25303/partition-local-store branch from c3d02d3 to 3bec4f3 Compare

June 22, 2021 07:50

tobim requested changes

View reviewed changes

Member

tobim left a comment

A few minor suggestions regarding the parameters for the index.

libvast/src/system/spawn_index.cpp Outdated Show resolved Hide resolved

libvast/src/system/spawn_index.cpp Outdated Show resolved Hide resolved

libvast/vast/system/index.hpp Show resolved Hide resolved

lava added 6 commits

June 23, 2021 12:35


          Fix unit tests and integrate review feedback

8f5c28d


          Implement unit and integration test for partition-local stores; plus …

757a152

…some fixups


          Move local store unit tests

60dc50d


          Fix integration test log upload


          Dont drop queries while active store is persisting

0f3c1d9


          Allow use of user-provided store backends

aa475e1

lava force-pushed the story/ch25303/partition-local-store branch from 818bb0f to aa475e1 Compare

June 23, 2021 13:19


          Add CHANGELOG entry

3d61295

dominiklohmann reviewed

View reviewed changes

libvast/src/system/disk_monitor.cpp Show resolved Hide resolved

dominiklohmann reviewed

View reviewed changes

libvast/vast/plugin.hpp Show resolved Hide resolved

libvast/vast/plugin.hpp Outdated Show resolved Hide resolved

libvast/vast/plugin.hpp Show resolved Hide resolved

mavam changed the title ~~Implement partition-local stores for VAST~~ Implement partition-local stores


          Fix bug in response promise creation and expand comments

31547ae

lava force-pushed the story/ch25303/partition-local-store branch from c632193 to 31547ae Compare

June 23, 2021 16:30

dominiklohmann removed the blocked label

tobim requested changes

View reviewed changes

Member

tobim left a comment

This is still preliminary, I'm slowly going through the local store implementation.

libvast/src/system/spawn_index.cpp Outdated Show resolved Hide resolved

libvast/src/system/spawn_index.cpp Show resolved Hide resolved

tobim requested changes

View reviewed changes

Member

tobim left a comment

I ran the current version through the full integration test suite and noticed that some of the tests fail that I expected to pass.

For example "Server Zeek conn log" step 1 does not produce any output. I had a brief look at the node log and it seems the evaluator does not produce any ids. I would like to have everything but the disk monitor and vast get in a working state before this gets merged.

Another note: We should also have a chat about explore, specifically whether we'll need to change the type of the store_actor when we get rid of the global id space.

libvast/vast/system/local_segment_store.hpp Show resolved Hide resolved

libvast/vast/system/local_segment_store.hpp Show resolved Hide resolved

libvast/src/system/local_segment_store.cpp Show resolved Hide resolved


          Rename plugin name from 'local-segment-store' to 'segment-store'

d7ecf51

lava force-pushed the story/ch25303/partition-local-store branch from 14d1fc8 to d7ecf51 Compare

June 30, 2021 07:41

dominiklohmann reviewed

View reviewed changes

libvast/src/system/local_segment_store.cpp Show resolved Hide resolved

tobim approved these changes

View reviewed changes

Member

tobim left a comment

I tested this locally and will deploy it on the testbed soon.

lava merged commit a0fab02 into master

lava deleted the story/ch25303/partition-local-store branch

June 30, 2021 12:12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature