Stateless plugins #8112
Merged
Stateless plugins #8112
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
9 tasks
fuziontech
approved these changes
Jan 18, 2022
Comment on lines
+79
to
+80
| if validated_data["is_stateless"] and len(validated_data["config_schema"]) > 0: | ||
| raise ValidationError("Stateless plugins cannot have a config!") |
| } else { | ||
| pluginConfig.vm = new LazyPluginVM() | ||
| pluginVMLoadPromises.push(loadPlugin(server, pluginConfig)) | ||
|
|
||
| if (prevConfig) { | ||
| void teardownPlugins(server, prevConfig) | ||
| } | ||
|
|
||
| if (plugin?.is_stateless) { | ||
| statelessVms[plugin.id] = pluginConfig.vm |
18 tasks
This was referenced Mar 2, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This reasonably small PR should bring us massive performance improvements on Cloud.
It introduces stateless plugins, which are plugins for which we can reuse VMs across teams as they do not rely on team-specific context.
Currently, we spin up 3091 VMs for GeoIp on each plugin server thread. That number is about to drop to 1.
Following Marius' calculation here, a simple VM takes up around 200kb of memory, with more complicated plugins taking up more. Using that lower bound, we should save about 620mb (
3091 * 200 - 200) of memory per thread per plugin server. That gives us savings of at least 2.5gb of RAM per plugins task on ECS (each task runs with 4 CPUs, thus 4 threads).Not to mention that this should really speed up our time to ingestion on reboots.
The more difficult part of stateless plugins would be if we could enforce/infer stateless plugins automatically, but both me and @mariusandra thought that's far from important right now. As such, we're using a flag defined in the plugin itself to determine this.
This flag can indeed be dangerous since a stateless plugin has access to context from multiple teams. Thus on Cloud we need to really keep an eye on plugins that set this.
How did you test this code?
Added test, tried out manually.