enhancement(vrl): Implement a VM for VRL#9829
Merged
StephenWakely merged 99 commits intomasterfrom Feb 1, 2022
Merged
Conversation
|
✔️ Deploy Preview for vector-project ready! 🔨 Explore the source changes: 5c5d40c 🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/61f91a7510f66600082d6c60 😎 Browse the preview: https://deploy-preview-9829--vector-project.netlify.app |
Contributor
|
Dang, that's a pretty encouraging benchmark! One suggestion I would make: have this as an entirely separate crate, like |
|
|
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
github-actions
Bot
added
domain: transforms
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Contributor
Author
|
Here are some initial numbers: This PR: With dynamic bytecode: |
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Contributor
Author
|
A comparison between the Enum VM and the Bytecode VM: Baseline == enum Soak Test ResultsBaseline: 4886e6d What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.
|
| EXPERIMENT | VALUE_min | VALUE_p90 | VALUE_p99 | VALUE_max | VALUE_skewness | VALUE_kurtosis |
|---|---|---|---|---|---|---|
| baseline | 10.36Mi | 10.60Mi | 10.62Mi | 10.63Mi | -0.35 | -0.95 |
| comparison | 10.63Mi | 10.83Mi | 10.87Mi | 10.87Mi | 0.19 | -0.96 |
Bytecode does increase performance, but not by a massive amount.
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Contributor
Soak Test ResultsBaseline: 4a92c2b What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.
|
| EXPERIMENT | VALUE_min | VALUE_p90 | VALUE_p99 | VALUE_max | VALUE_skewness | VALUE_kurtosis |
|---|---|---|---|---|---|---|
| baseline | 9.84Mi | 9.87Mi | 9.88Mi | 9.88Mi | 0.07 | -0.25 |
| comparison | 19.03Mi | 19.06Mi | 19.06Mi | 19.07Mi | -0.37 | 0.11 |
Member
I notice this implementation has separate opcode/primitive in the instruction stream enum. Did you also test with the opcodes containing the value? ie enum Instruction {
…
Constant(usize),
…
} |
Contributor
Author
No, not yet. |
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Contributor
Soak Test ResultsBaseline: b8a5b7f What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.
|
| EXPERIMENT | VALUE_min | VALUE_p90 | VALUE_p99 | VALUE_max | VALUE_skewness | VALUE_kurtosis |
|---|---|---|---|---|---|---|
| baseline | 10.75Mi | 10.80Mi | 10.80Mi | 10.81Mi | -0.90 | -0.55 |
| comparison | 19.12Mi | 19.15Mi | 19.16Mi | 19.16Mi | 0.15 | -0.25 |
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
JeanMertz
reviewed
Jan 27, 2022
| [features] | ||
| # Default features for *-unknown-linux-gnu and *-apple-darwin | ||
| default = ["api", "api-client", "enrichment-tables", "rdkafka-plain", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vendor-all", "vrl-cli", "datadog-pipelines"] | ||
| default = ["api", "api-client", "enrichment-tables", "rdkafka-plain", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vendor-all", "vrl-cli", "datadog-pipelines", "vrl-vm"] |
Contributor
There was a problem hiding this comment.
I'm commenting here to make sure we don't forget this 😄
|
|
||
| pub struct Compiler<'a> { | ||
| fns: &'a [Box<dyn Function>], | ||
| fns: &'a [Box<dyn Function + Send + Sync>], |
Contributor
There was a problem hiding this comment.
Does it make sense to move Send + Sync to the actual Function trait (e.g. Function: Send + Sync) or else perhaps add a type alias SyncFunction or something?
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Contributor
Soak Test ResultsBaseline: d19f8bc ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. Changes in throughput with confidence ≥ 90.00%:
Fine details of change detection per experiment.
Fine details of each soak run.
|
Contributor
|
@StephenWakely do you have an idea where the degradation in the (unrelated, since they don't use VRL) soaks comes from? They seem to report pretty consistently. Is that an artifact of improvements that have landed on master but weren't merged into this branch yet? |
Collaborator
|
Yeah, I'm guessing this PR just needs |
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Contributor
Soak Test ResultsBaseline: d09e040 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. Changes in throughput with confidence ≥ 90.00%:
Fine details of change detection per experiment.
Fine details of each soak run.
|
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
JeanMertz
approved these changes
Jan 31, 2022
Contributor
JeanMertz
left a comment
JeanMertz
left a comment
There was a problem hiding this comment.
All of this looks great! There's more we can do here to refine the API/docs/tests, but nothing that should block this PR from being merged.
Contributor
Soak Test ResultsBaseline: d09e040 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. Changes in throughput with confidence ≥ 90.00%:
Fine details of change detection per experiment.
Fine details of each soak run.
|
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Contributor
Soak Test ResultsBaseline: d09e040 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. Changes in throughput with confidence ≥ 90.00%:
Fine details of change detection per experiment.
Fine details of each soak run.
|
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Contributor
Soak Test ResultsBaseline: 828b060 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes with confidence ≥ 90.00%: Fine details of change detection per experiment.
Fine details of each soak run.
|
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Contributor
Soak Test ResultsBaseline: be03df2 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes with confidence ≥ 90.00%: Fine details of change detection per experiment.
Fine details of each soak run.
|
This was referenced Mar 11, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first iteration on a Virtual Machine for running VRL code.
The VM is currently behind a feature flag
vrl-vmwhich is off by default, so running this branch will still run the old tree walking interpreter.Most of the VM code is found in https://github.com/vectordotdev/vector/tree/stephen/vrl-vm/lib/vrl/compiler/src/vm
Currently, a VRL program is parsed to an AST of nodes that implement
Expression. Running the program involves callingresolveon each node.To compile to bytecode, the
Expressiontrait has a new method calledcompile_to_vm- https://github.com/vectordotdev/vector/blob/stephen/vrl-vm/lib/vrl/compiler/src/expression.rs#L58-L60. Compiling is now a process of callingcompile_to_vmwhich generates the bytecode for the VM.No existing tree walking code has been changed, so it is possible to run both tree walking and VM side by side to ensure that no actual functionality is changed.
Not all functions in the stdlib have been converted to run with the VM. The following are supported:
array
contains
del
downcase
exists
flatten
integer
is_array
is_boolean
is_float
is_integer
is_null
is_object
is_regex
is_string
is_timestamp
match_datadog_query
merge
now
object
parse_groks
parse_json
parse_key_value
parse_regex
parse_regex_all
parse_url
push
slice
starts_with
to_string
to_timestamp
upcase
Both the VRL Cli and Tests projects work with the VM. A feature flag of
vrl-vmneeds to be set in order to use it.Old text:
This is a very rough proof of concept for a ByteCode Virtual Machine and Vrl -> Bytecode compiler.
It is capable of running the following Vrl (and not much else):
Initial and very rough benchmarks indicate that it was able to reduce the runtime for 1,000,000 events from 17 seconds to 10.
I do need to double check these figures as I find them suprisingly optimistic. But, it does suggest that this approach could be a very beneficial avenue to further pursue.