fix(reduce transform): Fix flush not occuring when events arrive in high rate #16146

tomers · 2023-01-26T18:04:53Z

Fixes flush not occuring when events arrive in high frequency (in rate faster then expire_after between events).
Fixes #16145.

netlify · 2023-01-26T18:05:00Z

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	`b20a79c`
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63f11a47d464750008240540

netlify · 2023-01-26T18:05:02Z

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	`b20a79c`
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63f11a47ad17e500082c52d5

davidhuie-dd

I agree that we need a fix here, and I provided an alternative to explore. We'll also need some unit testing of this behavior since it's subtle.

src/transforms/reduce/mod.rs

github-actions · 2023-02-06T19:54:10Z

Regression Test Results

Run ID: f571c359-ad76-4bb2-afa7-c6a4fa309f9e
Baseline: 88b4c94
Comparison: b409ded
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment	Δ mean	Δ mean %	confidence
syslog_regex_logs2metric_ddmetrics	-202.4KiB/CPU-s	-5.24	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_humio_metrics	77.52KiB/CPU-s	1.28	100.00%	5.92MiB/CPU-s	322.74KiB/CPU-s	4.17KiB/CPU-s	0.053202	6.0MiB/CPU-s	364.45KiB/CPU-s	4.7KiB/CPU-s	0.059321	False	False
splunk_hec_route_s3	101.11KiB/CPU-s	0.85	100.00%	11.56MiB/CPU-s	633.3KiB/CPU-s	8.17KiB/CPU-s	0.053518	11.65MiB/CPU-s	581.29KiB/CPU-s	7.5KiB/CPU-s	0.048706	False	False
otlp_http_to_blackhole	8.78KiB/CPU-s	0.57	99.98%	1.5MiB/CPU-s	123.6KiB/CPU-s	1.6KiB/CPU-s	0.080721	1.5MiB/CPU-s	130.75KiB/CPU-s	1.69KiB/CPU-s	0.084907	False	False
datadog_agent_remap_datadog_logs_acks	136.41KiB/CPU-s	0.39	100.00%	33.97MiB/CPU-s	1.16MiB/CPU-s	15.32KiB/CPU-s	0.034144	34.1MiB/CPU-s	823.27KiB/CPU-s	10.63KiB/CPU-s	0.023574	False	False
http_text_to_http_json	88.18KiB/CPU-s	0.37	100.00%	23.18MiB/CPU-s	690.36KiB/CPU-s	8.91KiB/CPU-s	0.029083	23.27MiB/CPU-s	866.21KiB/CPU-s	11.18KiB/CPU-s	0.036356	False	False
datadog_agent_remap_blackhole	113.3KiB/CPU-s	0.36	100.00%	31.08MiB/CPU-s	902.08KiB/CPU-s	11.64KiB/CPU-s	0.028341	31.19MiB/CPU-s	963.15KiB/CPU-s	12.43KiB/CPU-s	0.030153	False	False
enterprise_http_to_http	9.94KiB/CPU-s	0.07	93.61%	13.61MiB/CPU-s	347.03KiB/CPU-s	4.48KiB/CPU-s	0.024898	13.62MiB/CPU-s	228.55KiB/CPU-s	2.95KiB/CPU-s	0.016386	False	False
splunk_hec_to_splunk_hec_logs_noack	2.37KiB/CPU-s	0.02	48.48%	13.62MiB/CPU-s	208.11KiB/CPU-s	2.69KiB/CPU-s	0.014917	13.63MiB/CPU-s	191.24KiB/CPU-s	2.47KiB/CPU-s	0.013705	False	False
splunk_hec_to_splunk_hec_logs_acks	-41.34B/CPU-s	-0.0	0.49%	13.62MiB/CPU-s	367.28KiB/CPU-s	4.74KiB/CPU-s	0.026339	13.62MiB/CPU-s	349.94KiB/CPU-s	4.51KiB/CPU-s	0.025096	False	False
fluent_elasticsearch	653.44B/CPU-s	0.0	76.09%	45.41MiB/CPU-s	30.05KiB/CPU-s	392.62B/CPU-s	0.000646	45.41MiB/CPU-s	30.01KiB/CPU-s	392.4B/CPU-s	0.000645	False	False
splunk_hec_indexer_ack_blackhole	-1.38KiB/CPU-s	-0.01	23.50%	13.62MiB/CPU-s	249.79KiB/CPU-s	3.22KiB/CPU-s	0.017912	13.62MiB/CPU-s	258.16KiB/CPU-s	3.33KiB/CPU-s	0.018514	False	False
file_to_blackhole	-5.42KiB/CPU-s	-0.01	20.42%	54.5MiB/CPU-s	1.1MiB/CPU-s	14.49KiB/CPU-s	0.02013	54.49MiB/CPU-s	1.15MiB/CPU-s	15.14KiB/CPU-s	0.021048	False	False
http_to_http_noack	-2.13KiB/CPU-s	-0.02	30.78%	13.62MiB/CPU-s	280.47KiB/CPU-s	3.62KiB/CPU-s	0.020114	13.61MiB/CPU-s	307.84KiB/CPU-s	3.97KiB/CPU-s	0.02208	False	False
datadog_agent_remap_blackhole_acks	-27.78KiB/CPU-s	-0.09	85.81%	30.81MiB/CPU-s	980.79KiB/CPU-s	12.67KiB/CPU-s	0.031084	30.78MiB/CPU-s	1.06MiB/CPU-s	14.05KiB/CPU-s	0.034524	False	False
syslog_humio_logs	-34.58KiB/CPU-s	-0.37	100.00%	9.16MiB/CPU-s	277.98KiB/CPU-s	3.59KiB/CPU-s	0.029631	9.13MiB/CPU-s	189.55KiB/CPU-s	2.45KiB/CPU-s	0.02028	False	False
otlp_grpc_to_blackhole	-4.74KiB/CPU-s	-0.45	100.00%	1.03MiB/CPU-s	42.94KiB/CPU-s	567.65B/CPU-s	0.040751	1.02MiB/CPU-s	54.25KiB/CPU-s	716.77B/CPU-s	0.051715	False	False
http_to_http_json	-68.82KiB/CPU-s	-0.49	100.00%	13.61MiB/CPU-s	244.8KiB/CPU-s	3.16KiB/CPU-s	0.017562	13.54MiB/CPU-s	309.75KiB/CPU-s	4.0KiB/CPU-s	0.022333	False	False
datadog_agent_remap_datadog_logs	-191.63KiB/CPU-s	-0.55	100.00%	34.16MiB/CPU-s	1.24MiB/CPU-s	16.4KiB/CPU-s	0.036333	33.98MiB/CPU-s	1.41MiB/CPU-s	18.67KiB/CPU-s	0.0416	False	False
syslog_splunk_hec_logs	-60.74KiB/CPU-s	-0.67	100.00%	8.92MiB/CPU-s	218.26KiB/CPU-s	2.82KiB/CPU-s	0.023904	8.86MiB/CPU-s	290.99KiB/CPU-s	3.76KiB/CPU-s	0.032083	False	False
syslog_log2metric_splunk_hec_metrics	-108.15KiB/CPU-s	-1.17	100.00%	9.03MiB/CPU-s	498.51KiB/CPU-s	6.44KiB/CPU-s	0.053906	8.92MiB/CPU-s	431.4KiB/CPU-s	5.57KiB/CPU-s	0.047201	False	False
socket_to_socket_blackhole	-156.53KiB/CPU-s	-1.2	100.00%	12.76MiB/CPU-s	493.18KiB/CPU-s	6.37KiB/CPU-s	0.037756	12.6MiB/CPU-s	612.48KiB/CPU-s	7.91KiB/CPU-s	0.047457	False	False
syslog_loki	-111.4KiB/CPU-s	-1.23	100.00%	8.81MiB/CPU-s	312.52KiB/CPU-s	4.03KiB/CPU-s	0.034639	8.7MiB/CPU-s	345.52KiB/CPU-s	4.46KiB/CPU-s	0.038776	False	False
http_to_http_acks	-99.75KiB/CPU-s	-1.81	94.83%	5.39MiB/CPU-s	2.78MiB/CPU-s	36.7KiB/CPU-s	0.515191	5.29MiB/CPU-s	2.71MiB/CPU-s	35.78KiB/CPU-s	0.511413	True	False
syslog_regex_logs2metric_ddmetrics	-202.4KiB/CPU-s	-5.24	100.00%	3.78MiB/CPU-s	487.88KiB/CPU-s	6.3KiB/CPU-s	0.12619	3.58MiB/CPU-s	454.62KiB/CPU-s	5.87KiB/CPU-s	0.124085	True	False

davidhuie-dd · 2023-02-07T01:15:46Z

@tomers looks like the test might've failed in other environments. maybe we need to play with the times?

Fixes flush not occuring when events arrive in high frequency (in rate faster then `expire_after` between events). Fixes vectordotdev#16145.

github-actions · 2023-02-14T23:49:29Z

Regression Detector Results

Run ID: 4d6ae017-e9c8-4b51-96fb-9f97f8c68742
Baseline: b6fdddc
Comparison: 4eb6ff4
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their ingress throughput performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in mean ingress throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in ingress throughput with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment	goal	Δ mean	Δ mean %	confidence
syslog_humio_logs	ingress throughput	-472.46KiB/CPU-s	-5.36	100.00%

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_splunk_hec_logs	ingress throughput	369.63KiB/CPU-s	4.28	100.00%	8.44MiB/CPU-s	273.6KiB/CPU-s	3.53KiB/CPU-s	0.031645	8.8MiB/CPU-s	223.26KiB/CPU-s	2.88KiB/CPU-s	0.024764	False	False
syslog_loki	ingress throughput	223.56KiB/CPU-s	2.64	100.00%	8.28MiB/CPU-s	275.84KiB/CPU-s	3.56KiB/CPU-s	0.032514	8.5MiB/CPU-s	268.87KiB/CPU-s	3.47KiB/CPU-s	0.030878	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	713.07KiB/CPU-s	2.16	100.00%	32.17MiB/CPU-s	1.37MiB/CPU-s	18.09KiB/CPU-s	0.042552	32.87MiB/CPU-s	1.2MiB/CPU-s	15.89KiB/CPU-s	0.036574	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	135.79KiB/CPU-s	1.49	100.00%	8.91MiB/CPU-s	359.98KiB/CPU-s	4.65KiB/CPU-s	0.039449	9.04MiB/CPU-s	257.57KiB/CPU-s	3.33KiB/CPU-s	0.027813	False	False
datadog_agent_remap_datadog_logs	ingress throughput	486.92KiB/CPU-s	1.42	100.00%	33.41MiB/CPU-s	1.13MiB/CPU-s	14.96KiB/CPU-s	0.033889	33.89MiB/CPU-s	1.16MiB/CPU-s	15.28KiB/CPU-s	0.034146	False	False
otlp_http_to_blackhole	ingress throughput	20.61KiB/CPU-s	1.33	100.00%	1.51MiB/CPU-s	120.74KiB/CPU-s	1.56KiB/CPU-s	0.078002	1.53MiB/CPU-s	117.32KiB/CPU-s	1.51KiB/CPU-s	0.074797	False	False
splunk_hec_route_s3	ingress throughput	131.27KiB/CPU-s	1.13	100.00%	11.37MiB/CPU-s	578.01KiB/CPU-s	7.46KiB/CPU-s	0.04962	11.5MiB/CPU-s	550.11KiB/CPU-s	7.1KiB/CPU-s	0.046699	False	False
datadog_agent_remap_blackhole	ingress throughput	319.45KiB/CPU-s	1.02	100.00%	30.5MiB/CPU-s	1.3MiB/CPU-s	17.13KiB/CPU-s	0.042476	30.81MiB/CPU-s	959.06KiB/CPU-s	12.38KiB/CPU-s	0.030397	False	False
http_to_http_noack	ingress throughput	21.33KiB/CPU-s	0.15	99.93%	13.61MiB/CPU-s	357.77KiB/CPU-s	4.62KiB/CPU-s	0.025669	13.63MiB/CPU-s	328.46KiB/CPU-s	4.24KiB/CPU-s	0.02353	False	False
enterprise_http_to_http	ingress throughput	8.02KiB/CPU-s	0.06	94.41%	13.62MiB/CPU-s	287.44KiB/CPU-s	3.71KiB/CPU-s	0.020614	13.62MiB/CPU-s	151.36KiB/CPU-s	1.95KiB/CPU-s	0.010849	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	835.26B/CPU-s	0.01	18.04%	13.62MiB/CPU-s	200.79KiB/CPU-s	2.59KiB/CPU-s	0.014391	13.63MiB/CPU-s	190.9KiB/CPU-s	2.46KiB/CPU-s	0.013681	False	False
fluent_elasticsearch	ingress throughput	36.2B/CPU-s	0.00	5.19%	45.41MiB/CPU-s	30.16KiB/CPU-s	394.09B/CPU-s	0.000648	45.41MiB/CPU-s	30.05KiB/CPU-s	392.72B/CPU-s	0.000646	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-650.31B/CPU-s	-0.00	10.96%	13.62MiB/CPU-s	253.91KiB/CPU-s	3.28KiB/CPU-s	0.018207	13.62MiB/CPU-s	251.2KiB/CPU-s	3.24KiB/CPU-s	0.018013	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-2.19KiB/CPU-s	-0.02	25.46%	13.62MiB/CPU-s	358.93KiB/CPU-s	4.63KiB/CPU-s	0.025739	13.62MiB/CPU-s	381.46KiB/CPU-s	4.92KiB/CPU-s	0.027358	False	False
file_to_blackhole	ingress throughput	-8.94KiB/CPU-s	-0.02	32.33%	54.5MiB/CPU-s	1.06MiB/CPU-s	14.05KiB/CPU-s	0.01951	54.49MiB/CPU-s	1.23MiB/CPU-s	16.2KiB/CPU-s	0.022531	False	False
http_to_http_json	ingress throughput	-4.59KiB/CPU-s	-0.03	68.96%	13.61MiB/CPU-s	251.54KiB/CPU-s	3.24KiB/CPU-s	0.018047	13.61MiB/CPU-s	244.1KiB/CPU-s	3.15KiB/CPU-s	0.017519	False	False
http_to_http_acks	ingress throughput	-19.45KiB/CPU-s	-0.36	29.33%	5.29MiB/CPU-s	2.77MiB/CPU-s	36.67KiB/CPU-s	0.52463	5.27MiB/CPU-s	2.76MiB/CPU-s	36.42KiB/CPU-s	0.523147	True	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	-40.77KiB/CPU-s	-1.14	100.00%	3.5MiB/CPU-s	357.65KiB/CPU-s	4.62KiB/CPU-s	0.099853	3.46MiB/CPU-s	322.56KiB/CPU-s	4.16KiB/CPU-s	0.091094	False	False
socket_to_socket_blackhole	ingress throughput	-159.84KiB/CPU-s	-1.19	100.00%	13.13MiB/CPU-s	248.65KiB/CPU-s	3.21KiB/CPU-s	0.018494	12.97MiB/CPU-s	495.63KiB/CPU-s	6.4KiB/CPU-s	0.037308	False	False
otlp_grpc_to_blackhole	ingress throughput	-14.66KiB/CPU-s	-1.38	100.00%	1.04MiB/CPU-s	40.12KiB/CPU-s	530.38B/CPU-s	0.03784	1.02MiB/CPU-s	52.5KiB/CPU-s	693.8B/CPU-s	0.050214	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	-445.37KiB/CPU-s	-1.42	100.00%	30.53MiB/CPU-s	1.18MiB/CPU-s	15.56KiB/CPU-s	0.038533	30.1MiB/CPU-s	1.59MiB/CPU-s	20.99KiB/CPU-s	0.052769	False	False
syslog_log2metric_humio_metrics	ingress throughput	-87.46KiB/CPU-s	-1.43	100.00%	5.99MiB/CPU-s	186.59KiB/CPU-s	2.41KiB/CPU-s	0.030423	5.9MiB/CPU-s	296.72KiB/CPU-s	3.83KiB/CPU-s	0.04908	False	False
http_text_to_http_json	ingress throughput	-980.91KiB/CPU-s	-3.90	100.00%	24.58MiB/CPU-s	691.4KiB/CPU-s	8.93KiB/CPU-s	0.027473	23.62MiB/CPU-s	690.91KiB/CPU-s	8.96KiB/CPU-s	0.028566	False	False
syslog_humio_logs	ingress throughput	-472.46KiB/CPU-s	-5.36	100.00%	8.61MiB/CPU-s	231.31KiB/CPU-s	2.99KiB/CPU-s	0.026242	8.15MiB/CPU-s	2.46MiB/CPU-s	32.49KiB/CPU-s	0.301693	True	False

github-actions · 2023-02-15T19:40:36Z

Regression Detector Results

Run ID: 83dae41e-320a-4eb0-8f29-170ac7c64ac9
Baseline: c4c03ca
Comparison: 6914d1f
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their ingress throughput performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in mean ingress throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in ingress throughput with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_splunk_hec_metrics	ingress throughput	302.52KiB/CPU-s	3.34	100.00%	8.85MiB/CPU-s	408.44KiB/CPU-s	5.27KiB/CPU-s	0.045057	9.15MiB/CPU-s	289.53KiB/CPU-s	3.74KiB/CPU-s	0.030908	False	False
splunk_hec_route_s3	ingress throughput	338.89KiB/CPU-s	2.96	100.00%	11.19MiB/CPU-s	688.81KiB/CPU-s	8.89KiB/CPU-s	0.060087	11.52MiB/CPU-s	671.43KiB/CPU-s	8.67KiB/CPU-s	0.056889	False	False
syslog_humio_logs	ingress throughput	208.74KiB/CPU-s	2.37	100.00%	8.6MiB/CPU-s	295.74KiB/CPU-s	3.82KiB/CPU-s	0.033589	8.8MiB/CPU-s	289.44KiB/CPU-s	3.74KiB/CPU-s	0.032111	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	668.16KiB/CPU-s	2.02	100.00%	32.32MiB/CPU-s	1.71MiB/CPU-s	22.58KiB/CPU-s	0.052886	32.97MiB/CPU-s	1.37MiB/CPU-s	18.13KiB/CPU-s	0.041601	False	False
syslog_loki	ingress throughput	167.76KiB/CPU-s	1.97	100.00%	8.34MiB/CPU-s	285.57KiB/CPU-s	3.68KiB/CPU-s	0.033447	8.5MiB/CPU-s	349.99KiB/CPU-s	4.52KiB/CPU-s	0.040202	False	False
datadog_agent_remap_datadog_logs	ingress throughput	565.4KiB/CPU-s	1.65	100.00%	33.42MiB/CPU-s	1.42MiB/CPU-s	18.72KiB/CPU-s	0.042404	33.98MiB/CPU-s	1.19MiB/CPU-s	15.72KiB/CPU-s	0.035031	False	False
http_to_http_acks	ingress throughput	17.26KiB/CPU-s	0.32	26.25%	5.27MiB/CPU-s	2.75MiB/CPU-s	36.36KiB/CPU-s	0.521701	5.29MiB/CPU-s	2.76MiB/CPU-s	36.47KiB/CPU-s	0.521528	True	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	8.3KiB/CPU-s	0.22	73.00%	3.72MiB/CPU-s	402.19KiB/CPU-s	5.19KiB/CPU-s	0.105711	3.72MiB/CPU-s	421.7KiB/CPU-s	5.44KiB/CPU-s	0.1106	True	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	9.23KiB/CPU-s	0.07	97.32%	13.62MiB/CPU-s	260.14KiB/CPU-s	3.36KiB/CPU-s	0.018656	13.63MiB/CPU-s	191.6KiB/CPU-s	2.47KiB/CPU-s	0.013731	False	False
enterprise_http_to_http	ingress throughput	5.95KiB/CPU-s	0.04	72.80%	13.61MiB/CPU-s	331.56KiB/CPU-s	4.28KiB/CPU-s	0.023785	13.62MiB/CPU-s	257.76KiB/CPU-s	3.33KiB/CPU-s	0.018483	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	3.89KiB/CPU-s	0.03	44.42%	13.61MiB/CPU-s	363.64KiB/CPU-s	4.69KiB/CPU-s	0.026083	13.62MiB/CPU-s	359.74KiB/CPU-s	4.64KiB/CPU-s	0.025795	False	False
http_to_http_noack	ingress throughput	234.75B/CPU-s	0.00	3.01%	13.61MiB/CPU-s	335.64KiB/CPU-s	4.33KiB/CPU-s	0.024078	13.61MiB/CPU-s	330.81KiB/CPU-s	4.27KiB/CPU-s	0.023732	False	False
fluent_elasticsearch	ingress throughput	-148.58B/CPU-s	-0.00	20.98%	45.41MiB/CPU-s	30.05KiB/CPU-s	392.67B/CPU-s	0.000646	45.41MiB/CPU-s	30.4KiB/CPU-s	397.08B/CPU-s	0.000654	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-773.7B/CPU-s	-0.01	13.13%	13.62MiB/CPU-s	248.77KiB/CPU-s	3.21KiB/CPU-s	0.017838	13.62MiB/CPU-s	252.33KiB/CPU-s	3.26KiB/CPU-s	0.018094	False	False
file_to_blackhole	ingress throughput	-23.28KiB/CPU-s	-0.04	65.95%	54.49MiB/CPU-s	1.12MiB/CPU-s	14.75KiB/CPU-s	0.020493	54.47MiB/CPU-s	1.48MiB/CPU-s	19.47KiB/CPU-s	0.027088	False	False
http_to_http_json	ingress throughput	-41.87KiB/CPU-s	-0.30	100.00%	13.57MiB/CPU-s	300.18KiB/CPU-s	3.87KiB/CPU-s	0.021595	13.53MiB/CPU-s	340.23KiB/CPU-s	4.39KiB/CPU-s	0.02455	False	False
datadog_agent_remap_blackhole	ingress throughput	-99.26KiB/CPU-s	-0.32	99.89%	30.04MiB/CPU-s	1.54MiB/CPU-s	20.31KiB/CPU-s	0.051144	29.95MiB/CPU-s	1.7MiB/CPU-s	22.48KiB/CPU-s	0.056795	False	False
syslog_log2metric_humio_metrics	ingress throughput	-56.67KiB/CPU-s	-0.93	100.00%	5.92MiB/CPU-s	239.74KiB/CPU-s	3.1KiB/CPU-s	0.039538	5.87MiB/CPU-s	174.48KiB/CPU-s	2.25KiB/CPU-s	0.029047	False	False
otlp_grpc_to_blackhole	ingress throughput	-12.99KiB/CPU-s	-1.23	100.00%	1.03MiB/CPU-s	40.13KiB/CPU-s	530.47B/CPU-s	0.037953	1.02MiB/CPU-s	55.0KiB/CPU-s	726.61B/CPU-s	0.052668	False	False
socket_to_socket_blackhole	ingress throughput	-264.53KiB/CPU-s	-1.94	100.00%	13.31MiB/CPU-s	319.37KiB/CPU-s	4.12KiB/CPU-s	0.023424	13.06MiB/CPU-s	391.06KiB/CPU-s	5.05KiB/CPU-s	0.029249	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	-770.36KiB/CPU-s	-2.43	100.00%	30.96MiB/CPU-s	1.52MiB/CPU-s	20.07KiB/CPU-s	0.049026	30.2MiB/CPU-s	1.94MiB/CPU-s	25.64KiB/CPU-s	0.06421	False	False
syslog_splunk_hec_logs	ingress throughput	-225.21KiB/CPU-s	-2.48	100.00%	8.85MiB/CPU-s	104.3KiB/CPU-s	1.35KiB/CPU-s	0.011506	8.63MiB/CPU-s	356.75KiB/CPU-s	4.6KiB/CPU-s	0.040359	False	False
otlp_http_to_blackhole	ingress throughput	-47.97KiB/CPU-s	-3.01	100.00%	1.56MiB/CPU-s	104.48KiB/CPU-s	1.35KiB/CPU-s	0.065475	1.51MiB/CPU-s	130.17KiB/CPU-s	1.68KiB/CPU-s	0.084105	False	False
http_text_to_http_json	ingress throughput	-1.0MiB/CPU-s	-4.19	100.00%	23.94MiB/CPU-s	825.49KiB/CPU-s	10.66KiB/CPU-s	0.033669	22.94MiB/CPU-s	741.36KiB/CPU-s	9.56KiB/CPU-s	0.031559	False	False

tomers · 2023-02-21T08:05:31Z

@davidhuie-dd can you please approve latest changes? thanks

github-actions · 2023-02-22T18:29:43Z

Regression Detector Results

Run ID: acfea22e-f473-4745-b214-26c1f31e856b
Baseline: 32c8452
Comparison: b20a79c
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment	goal	Δ mean	Δ mean %	confidence
file_to_blackhole	egress throughput	567.63KiB/CPU-s	9.28	99.91%
http_text_to_http_json	ingress throughput	-1.33MiB/CPU-s	-5.32	100.00%

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
file_to_blackhole	egress throughput	567.63KiB/CPU-s	9.28	99.91%	5.97MiB/CPU-s	4.37MiB/CPU-s	119.89KiB/CPU-s	0.731084	6.53MiB/CPU-s	3.99MiB/CPU-s	121.83KiB/CPU-s	0.532387	0.611661	True	False
syslog_loki	ingress throughput	229.93KiB/CPU-s	2.67	100.00%	8.41MiB/CPU-s	427.32KiB/CPU-s	5.26KiB/CPU-s	0.049623	8.63MiB/CPU-s	246.11KiB/CPU-s	3.03KiB/CPU-s	0.0	0.027837	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	608.54KiB/CPU-s	1.81	100.00%	32.83MiB/CPU-s	1.7MiB/CPU-s	21.46KiB/CPU-s	0.051871	33.43MiB/CPU-s	1.05MiB/CPU-s	13.24KiB/CPU-s	0.0	0.031429	False	False
datadog_agent_remap_blackhole	ingress throughput	334.81KiB/CPU-s	1.06	100.00%	30.89MiB/CPU-s	1.27MiB/CPU-s	15.99KiB/CPU-s	0.041091	31.21MiB/CPU-s	1.06MiB/CPU-s	13.32KiB/CPU-s	0.0	0.033878	False	False
datadog_agent_remap_datadog_logs	ingress throughput	317.69KiB/CPU-s	0.93	100.00%	33.31MiB/CPU-s	1.25MiB/CPU-s	15.78KiB/CPU-s	0.037602	33.62MiB/CPU-s	1.14MiB/CPU-s	14.41KiB/CPU-s	0.0	0.034027	False	False
syslog_splunk_hec_logs	ingress throughput	78.21KiB/CPU-s	0.88	100.00%	8.72MiB/CPU-s	315.94KiB/CPU-s	3.89KiB/CPU-s	0.035397	8.79MiB/CPU-s	332.74KiB/CPU-s	4.09KiB/CPU-s	0.0	0.036955	False	False
socket_to_socket_blackhole	ingress throughput	36.87KiB/CPU-s	0.27	100.00%	13.15MiB/CPU-s	522.87KiB/CPU-s	6.43KiB/CPU-s	0.038838	13.18MiB/CPU-s	314.76KiB/CPU-s	3.87KiB/CPU-s	0.0	0.023316	False	False
http_to_http_json	ingress throughput	7.27KiB/CPU-s	0.05	93.25%	13.61MiB/CPU-s	238.01KiB/CPU-s	2.93KiB/CPU-s	0.017073	13.62MiB/CPU-s	218.7KiB/CPU-s	2.69KiB/CPU-s	0.0	0.015679	False	False
enterprise_http_to_http	ingress throughput	3.82KiB/CPU-s	0.03	75.21%	13.62MiB/CPU-s	222.03KiB/CPU-s	2.73KiB/CPU-s	0.015918	13.62MiB/CPU-s	151.3KiB/CPU-s	1.86KiB/CPU-s	0.0	0.010844	False	False
http_to_http_noack	ingress throughput	3.46KiB/CPU-s	0.02	40.70%	13.61MiB/CPU-s	386.93KiB/CPU-s	4.76KiB/CPU-s	0.027769	13.61MiB/CPU-s	356.11KiB/CPU-s	4.38KiB/CPU-s	0.0	0.025551	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	96.11B/CPU-s	0.00	1.13%	13.61MiB/CPU-s	380.46KiB/CPU-s	4.68KiB/CPU-s	0.02729	13.61MiB/CPU-s	379.22KiB/CPU-s	4.66KiB/CPU-s	0.0	0.0272	False	False
fluent_elasticsearch	ingress throughput	277.75B/CPU-s	0.00	39.80%	45.41MiB/CPU-s	29.97KiB/CPU-s	373.56B/CPU-s	0.000644	45.41MiB/CPU-s	30.46KiB/CPU-s	379.58B/CPU-s	0.0	0.000655	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-1.04KiB/CPU-s	-0.01	17.42%	13.61MiB/CPU-s	267.71KiB/CPU-s	3.29KiB/CPU-s	0.019201	13.61MiB/CPU-s	273.18KiB/CPU-s	3.36KiB/CPU-s	0.0	0.019595	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-985.95B/CPU-s	-0.01	16.69%	13.62MiB/CPU-s	259.25KiB/CPU-s	3.19KiB/CPU-s	0.018593	13.61MiB/CPU-s	265.93KiB/CPU-s	3.27KiB/CPU-s	0.0	0.019074	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	-52.23KiB/CPU-s	-0.56	100.00%	9.1MiB/CPU-s	211.88KiB/CPU-s	2.61KiB/CPU-s	0.022747	9.04MiB/CPU-s	409.99KiB/CPU-s	5.04KiB/CPU-s	0.0	0.044263	False	False
otlp_grpc_to_blackhole	ingress throughput	-7.94KiB/CPU-s	-0.75	100.00%	1.04MiB/CPU-s	45.95KiB/CPU-s	579.04B/CPU-s	0.043344	1.03MiB/CPU-s	50.35KiB/CPU-s	634.34B/CPU-s	0.0	0.047849	False	False
otlp_http_to_blackhole	ingress throughput	-12.4KiB/CPU-s	-0.79	100.00%	1.53MiB/CPU-s	117.35KiB/CPU-s	1.44KiB/CPU-s	0.074861	1.52MiB/CPU-s	124.41KiB/CPU-s	1.53KiB/CPU-s	0.0	0.079994	False	False
splunk_hec_route_s3	ingress throughput	-134.8KiB/CPU-s	-1.16	100.00%	11.35MiB/CPU-s	624.35KiB/CPU-s	7.68KiB/CPU-s	0.053728	11.22MiB/CPU-s	738.51KiB/CPU-s	9.08KiB/CPU-s	0.0	0.064298	False	False
http_to_http_acks	ingress throughput	-68.78KiB/CPU-s	-1.28	84.86%	5.25MiB/CPU-s	2.68MiB/CPU-s	33.76KiB/CPU-s	0.510013	5.19MiB/CPU-s	2.7MiB/CPU-s	34.03KiB/CPU-s	0.0	0.520755	True	False
syslog_humio_logs	ingress throughput	-137.98KiB/CPU-s	-1.52	100.00%	8.87MiB/CPU-s	288.54KiB/CPU-s	3.55KiB/CPU-s	0.031761	8.74MiB/CPU-s	418.33KiB/CPU-s	5.15KiB/CPU-s	0.0	0.046758	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	-73.6KiB/CPU-s	-2.03	100.00%	3.55MiB/CPU-s	363.43KiB/CPU-s	4.47KiB/CPU-s	0.10004	3.48MiB/CPU-s	469.29KiB/CPU-s	5.78KiB/CPU-s	0.0	0.131849	True	False
syslog_log2metric_humio_metrics	ingress throughput	-168.16KiB/CPU-s	-2.77	100.00%	5.94MiB/CPU-s	239.27KiB/CPU-s	2.95KiB/CPU-s	0.039361	5.77MiB/CPU-s	313.81KiB/CPU-s	3.86KiB/CPU-s	0.0	0.053091	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	-1016.99KiB/CPU-s	-3.25	100.00%	30.53MiB/CPU-s	1.39MiB/CPU-s	17.46KiB/CPU-s	0.04538	29.54MiB/CPU-s	1.58MiB/CPU-s	19.87KiB/CPU-s	0.0	0.053352	False	False
http_text_to_http_json	ingress throughput	-1.33MiB/CPU-s	-5.32	100.00%	25.04MiB/CPU-s	979.92KiB/CPU-s	12.06KiB/CPU-s	0.038215	23.71MiB/CPU-s	640.52KiB/CPU-s	7.88KiB/CPU-s	0.0	0.026384	False	False

fuchsnj

This makes sense to me. Thanks!

…ive in high rate (#16146)" This reverts commit ab45939.

…metrics_ms` (#17084) Revert "fix(reduce transform): Fix flush not occuring when events arrive in high rate (#16146)" This reverts commit ab45939.

github-actions bot added the domain: transforms Anything related to Vector's transform components label Jan 26, 2023

tomers changed the title ~~transforms/reduce: fix flush not occusring~~ fix(reduce transform): Fix flush not occuring when events arrive in high rate Jan 26, 2023

tomers force-pushed the flush-when-no-start-or-end branch from 5d61c7e to 7089a4f Compare January 29, 2023 17:45

davidhuie-dd suggested changes Jan 30, 2023

View reviewed changes

src/transforms/reduce/mod.rs Outdated Show resolved Hide resolved

tomers force-pushed the flush-when-no-start-or-end branch from 7089a4f to ac438f3 Compare January 31, 2023 15:50

tomers mentioned this pull request Feb 1, 2023

feat(transforms): Introduce aggregate configuration #8749

Closed

davidhuie-dd reviewed Feb 1, 2023

View reviewed changes

src/transforms/reduce/mod.rs Outdated Show resolved Hide resolved

tomers force-pushed the flush-when-no-start-or-end branch 2 times, most recently from 6a90502 to cd08bca Compare February 2, 2023 20:17

davidhuie-dd self-assigned this Feb 2, 2023

tomers force-pushed the flush-when-no-start-or-end branch from cd08bca to b409ded Compare February 4, 2023 06:10

davidhuie-dd approved these changes Feb 6, 2023

View reviewed changes

src/transforms/reduce/mod.rs Outdated Show resolved Hide resolved

transforms/reduce: fix flush not occurring

63e15a4

Fixes flush not occuring when events arrive in high frequency (in rate faster then `expire_after` between events). Fixes vectordotdev#16145.

tomers force-pushed the flush-when-no-start-or-end branch from b409ded to 4eb6ff4 Compare February 12, 2023 13:26

tomers requested a review from a team February 12, 2023 13:26

davidhuie-dd approved these changes Feb 15, 2023

View reviewed changes

tomers force-pushed the flush-when-no-start-or-end branch from 6914d1f to f5e04c6 Compare February 16, 2023 07:23

jszwedko assigned fuchsnj Feb 16, 2023

transforms/reduce: add unit-tests

b20a79c

tomers force-pushed the flush-when-no-start-or-end branch from f5e04c6 to b20a79c Compare February 18, 2023 18:34

fuchsnj approved these changes Feb 23, 2023

View reviewed changes

davidhuie-dd merged commit ab45939 into vectordotdev:master Feb 23, 2023

tomers deleted the flush-when-no-start-or-end branch February 23, 2023 21:06

fuchsnj mentioned this pull request Mar 6, 2023

Can i use reduce transform(with merge strategies as sum) to sum events at a periodic time of 15/20seconds. #16695

Open

fuchsnj mentioned this pull request Mar 15, 2023

[Dedupe transformation] Add an option to flush the cache every X seconds / milliseconds and to count deduped logs #16681

Closed

jszwedko added a commit that referenced this pull request Apr 7, 2023

Revert "fix(reduce transform): Fix flush not occuring when events arr…

1a8ff1a

…ive in high rate (#16146)" This reverts commit ab45939.

jszwedko mentioned this pull request Apr 7, 2023

fix(reduce transform): Revert flushing on interval change to expire_metrics_ms #17084

Merged

jszwedko mentioned this pull request May 7, 2024

enhancement(reduce transform): New setting for reduce transform: end_every_period_ms #20440

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(reduce transform): Fix flush not occuring when events arrive in high rate #16146

fix(reduce transform): Fix flush not occuring when events arrive in high rate #16146

tomers commented Jan 26, 2023

netlify bot commented Jan 26, 2023 •

edited

netlify bot commented Jan 26, 2023 •

edited

davidhuie-dd left a comment

github-actions bot commented Feb 6, 2023

davidhuie-dd commented Feb 7, 2023

github-actions bot commented Feb 14, 2023

github-actions bot commented Feb 15, 2023

tomers commented Feb 21, 2023

github-actions bot commented Feb 22, 2023

fuchsnj left a comment

fix(reduce transform): Fix flush not occuring when events arrive in high rate #16146

fix(reduce transform): Fix flush not occuring when events arrive in high rate #16146

Conversation

tomers commented Jan 26, 2023

netlify bot commented Jan 26, 2023 • edited

✅ Deploy Preview for vector-project canceled.

netlify bot commented Jan 26, 2023 • edited

✅ Deploy Preview for vrl-playground canceled.

davidhuie-dd left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 6, 2023

Regression Test Results

davidhuie-dd commented Feb 7, 2023

github-actions bot commented Feb 14, 2023

Regression Detector Results

github-actions bot commented Feb 15, 2023

Regression Detector Results

tomers commented Feb 21, 2023

github-actions bot commented Feb 22, 2023

Regression Detector Results

fuchsnj left a comment

Choose a reason for hiding this comment

netlify bot commented Jan 26, 2023 •

edited

netlify bot commented Jan 26, 2023 •

edited