Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(tests): Remove unused tests #15429

Merged
merged 2 commits into from
Dec 2, 2022
Merged

chore(tests): Remove unused tests #15429

merged 2 commits into from
Dec 2, 2022

Conversation

bruceg
Copy link
Member

@bruceg bruceg commented Dec 1, 2022

tests/api.rs has been entirely commented out since 45da52c (chore: begin upgrade to tokio 1.x (#5872) April 2021), and tests/enterprise.rs will be moving out as well.

Closes #13046

@bruceg
chore(tests): Remove unused tests
d6d3887 
`tests/api.rs` has been entirely commented out since 45da52c (chore: begin
upgrade to tokio 1.x (#5872) April 2021), and `tests/enterprise.rs` will be
moving out as well.
@bruceg bruceg added the domain: tests Anything related to Vector's internal tests label Dec 1, 2022
@bruceg bruceg requested a review from a team December 1, 2022 23:47
@netlify
Copy link

netlify bot commented Dec 1, 2022 •
edited
Loading

✅ Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit 13fd458
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/638a0ba76d7e4b0009999cad

@netlify
Copy link

netlify bot commented Dec 1, 2022 •
edited
Loading

✅ Deploy Preview for vrl-playground canceled.

Name Link
🔨 Latest commit 13fd458
🔍 Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/638a0ba76cf30d00086a5ffb

jszwedko
jszwedko reviewed Dec 2, 2022
View reviewed changes
tests/enterprise.rs
@@ -1,128 +0,0 @@
#![cfg(feature = "enterprise-tests")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@001wwang just calling your attention to this in case you want to re-introduce them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the heads up. Yea, these might make their way into OPW (in a different form) depending on how reporting unfolds.

@github-actions
Copy link

github-actions bot commented Dec 2, 2022

Regression Test Results

Baseline: f19f3f1
Comparison: d6d3887
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Experiments where analysis encountered missing or malformed data:

otlp_grpc_to_blackhole

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
otlp_http_to_blackhole 20.36KiB/CPU-s 1.41 100.00% 1.41MiB/CPU-s 140.46KiB/CPU-s 1.81KiB/CPU-s 0.0 0.097042 1.43MiB/CPU-s 138.11KiB/CPU-s 1.78KiB/CPU-s 0.0 0.094094 False False
syslog_log2metric_humio_metrics 67.45KiB/CPU-s 1.11 100.00% 5.94MiB/CPU-s 160.0KiB/CPU-s 2.07KiB/CPU-s 0.0 0.026289 6.01MiB/CPU-s 162.76KiB/CPU-s 2.1KiB/CPU-s 0.0 0.026449 False False
http_text_to_http_json 95.87KiB/CPU-s 0.41 100.00% 22.77MiB/CPU-s 563.92KiB/CPU-s 7.28KiB/CPU-s 0.0 0.024189 22.86MiB/CPU-s 556.24KiB/CPU-s 7.19KiB/CPU-s 0.0 0.023762 False False
splunk_hec_route_s3 47.3KiB/CPU-s 0.39 100.00% 11.76MiB/CPU-s 532.22KiB/CPU-s 6.87KiB/CPU-s 0.0 0.044178 11.81MiB/CPU-s 525.75KiB/CPU-s 6.78KiB/CPU-s 0.0 0.043471 False False
syslog_humio_logs 36.95KiB/CPU-s 0.39 100.00% 9.19MiB/CPU-s 182.32KiB/CPU-s 2.35KiB/CPU-s 0.0 0.019374 9.23MiB/CPU-s 125.79KiB/CPU-s 1.63KiB/CPU-s 0.0 0.013314 False False
http_pipelines_blackhole 413.02B/CPU-s 0.05 76.54% 864.36KiB/CPU-s 10.67KiB/CPU-s 141.16B/CPU-s 0.0 0.012342 864.76KiB/CPU-s 24.02KiB/CPU-s 317.5B/CPU-s 0.0 0.027769 False False
socket_to_socket_blackhole 6.68KiB/CPU-s 0.05 88.58% 12.5MiB/CPU-s 203.3KiB/CPU-s 2.62KiB/CPU-s 0.0 0.015877 12.51MiB/CPU-s 256.97KiB/CPU-s 3.32KiB/CPU-s 0.0 0.020058 False False
http_to_http_noack 3.92KiB/CPU-s 0.03 59.46% 13.62MiB/CPU-s 284.09KiB/CPU-s 3.67KiB/CPU-s 0.0 0.020374 13.62MiB/CPU-s 228.7KiB/CPU-s 2.95KiB/CPU-s 0.0 0.016397 False False
splunk_hec_indexer_ack_blackhole 2.37KiB/CPU-s 0.02 39.21% 13.62MiB/CPU-s 258.58KiB/CPU-s 3.34KiB/CPU-s 0.0 0.018545 13.62MiB/CPU-s 248.2KiB/CPU-s 3.2KiB/CPU-s 0.0 0.017798 False False
enterprise_http_to_http 532.22B/CPU-s 0.00 15.19% 13.62MiB/CPU-s 148.81KiB/CPU-s 1.92KiB/CPU-s 0.0 0.010666 13.62MiB/CPU-s 148.19KiB/CPU-s 1.91KiB/CPU-s 0.0 0.010621 False False
splunk_hec_to_splunk_hec_logs_acks -1.25KiB/CPU-s -0.01 14.58% 13.61MiB/CPU-s 375.29KiB/CPU-s 4.84KiB/CPU-s 0.0 0.026917 13.61MiB/CPU-s 370.89KiB/CPU-s 4.78KiB/CPU-s 0.0 0.026604 False False
splunk_hec_to_splunk_hec_logs_noack -1.4KiB/CPU-s -0.01 25.07% 13.62MiB/CPU-s 235.99KiB/CPU-s 3.05KiB/CPU-s 0.0 0.016920 13.62MiB/CPU-s 244.74KiB/CPU-s 3.16KiB/CPU-s 0.0 0.017549 False False
file_to_blackhole -11.57KiB/CPU-s -0.02 42.50% 54.49MiB/CPU-s 1.1MiB/CPU-s 14.48KiB/CPU-s 0.0 0.020126 54.48MiB/CPU-s 1.11MiB/CPU-s 14.69KiB/CPU-s 0.0 0.020421 False False
fluent_elasticsearch -18.72KiB/CPU-s -0.04 96.75% 45.41MiB/CPU-s 30.13KiB/CPU-s 394.26B/CPU-s 0.0 0.000648 45.39MiB/CPU-s 684.86KiB/CPU-s 8.75KiB/CPU-s 0.0 0.014732 False False
http_to_http_json -12.17KiB/CPU-s -0.09 88.24% 13.5MiB/CPU-s 418.04KiB/CPU-s 5.4KiB/CPU-s 0.0 0.030231 13.49MiB/CPU-s 433.36KiB/CPU-s 5.6KiB/CPU-s 0.0 0.031366 False False
datadog_agent_remap_datadog_logs_acks -61.38KiB/CPU-s -0.18 99.98% 33.49MiB/CPU-s 1.07MiB/CPU-s 14.19KiB/CPU-s 0.0 0.032073 33.43MiB/CPU-s 617.7KiB/CPU-s 7.98KiB/CPU-s 0.0 0.018042 False False
http_to_http_acks -12.99KiB/CPU-s -0.24 19.50% 5.34MiB/CPU-s 2.83MiB/CPU-s 37.45KiB/CPU-s 0.0 0.530791 5.32MiB/CPU-s 2.79MiB/CPU-s 36.95KiB/CPU-s 0.0 0.524664 True False
syslog_splunk_hec_logs -30.16KiB/CPU-s -0.32 100.00% 9.11MiB/CPU-s 173.37KiB/CPU-s 2.24KiB/CPU-s 0.0 0.018576 9.08MiB/CPU-s 149.38KiB/CPU-s 1.93KiB/CPU-s 0.0 0.016057 False False
syslog_loki -46.59KiB/CPU-s -0.53 100.00% 8.56MiB/CPU-s 132.88KiB/CPU-s 1.72KiB/CPU-s 0.0 0.015161 8.51MiB/CPU-s 189.12KiB/CPU-s 2.44KiB/CPU-s 0.0 0.021693 False False
http_pipelines_no_grok_blackhole -39.27KiB/CPU-s -0.62 100.00% 6.15MiB/CPU-s 43.31KiB/CPU-s 572.56B/CPU-s 0.0 0.006873 6.12MiB/CPU-s 55.68KiB/CPU-s 736.09B/CPU-s 0.0 0.008891 False False
syslog_log2metric_splunk_hec_metrics -66.65KiB/CPU-s -0.68 100.00% 9.54MiB/CPU-s 180.44KiB/CPU-s 2.33KiB/CPU-s 0.0 0.018461 9.48MiB/CPU-s 311.83KiB/CPU-s 4.03KiB/CPU-s 0.0 0.032123 False False
syslog_regex_logs2metric_ddmetrics -29.87KiB/CPU-s -0.79 100.00% 3.68MiB/CPU-s 391.76KiB/CPU-s 5.06KiB/CPU-s 0.0 0.103887 3.65MiB/CPU-s 406.51KiB/CPU-s 5.25KiB/CPU-s 0.0 0.108659 True False
datadog_agent_remap_blackhole_acks -266.18KiB/CPU-s -0.85 100.00% 30.64MiB/CPU-s 919.05KiB/CPU-s 11.86KiB/CPU-s 0.0 0.029291 30.38MiB/CPU-s 1.16MiB/CPU-s 15.32KiB/CPU-s 0.0 0.038157 False False
http_pipelines_blackhole_acks -7.66KiB/CPU-s -1.21 100.00% 632.11KiB/CPU-s 12.69KiB/CPU-s 167.82B/CPU-s 0.0 0.020081 624.46KiB/CPU-s 29.04KiB/CPU-s 383.7B/CPU-s 0.0 0.046508 False False
datadog_agent_remap_blackhole -468.69KiB/CPU-s -1.46 100.00% 31.36MiB/CPU-s 668.25KiB/CPU-s 8.63KiB/CPU-s 0.0 0.020807 30.9MiB/CPU-s 1021.66KiB/CPU-s 13.18KiB/CPU-s 0.0 0.032282 False False
datadog_agent_remap_datadog_logs -606.81KiB/CPU-s -1.78 100.00% 33.23MiB/CPU-s 689.42KiB/CPU-s 8.91KiB/CPU-s 0.0 0.020260 32.64MiB/CPU-s 1.3MiB/CPU-s 17.19KiB/CPU-s 0.0 0.039864 False False

@github-actions
Copy link

github-actions bot commented Dec 2, 2022

Regression Test Results

Baseline: 95fabca
Comparison: 13fd458
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Experiments where analysis encountered missing or malformed data:

otlp_grpc_to_blackhole

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_splunk_hec_metrics 286.39KiB/CPU-s 3.03 100.00% 9.24MiB/CPU-s 388.16KiB/CPU-s 5.01KiB/CPU-s 0.0 0.041038 9.52MiB/CPU-s 249.67KiB/CPU-s 3.22KiB/CPU-s 0.0 0.025621 False False
socket_to_socket_blackhole 269.3KiB/CPU-s 2.15 100.00% 12.24MiB/CPU-s 327.32KiB/CPU-s 4.23KiB/CPU-s 0.0 0.026105 12.51MiB/CPU-s 291.98KiB/CPU-s 3.77KiB/CPU-s 0.0 0.022797 False False
datadog_agent_remap_blackhole 446.16KiB/CPU-s 1.43 100.00% 30.53MiB/CPU-s 1.34MiB/CPU-s 17.71KiB/CPU-s 0.0 0.043874 30.96MiB/CPU-s 877.64KiB/CPU-s 11.32KiB/CPU-s 0.0 0.027677 False False
datadog_agent_remap_blackhole_acks 394.27KiB/CPU-s 1.25 100.00% 30.75MiB/CPU-s 1.2MiB/CPU-s 15.87KiB/CPU-s 0.0 0.039022 31.14MiB/CPU-s 967.86KiB/CPU-s 12.5KiB/CPU-s 0.0 0.030353 False False
syslog_loki 103.17KiB/CPU-s 1.19 100.00% 8.44MiB/CPU-s 295.99KiB/CPU-s 3.82KiB/CPU-s 0.0 0.034264 8.54MiB/CPU-s 198.15KiB/CPU-s 2.56KiB/CPU-s 0.0 0.022667 False False
splunk_hec_route_s3 130.22KiB/CPU-s 1.08 100.00% 11.74MiB/CPU-s 538.28KiB/CPU-s 6.95KiB/CPU-s 0.0 0.044763 11.87MiB/CPU-s 504.08KiB/CPU-s 6.51KiB/CPU-s 0.0 0.041470 False False
syslog_humio_logs 99.05KiB/CPU-s 1.06 100.00% 9.14MiB/CPU-s 290.43KiB/CPU-s 3.75KiB/CPU-s 0.0 0.031029 9.24MiB/CPU-s 119.03KiB/CPU-s 1.54KiB/CPU-s 0.0 0.012584 False False
datadog_agent_remap_datadog_logs_acks 300.14KiB/CPU-s 0.89 100.00% 32.96MiB/CPU-s 1.22MiB/CPU-s 16.12KiB/CPU-s 0.0 0.037025 33.25MiB/CPU-s 958.34KiB/CPU-s 12.37KiB/CPU-s 0.0 0.028142 False False
syslog_log2metric_humio_metrics 50.96KiB/CPU-s 0.82 100.00% 6.07MiB/CPU-s 166.5KiB/CPU-s 2.15KiB/CPU-s 0.0 0.026786 6.12MiB/CPU-s 138.0KiB/CPU-s 1.78KiB/CPU-s 0.0 0.022020 False False
syslog_splunk_hec_logs 68.05KiB/CPU-s 0.74 100.00% 9.01MiB/CPU-s 216.77KiB/CPU-s 2.8KiB/CPU-s 0.0 0.023501 9.07MiB/CPU-s 166.76KiB/CPU-s 2.15KiB/CPU-s 0.0 0.017947 False False
otlp_http_to_blackhole 7.18KiB/CPU-s 0.49 99.56% 1.42MiB/CPU-s 138.35KiB/CPU-s 1.79KiB/CPU-s 0.0 0.094820 1.43MiB/CPU-s 138.21KiB/CPU-s 1.78KiB/CPU-s 0.0 0.094259 False False
http_text_to_http_json 75.67KiB/CPU-s 0.33 100.00% 22.71MiB/CPU-s 560.0KiB/CPU-s 7.23KiB/CPU-s 0.0 0.024076 22.79MiB/CPU-s 560.55KiB/CPU-s 7.24KiB/CPU-s 0.0 0.024022 False False
http_pipelines_no_grok_blackhole 15.01KiB/CPU-s 0.24 100.00% 6.08MiB/CPU-s 151.21KiB/CPU-s 1.95KiB/CPU-s 0.0 0.024304 6.09MiB/CPU-s 150.86KiB/CPU-s 1.95KiB/CPU-s 0.0 0.024189 False False
http_pipelines_blackhole_acks 920.55B/CPU-s 0.14 98.48% 627.75KiB/CPU-s 20.36KiB/CPU-s 269.11B/CPU-s 0.0 0.032434 628.65KiB/CPU-s 20.2KiB/CPU-s 266.92B/CPU-s 0.0 0.032123 False False
http_to_http_acks 3.94KiB/CPU-s 0.07 5.97% 5.28MiB/CPU-s 2.82MiB/CPU-s 37.28KiB/CPU-s 0.0 0.534593 5.28MiB/CPU-s 2.8MiB/CPU-s 37.04KiB/CPU-s 0.0 0.530547 True False
enterprise_http_to_http 260.95B/CPU-s 0.00 7.45% 13.62MiB/CPU-s 149.36KiB/CPU-s 1.93KiB/CPU-s 0.0 0.010705 13.62MiB/CPU-s 149.02KiB/CPU-s 1.93KiB/CPU-s 0.0 0.010681 False False
splunk_hec_to_splunk_hec_logs_noack 138.18B/CPU-s 0.00 2.36% 13.62MiB/CPU-s 250.66KiB/CPU-s 3.23KiB/CPU-s 0.0 0.017975 13.62MiB/CPU-s 249.37KiB/CPU-s 3.22KiB/CPU-s 0.0 0.017882 False False
splunk_hec_indexer_ack_blackhole -815.72B/CPU-s -0.01 13.40% 13.62MiB/CPU-s 256.84KiB/CPU-s 3.31KiB/CPU-s 0.0 0.018420 13.61MiB/CPU-s 260.51KiB/CPU-s 3.36KiB/CPU-s 0.0 0.018685 False False
splunk_hec_to_splunk_hec_logs_acks -756.72B/CPU-s -0.01 8.70% 13.62MiB/CPU-s 369.75KiB/CPU-s 4.77KiB/CPU-s 0.0 0.026517 13.62MiB/CPU-s 371.32KiB/CPU-s 4.79KiB/CPU-s 0.0 0.026631 False False
file_to_blackhole -12.57KiB/CPU-s -0.02 44.23% 54.5MiB/CPU-s 1.1MiB/CPU-s 14.58KiB/CPU-s 0.0 0.020268 54.49MiB/CPU-s 1.19MiB/CPU-s 15.73KiB/CPU-s 0.0 0.021866 False False
fluent_elasticsearch -9.16KiB/CPU-s -0.02 96.40% 45.41MiB/CPU-s 30.02KiB/CPU-s 392.63B/CPU-s 0.0 0.000646 45.4MiB/CPU-s 340.86KiB/CPU-s 4.35KiB/CPU-s 0.0 0.007331 False False
http_to_http_noack -8.58KiB/CPU-s -0.06 91.10% 13.62MiB/CPU-s 219.99KiB/CPU-s 2.84KiB/CPU-s 0.0 0.015772 13.61MiB/CPU-s 322.85KiB/CPU-s 4.17KiB/CPU-s 0.0 0.023160 False False
http_to_http_json -56.54KiB/CPU-s -0.41 100.00% 13.47MiB/CPU-s 407.24KiB/CPU-s 5.26KiB/CPU-s 0.0 0.029514 13.42MiB/CPU-s 479.55KiB/CPU-s 6.19KiB/CPU-s 0.0 0.034897 False False
datadog_agent_remap_datadog_logs -206.98KiB/CPU-s -0.61 100.00% 33.4MiB/CPU-s 648.54KiB/CPU-s 8.38KiB/CPU-s 0.0 0.018962 33.2MiB/CPU-s 1.2MiB/CPU-s 15.87KiB/CPU-s 0.0 0.036201 False False
http_pipelines_blackhole -13.61KiB/CPU-s -1.59 100.00% 854.28KiB/CPU-s 20.47KiB/CPU-s 270.82B/CPU-s 0.0 0.023961 840.67KiB/CPU-s 39.85KiB/CPU-s 526.76B/CPU-s 0.0 0.047399 False False
syslog_regex_logs2metric_ddmetrics -167.26KiB/CPU-s -4.38 100.00% 3.73MiB/CPU-s 434.69KiB/CPU-s 5.61KiB/CPU-s 0.0 0.113901 3.56MiB/CPU-s 465.67KiB/CPU-s 6.01KiB/CPU-s 0.0 0.127614 True False

@bruceg bruceg enabled auto-merge (squash) December 2, 2022 17:31
@jszwedko jszwedko merged commit 3d5b53d into master Dec 2, 2022
@jszwedko jszwedko deleted the bruceg/cleanup-tests branch December 2, 2022 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jszwedko jszwedko jszwedko approved these changes

@001wwang 001wwang 001wwang left review comments

Assignees
No one assigned
Labels
domain: tests Anything related to Vector's internal tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Audit and remove stand-alone tests
3 participants
@bruceg @jszwedko @001wwang