feat(sources): opentelemetry log #13320

caibirdme · 2022-06-24T12:31:49Z

Example:

sources:
  otel:
    type: opentelemetry
    address: 0.0.0.0:6788

the opentelemetry log will be converted into EventLog:

{
  // optional
  "attributes": {"k1": "v1"},
  "resources": {"k1": "v1"},
  "message": "xxx",
  "trace_id": "xxx", 
  "span_id": "xxx",
  "severity_number": 1,
  "severity_text": "xxx",
  "flags": 1,

  // required
  "timestamp": ts_nano,
  "observed_time_unix_nano": ts_nano, //source will set this as current ts if it's not present
  "dropped_attributes_count": 0

}

I've been testing this source in our production for a week, and it works well! Hope this feature can be merged and I can continue to work on opentelemetry-metrics

bits-bot · 2022-06-24T12:31:53Z

All committers have signed the CLA.

netlify · 2022-06-24T12:32:10Z

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	`b77dab6`
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62d19d5efbaf8b0008b9969f

caibirdme · 2022-06-24T12:33:18Z

@StephenWakely I've tidied up the code, and please CR again

prognant

Thanks a lot for pursuing the effort.
Two major things, the source is names opentelemetry which implies that this source would be extended to support otel metrics (and traces) at some point, so IMHO the source config struct name should not refer to logs (or metrics/traces).
Can you also bootstrap documentation for this new source (inside ./website/cue/reference/components/sources/opentelemetry.cue) ?

Can I close #13135 as this PR seems to have precedence now ?

Thanks !

lib/opentelemetry/src/convert.rs

src/sources/opentelemetry/log.rs

src/sources/opentelemetry/mod.rs

…opentelemetry-add-log

…nto opentelemetry-add-log

caibirdme · 2022-06-30T07:59:27Z

Hi @prognant , I've updated the code, please review again, thx

prognant

Thanks for the update. I'll do a thorough review asap. In the meantime could you please add few test, at least one test with logs flowing through the source ?

src/sources/opentelemetry/log.rs

…nto opentelemetry-add-log

caibirdme · 2022-07-02T15:35:01Z

I don't know why there're two CI failures. I passed the examination in my dev environment by make check-all

github-actions · 2022-07-14T16:47:53Z

Soak Test Results

Baseline: 9a9340a
Comparison: ac77281
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_pipelines_blackhole	29.81KiB	1.91	100.00%	1.52MiB	103.18KiB	2.11KiB	0.0661596	1.55MiB	143.79KiB	2.93KiB	0.0904687	False	False
syslog_log2metric_humio_metrics	152.37KiB	1.14	100.00%	13.08MiB	294.86KiB	6.02KiB	0.0220087	13.23MiB	478.88KiB	9.76KiB	0.0353426	False	False
http_pipelines_blackhole_acks	12.84KiB	1.13	100.00%	1.1MiB	118.18KiB	2.41KiB	0.104423	1.12MiB	93.24KiB	1.9KiB	0.0814583	False	False
syslog_regex_logs2metric_ddmetrics	105.56KiB	0.82	100.00%	12.64MiB	698.73KiB	14.23KiB	0.0539634	12.75MiB	644.89KiB	13.15KiB	0.0494023	False	False
http_text_to_http_json	228.08KiB	0.57	100.00%	39.02MiB	1.03MiB	21.48KiB	0.0263361	39.24MiB	839.2KiB	17.14KiB	0.0208819	False	False
datadog_agent_remap_blackhole	208.64KiB	0.32	93.07%	63.91MiB	4.56MiB	94.86KiB	0.0712882	64.12MiB	3.1MiB	64.74KiB	0.0483837	False	False
datadog_agent_remap_blackhole_acks	78.77KiB	0.12	50.76%	63.3MiB	4.46MiB	92.81KiB	0.0703864	63.37MiB	3.23MiB	67.44KiB	0.0509156	False	False
splunk_hec_to_splunk_hec_logs_noack	18.97KiB	0.08	87.93%	23.82MiB	496.3KiB	10.12KiB	0.0203439	23.84MiB	335.49KiB	6.85KiB	0.0137417	False	False
syslog_loki	7.72KiB	0.05	28.72%	14.68MiB	559.45KiB	11.45KiB	0.0371968	14.69MiB	865.02KiB	17.58KiB	0.0574845	False	False
splunk_hec_to_splunk_hec_logs_acks	-15.54KiB	-0.06	46.81%	23.75MiB	833.69KiB	16.96KiB	0.0342693	23.74MiB	894.17KiB	18.18KiB	0.0367788	False	False
splunk_hec_indexer_ack_blackhole	-20.84KiB	-0.09	57.84%	23.76MiB	847.77KiB	17.25KiB	0.0348355	23.74MiB	951.74KiB	19.35KiB	0.0391412	False	False
socket_to_socket_blackhole	-17.71KiB	-0.13	100.00%	13.64MiB	153.63KiB	3.14KiB	0.0109939	13.63MiB	130.67KiB	2.67KiB	0.00936302	False	False
datadog_agent_remap_datadog_logs_acks	-92.78KiB	-0.14	55.96%	63.11MiB	3.63MiB	75.74KiB	0.0574766	63.02MiB	4.49MiB	93.38KiB	0.071175	False	False
syslog_log2metric_splunk_hec_metrics	-28.86KiB	-0.15	86.68%	18.94MiB	682.82KiB	13.91KiB	0.0352015	18.91MiB	650.45KiB	13.26KiB	0.0335824	False	False
http_to_http_json	-37.27KiB	-0.15	99.60%	23.85MiB	348.34KiB	7.11KiB	0.0142622	23.81MiB	530.08KiB	10.82KiB	0.0217365	False	False
file_to_blackhole	-152.14KiB	-0.16	77.86%	95.34MiB	3.47MiB	71.83KiB	0.0363371	95.19MiB	4.89MiB	101.57KiB	0.0513544	False	False
splunk_hec_route_s3	-37.25KiB	-0.2	39.96%	18.3MiB	2.47MiB	51.43KiB	0.13501	18.26MiB	2.35MiB	49.09KiB	0.128518	False	False
fluent_elasticsearch	-188.57KiB	-0.23	100.00%	79.47MiB	53.8KiB	1.09KiB	0.000661015	79.29MiB	1.92MiB	39.41KiB	0.0241508	False	False
http_to_http_noack	-59.04KiB	-0.24	99.12%	23.83MiB	517.73KiB	10.59KiB	0.0212138	23.77MiB	976.69KiB	19.9KiB	0.0401166	False	False
datadog_agent_remap_datadog_logs	-257.1KiB	-0.4	99.21%	62.17MiB	1.53MiB	31.98KiB	0.0245568	61.92MiB	4.38MiB	91.26KiB	0.0707722	False	False
syslog_humio_logs	-117.37KiB	-0.65	100.00%	17.63MiB	144.54KiB	2.95KiB	0.00800614	17.51MiB	141.5KiB	2.9KiB	0.00788924	False	False
http_to_http_acks	-233.89KiB	-1.21	72.32%	18.93MiB	7.63MiB	159.43KiB	0.402744	18.7MiB	6.91MiB	144.29KiB	0.369482	True	True
syslog_splunk_hec_logs	-263.4KiB	-1.48	100.00%	17.37MiB	948.52KiB	19.32KiB	0.0533023	17.12MiB	956.46KiB	19.5KiB	0.0545565	False	False
http_pipelines_no_grok_blackhole	-306.63KiB	-2.75	100.00%	10.88MiB	551.77KiB	11.27KiB	0.0495047	10.58MiB	1.1MiB	22.91KiB	0.103932	False	False

…opentelemetry-add-log

caibirdme · 2022-07-15T05:00:34Z

Yep does look like the incremental compilation bug - I have a commit converting the nanos -> Value::Timestamp and am looking at updating the tests to properly use the named outputs.

Any code to learn from? I didn't find any exmaple to test named output...

prognant · 2022-07-15T08:45:49Z

Any code to learn from? I didn't find any exmaple to test named output...

You may find a test that asserts multiple outputs behaviour here

spencergilbert · 2022-07-15T13:44:05Z

I should have a PR with final updates and working tests in the next hour or so. 👍

caibirdme · 2022-07-15T14:18:37Z

I should have a PR with final updates and working tests in the next hour or so. 👍

I've commited a pr which convert the timestamp and have the unit test passed

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

prognant

Apart the one comment I left it looks good to me 👍

Cargo.toml

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

github-actions · 2022-07-15T15:49:09Z

Soak Test Results

Baseline: fd0b6c1
Comparison: e8cffd0
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_regex_logs2metric_ddmetrics	130.54KiB	1	100.00%	12.73MiB	705.52KiB	14.37KiB	0.0541264	12.85MiB	732.54KiB	14.92KiB	0.0556422	False	False
socket_to_socket_blackhole	116.9KiB	0.84	100.00%	13.63MiB	348.78KiB	7.12KiB	0.0249883	13.74MiB	173.71KiB	3.55KiB	0.0123417	False	False
syslog_loki	118.29KiB	0.79	100.00%	14.67MiB	574.87KiB	11.76KiB	0.0382704	14.78MiB	732.5KiB	14.89KiB	0.0483831	False	False
datadog_agent_remap_datadog_logs	191.24KiB	0.31	94.34%	60.89MiB	1.83MiB	38.43KiB	0.0301039	61.08MiB	4.45MiB	92.61KiB	0.0727815	False	False
syslog_humio_logs	49.88KiB	0.28	100.00%	17.67MiB	404.39KiB	8.26KiB	0.0223412	17.72MiB	234.46KiB	4.8KiB	0.0129177	False	False
splunk_hec_to_splunk_hec_logs_noack	11.01KiB	0.05	68.73%	23.83MiB	424.51KiB	8.67KiB	0.0173958	23.84MiB	323.77KiB	6.61KiB	0.0132617	False	False
splunk_hec_to_splunk_hec_logs_acks	5.96KiB	0.02	19.52%	23.75MiB	845.33KiB	17.19KiB	0.0347513	23.76MiB	831.81KiB	16.92KiB	0.0341872	False	False
splunk_hec_indexer_ack_blackhole	-504.26B	-0	1.45%	23.75MiB	932.9KiB	18.98KiB	0.0383585	23.75MiB	945.35KiB	19.23KiB	0.0388712	False	False
http_pipelines_blackhole_acks	-141.75B	-0.01	4.10%	1.18MiB	108.09KiB	2.2KiB	0.0895228	1.18MiB	76.11KiB	1.55KiB	0.063045	False	False
splunk_hec_route_s3	-5.29KiB	-0.03	6.29%	18.93MiB	2.32MiB	48.39KiB	0.122694	18.92MiB	2.22MiB	46.4KiB	0.117198	False	False
file_to_blackhole	-62.93KiB	-0.06	50.68%	95.36MiB	2.87MiB	59.59KiB	0.0301375	95.3MiB	3.36MiB	69.87KiB	0.0352443	False	False
datadog_agent_remap_datadog_logs_acks	-43.2KiB	-0.07	31.14%	59.75MiB	2.81MiB	58.75KiB	0.0469726	59.71MiB	4.34MiB	90.38KiB	0.0727006	False	False
http_to_http_json	-26.73KiB	-0.11	97.22%	23.84MiB	349.74KiB	7.14KiB	0.0143212	23.82MiB	480.78KiB	9.83KiB	0.0197085	False	False
syslog_splunk_hec_logs	-34.89KiB	-0.2	90.96%	17.44MiB	792.27KiB	16.12KiB	0.0443617	17.4MiB	628.62KiB	12.83KiB	0.0352673	False	False
fluent_elasticsearch	-160.19KiB	-0.2	100.00%	79.47MiB	52.71KiB	1.07KiB	0.000647535	79.32MiB	1.55MiB	32.04KiB	0.0195976	False	False
http_to_http_noack	-87.18KiB	-0.36	99.95%	23.83MiB	505.69KiB	10.33KiB	0.0207201	23.74MiB	1.1MiB	22.93KiB	0.0462992	False	False
http_to_http_acks	-70.97KiB	-0.38	23.75%	18.37MiB	7.72MiB	161.51KiB	0.420326	18.3MiB	8.17MiB	170.41KiB	0.446097	True	True
datadog_agent_remap_blackhole	-372.61KiB	-0.57	99.89%	63.67MiB	4.2MiB	87.55KiB	0.0659603	63.31MiB	3.53MiB	73.52KiB	0.0556866	False	False
syslog_log2metric_splunk_hec_metrics	-122.84KiB	-0.65	99.97%	18.38MiB	1.03MiB	21.42KiB	0.0558339	18.26MiB	1.24MiB	25.85KiB	0.0680254	False	False
datadog_agent_remap_blackhole_acks	-431.97KiB	-0.66	100.00%	63.49MiB	4.49MiB	93.5KiB	0.0707222	63.07MiB	2.1MiB	43.9KiB	0.0332219	False	False
syslog_log2metric_humio_metrics	-99.99KiB	-0.73	100.00%	13.33MiB	308.75KiB	6.3KiB	0.0226146	13.23MiB	587.89KiB	11.98KiB	0.0433781	False	False
http_pipelines_blackhole	-12.97KiB	-0.77	100.00%	1.64MiB	11.44KiB	239.39B	0.00679658	1.63MiB	119.52KiB	2.44KiB	0.0715682	False	False
http_text_to_http_json	-372.97KiB	-0.94	100.00%	38.61MiB	841.23KiB	17.17KiB	0.0212739	38.24MiB	827.51KiB	16.89KiB	0.0211264	False	False
http_pipelines_no_grok_blackhole	-165.1KiB	-1.42	100.00%	11.38MiB	87.54KiB	1.79KiB	0.00751238	11.22MiB	1.09MiB	22.64KiB	0.0969043	False	False

StephenWakely

Looking good!

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

github-actions · 2022-07-15T16:58:50Z

Soak Test Results

Baseline: e8f32c3
Comparison: 3aefaa2
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_humio_logs	510.84KiB	2.89	100.00%	17.26MiB	540.57KiB	11.04KiB	0.0305739	17.76MiB	166.11KiB	3.4KiB	0.00913134	False	False
http_pipelines_blackhole_acks	31.38KiB	2.71	100.00%	1.13MiB	116.49KiB	2.37KiB	0.100653	1.16MiB	89.61KiB	1.83KiB	0.0753822	False	False
syslog_log2metric_splunk_hec_metrics	187.32KiB	0.96	100.00%	19.06MiB	616.0KiB	12.55KiB	0.031561	19.24MiB	857.83KiB	17.46KiB	0.0435332	False	False
syslog_splunk_hec_logs	152.38KiB	0.85	100.00%	17.43MiB	797.4KiB	16.23KiB	0.0446623	17.58MiB	744.7KiB	15.17KiB	0.0413577	False	False
http_text_to_http_json	141.39KiB	0.36	100.00%	38.5MiB	995.3KiB	20.32KiB	0.0252397	38.64MiB	829.21KiB	16.93KiB	0.0209528	False	False
syslog_regex_logs2metric_ddmetrics	47.87KiB	0.36	97.77%	12.95MiB	725.11KiB	14.77KiB	0.0546594	13.0MiB	728.65KiB	14.85KiB	0.0547282	False	False
socket_to_socket_blackhole	26.84KiB	0.2	99.79%	13.39MiB	370.01KiB	7.55KiB	0.0269707	13.42MiB	213.03KiB	4.35KiB	0.015498	False	False
splunk_hec_route_s3	33.16KiB	0.17	37.79%	19.23MiB	2.37MiB	49.29KiB	0.123101	19.26MiB	2.19MiB	45.79KiB	0.113716	False	False
splunk_hec_to_splunk_hec_logs_noack	15.84KiB	0.06	82.15%	23.82MiB	473.13KiB	9.66KiB	0.0193925	23.84MiB	329.63KiB	6.73KiB	0.0135022	False	False
splunk_hec_to_splunk_hec_logs_acks	10.97KiB	0.05	34.18%	23.74MiB	879.25KiB	17.88KiB	0.0361579	23.75MiB	844.89KiB	17.19KiB	0.0347294	False	False
splunk_hec_indexer_ack_blackhole	4.0KiB	0.02	12.46%	23.75MiB	890.54KiB	18.12KiB	0.0366044	23.76MiB	883.49KiB	17.98KiB	0.0363086	False	False
file_to_blackhole	-124.61KiB	-0.13	69.44%	95.32MiB	3.69MiB	76.35KiB	0.0386634	95.2MiB	4.56MiB	94.66KiB	0.0478776	False	False
http_to_http_json	-34.04KiB	-0.14	99.20%	23.84MiB	348.86KiB	7.12KiB	0.0142855	23.81MiB	523.26KiB	10.68KiB	0.0214565	False	False
fluent_elasticsearch	-167.27KiB	-0.21	100.00%	79.47MiB	52.52KiB	1.06KiB	0.000645277	79.31MiB	1.45MiB	29.84KiB	0.018282	False	False
http_to_http_noack	-76.59KiB	-0.31	99.81%	23.83MiB	521.46KiB	10.66KiB	0.0213687	23.75MiB	1.06MiB	22.21KiB	0.0448238	False	False
datadog_agent_remap_datadog_logs_acks	-222.5KiB	-0.35	94.95%	61.44MiB	2.94MiB	61.52KiB	0.0478143	61.22MiB	4.59MiB	95.64KiB	0.07503	False	False
syslog_log2metric_humio_metrics	-52.34KiB	-0.38	99.99%	13.29MiB	367.75KiB	7.5KiB	0.0270239	13.24MiB	554.61KiB	11.29KiB	0.0409133	False	False
datadog_agent_remap_blackhole	-284.78KiB	-0.44	98.85%	63.13MiB	4.35MiB	90.62KiB	0.0689003	62.85MiB	3.2MiB	66.85KiB	0.0509685	False	False
http_pipelines_blackhole	-7.29KiB	-0.44	99.28%	1.6MiB	44.9KiB	939.78B	0.0273508	1.6MiB	125.24KiB	2.55KiB	0.0766251	False	False
datadog_agent_remap_datadog_logs	-327.6KiB	-0.52	99.81%	61.04MiB	2.07MiB	43.46KiB	0.0339578	60.72MiB	4.6MiB	95.83KiB	0.0757835	False	False
datadog_agent_remap_blackhole_acks	-419.36KiB	-0.64	100.00%	63.94MiB	4.08MiB	85.0KiB	0.0637702	63.53MiB	2.68MiB	56.17KiB	0.0422116	False	False
syslog_loki	-152.45KiB	-1.04	100.00%	14.34MiB	390.69KiB	8.0KiB	0.0265914	14.2MiB	742.48KiB	15.09KiB	0.0510658	False	False
http_to_http_acks	-217.07KiB	-1.15	66.82%	18.47MiB	7.81MiB	163.36KiB	0.422837	18.26MiB	7.31MiB	152.77KiB	0.400439	True	True
http_pipelines_no_grok_blackhole	-161.24KiB	-1.38	100.00%	11.37MiB	191.58KiB	3.91KiB	0.0164442	11.22MiB	1.14MiB	23.8KiB	0.101899	False	False

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

github-actions · 2022-07-15T17:17:15Z

Soak Test Results

Baseline: e8f32c3
Comparison: 695873f
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_regex_logs2metric_ddmetrics	316.11KiB	2.43	100.00%	12.72MiB	729.69KiB	14.86KiB	0.0560152	13.03MiB	752.47KiB	15.33KiB	0.0563954	False	False
datadog_agent_remap_blackhole_acks	1.27MiB	2.03	100.00%	62.54MiB	4.44MiB	92.42KiB	0.070918	63.81MiB	2.48MiB	51.94KiB	0.0388009	False	False
syslog_splunk_hec_logs	216.27KiB	1.27	100.00%	16.67MiB	776.69KiB	15.8KiB	0.045489	16.88MiB	621.65KiB	12.68KiB	0.0359531	False	False
syslog_humio_logs	210.49KiB	1.17	100.00%	17.62MiB	249.87KiB	5.1KiB	0.0138459	17.83MiB	296.55KiB	6.08KiB	0.0162432	False	False
socket_to_socket_blackhole	133.53KiB	0.96	100.00%	13.64MiB	141.59KiB	2.89KiB	0.0101386	13.77MiB	138.85KiB	2.83KiB	0.00984785	False	False
datadog_agent_remap_datadog_logs_acks	529.28KiB	0.86	100.00%	60.45MiB	3.15MiB	65.86KiB	0.0521101	60.97MiB	4.49MiB	93.49KiB	0.0736529	False	False
syslog_log2metric_splunk_hec_metrics	107.38KiB	0.55	100.00%	19.05MiB	532.87KiB	10.87KiB	0.027304	19.16MiB	868.72KiB	17.68KiB	0.044269	False	False
http_to_http_acks	51.58KiB	0.28	16.91%	18.26MiB	8.23MiB	172.02KiB	0.450315	18.31MiB	8.12MiB	169.48KiB	0.443285	True	True
splunk_hec_to_splunk_hec_logs_noack	15.78KiB	0.06	82.40%	23.82MiB	463.44KiB	9.46KiB	0.0189949	23.84MiB	334.2KiB	6.82KiB	0.0136888	False	False
http_pipelines_blackhole_acks	730.73B	0.06	21.44%	1.17MiB	103.38KiB	2.11KiB	0.0864517	1.17MiB	76.77KiB	1.56KiB	0.0641622	False	False
splunk_hec_indexer_ack_blackhole	6.64KiB	0.03	19.56%	23.74MiB	944.22KiB	19.21KiB	0.0388251	23.75MiB	918.18KiB	18.68KiB	0.0377439	False	False
splunk_hec_to_splunk_hec_logs_acks	-16.74KiB	-0.07	49.91%	23.76MiB	826.55KiB	16.82KiB	0.03397	23.74MiB	900.83KiB	18.32KiB	0.0370486	False	False
file_to_blackhole	-73.7KiB	-0.08	50.05%	95.34MiB	3.34MiB	69.18KiB	0.0349949	95.27MiB	4.06MiB	84.39KiB	0.0426084	False	False
datadog_agent_remap_blackhole	-79.17KiB	-0.12	48.09%	62.82MiB	4.66MiB	97.01KiB	0.0741102	62.74MiB	3.61MiB	75.29KiB	0.0574777	False	False
http_to_http_json	-42.28KiB	-0.17	99.85%	23.85MiB	338.56KiB	6.91KiB	0.0138615	23.81MiB	558.43KiB	11.4KiB	0.0229033	False	False
fluent_elasticsearch	-196.33KiB	-0.24	100.00%	79.47MiB	52.96KiB	1.07KiB	0.000650617	79.28MiB	1.69MiB	34.8KiB	0.0213273	False	False
datadog_agent_remap_datadog_logs	-183.1KiB	-0.29	97.28%	62.6MiB	349.2KiB	7.15KiB	0.00544658	62.42MiB	3.96MiB	82.56KiB	0.0634731	False	False
splunk_hec_route_s3	-60.61KiB	-0.32	62.98%	18.77MiB	2.32MiB	48.25KiB	0.123298	18.71MiB	2.27MiB	47.4KiB	0.121158	False	False
http_to_http_noack	-103.95KiB	-0.43	100.00%	23.84MiB	409.75KiB	8.38KiB	0.016783	23.74MiB	1.14MiB	23.67KiB	0.0478221	False	False
http_text_to_http_json	-262.65KiB	-0.66	100.00%	38.91MiB	784.97KiB	16.02KiB	0.0196958	38.66MiB	818.73KiB	16.72KiB	0.0206794	False	False
http_pipelines_blackhole	-13.47KiB	-0.81	100.00%	1.63MiB	8.46KiB	177.08B	0.00506257	1.62MiB	104.49KiB	2.13KiB	0.0630015	False	False
syslog_loki	-155.13KiB	-1.06	100.00%	14.27MiB	580.07KiB	11.87KiB	0.0396951	14.12MiB	807.46KiB	16.41KiB	0.0558487	False	False
syslog_log2metric_humio_metrics	-153.47KiB	-1.13	100.00%	13.29MiB	247.99KiB	5.06KiB	0.0182145	13.14MiB	598.47KiB	12.18KiB	0.0444583	False	False
http_pipelines_no_grok_blackhole	-211.16KiB	-1.8	100.00%	11.47MiB	56.06KiB	1.14KiB	0.00477361	11.26MiB	1.09MiB	22.68KiB	0.0966834	False	False

github-actions · 2022-07-15T17:35:53Z

Soak Test Results

Baseline: e8f32c3
Comparison: c6b0ce7
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_splunk_hec_logs	350.69KiB	2.04	100.00%	16.81MiB	870.77KiB	17.71KiB	0.0505847	17.15MiB	778.06KiB	15.86KiB	0.0442962	False	False
syslog_log2metric_splunk_hec_metrics	248.08KiB	1.28	100.00%	18.86MiB	855.93KiB	17.43KiB	0.0442991	19.11MiB	801.73KiB	16.33KiB	0.0409678	False	False
datadog_agent_remap_blackhole	546.68KiB	0.93	99.20%	57.55MiB	7.51MiB	156.4KiB	0.130421	58.09MiB	6.43MiB	134.27KiB	0.110663	False	False
syslog_humio_logs	147.33KiB	0.83	100.00%	17.36MiB	579.68KiB	11.83KiB	0.0326029	17.5MiB	538.83KiB	11.03KiB	0.0300561	False	False
syslog_regex_logs2metric_ddmetrics	107.53KiB	0.8	100.00%	13.11MiB	720.98KiB	14.68KiB	0.0536879	13.22MiB	588.21KiB	11.99KiB	0.0434534	False	False
http_to_http_acks	130.01KiB	0.71	42.29%	18.01MiB	7.66MiB	160.07KiB	0.425201	18.13MiB	8.13MiB	169.5KiB	0.44811	True	True
splunk_hec_route_s3	98.79KiB	0.51	85.45%	18.77MiB	2.34MiB	48.78KiB	0.12478	18.87MiB	2.26MiB	47.18KiB	0.119589	False	False
splunk_hec_to_splunk_hec_logs_noack	16.85KiB	0.07	84.59%	23.82MiB	472.34KiB	9.64KiB	0.0193604	23.84MiB	335.05KiB	6.84KiB	0.0137237	False	False
splunk_hec_to_splunk_hec_logs_acks	3.0KiB	0.01	9.47%	23.74MiB	881.31KiB	17.92KiB	0.0362414	23.75MiB	872.03KiB	17.73KiB	0.0358555	False	False
splunk_hec_indexer_ack_blackhole	946.59B	0	2.72%	23.74MiB	942.22KiB	19.16KiB	0.0387511	23.74MiB	946.52KiB	19.25KiB	0.0389264	False	False
file_to_blackhole	-41.67KiB	-0.04	27.74%	95.33MiB	3.99MiB	82.77KiB	0.0418824	95.29MiB	4.0MiB	83.18KiB	0.0419456	False	False
http_to_http_json	-43.54KiB	-0.18	99.87%	23.85MiB	344.38KiB	7.03KiB	0.014101	23.8MiB	567.88KiB	11.59KiB	0.0232941	False	False
datadog_agent_remap_blackhole_acks	-125.21KiB	-0.19	76.27%	63.65MiB	4.32MiB	89.96KiB	0.067863	63.52MiB	2.67MiB	55.96KiB	0.0420805	False	False
http_pipelines_blackhole_acks	-2.23KiB	-0.2	50.60%	1.12MiB	115.12KiB	2.34KiB	0.100638	1.11MiB	111.57KiB	2.27KiB	0.0977256	False	False
fluent_elasticsearch	-296.99KiB	-0.36	100.00%	79.47MiB	53.72KiB	1.08KiB	0.000660006	79.18MiB	3.26MiB	67.05KiB	0.0411962	False	False
syslog_log2metric_humio_metrics	-50.18KiB	-0.37	100.00%	13.19MiB	269.56KiB	5.5KiB	0.0199575	13.14MiB	502.98KiB	10.24KiB	0.0373785	False	False
http_to_http_noack	-104.37KiB	-0.43	99.99%	23.83MiB	511.73KiB	10.46KiB	0.0209675	23.73MiB	1.18MiB	24.5KiB	0.0495282	False	False
http_text_to_http_json	-188.08KiB	-0.48	100.00%	38.61MiB	947.92KiB	19.35KiB	0.0239714	38.43MiB	965.28KiB	19.71KiB	0.0245273	False	False
http_pipelines_blackhole	-9.82KiB	-0.59	100.00%	1.62MiB	29.54KiB	618.41B	0.0178382	1.61MiB	106.47KiB	2.17KiB	0.0646858	False	False
datadog_agent_remap_datadog_logs	-392.18KiB	-0.68	89.83%	56.25MiB	7.61MiB	159.53KiB	0.135195	55.87MiB	8.59MiB	178.77KiB	0.153682	False	False
socket_to_socket_blackhole	-117.66KiB	-0.84	100.00%	13.65MiB	199.59KiB	4.07KiB	0.0142732	13.54MiB	485.1KiB	9.9KiB	0.0349847	False	False
syslog_loki	-176.39KiB	-1.17	100.00%	14.71MiB	246.68KiB	5.05KiB	0.0163689	14.54MiB	715.87KiB	14.55KiB	0.0480667	False	False
http_pipelines_no_grok_blackhole	-217.82KiB	-1.88	100.00%	11.34MiB	242.69KiB	4.95KiB	0.0208948	11.13MiB	1.11MiB	23.1KiB	0.0996762	False	False
datadog_agent_remap_datadog_logs_acks	-1.33MiB	-2.11	100.00%	62.85MiB	2.85MiB	59.66KiB	0.0453682	61.52MiB	4.81MiB	100.13KiB	0.0781748	False	False

github-actions · 2022-07-15T18:33:45Z

Soak Test Results

Baseline: e8f32c3
Comparison: b77dab6
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
socket_to_socket_blackhole	148.38KiB	1.06	100.00%	13.66MiB	152.81KiB	3.12KiB	0.0109193	13.81MiB	133.49KiB	2.73KiB	0.00943843	False	False
syslog_regex_logs2metric_ddmetrics	122.06KiB	0.91	100.00%	13.06MiB	660.93KiB	13.47KiB	0.0494093	13.18MiB	679.14KiB	13.83KiB	0.0503112	False	False
syslog_humio_logs	161.3KiB	0.89	100.00%	17.75MiB	157.99KiB	3.22KiB	0.00869119	17.91MiB	172.58KiB	3.53KiB	0.0094106	False	False
syslog_splunk_hec_logs	148.94KiB	0.86	100.00%	16.95MiB	912.42KiB	18.57KiB	0.0525458	17.1MiB	882.36KiB	17.97KiB	0.0503823	False	False
http_to_http_acks	143.15KiB	0.78	44.56%	17.88MiB	8.41MiB	175.75KiB	0.470142	18.02MiB	7.98MiB	166.52KiB	0.442639	True	True
syslog_log2metric_splunk_hec_metrics	148.28KiB	0.76	100.00%	19.01MiB	580.26KiB	11.83KiB	0.0298049	19.15MiB	773.3KiB	15.75KiB	0.0394197	False	False
splunk_hec_route_s3	130.88KiB	0.69	94.93%	18.53MiB	2.29MiB	47.61KiB	0.123367	18.66MiB	2.25MiB	47.08KiB	0.12063	False	False
datadog_agent_remap_blackhole_acks	385.38KiB	0.59	99.88%	64.19MiB	4.6MiB	95.69KiB	0.0715861	64.57MiB	3.36MiB	70.15KiB	0.0519854	False	False
datadog_agent_remap_datadog_logs_acks	232.01KiB	0.38	92.66%	58.98MiB	3.95MiB	82.49KiB	0.06693	59.21MiB	4.8MiB	99.92KiB	0.0810573	False	False
splunk_hec_to_splunk_hec_logs_noack	27.08KiB	0.11	96.28%	23.81MiB	543.44KiB	11.08KiB	0.0222842	23.84MiB	332.45KiB	6.79KiB	0.0136173	False	False
splunk_hec_to_splunk_hec_logs_acks	-5.67KiB	-0.02	18.25%	23.75MiB	836.51KiB	17.01KiB	0.0343834	23.75MiB	872.92KiB	17.75KiB	0.0358885	False	False
splunk_hec_indexer_ack_blackhole	-6.35KiB	-0.03	20.71%	23.76MiB	822.73KiB	16.74KiB	0.0338068	23.75MiB	856.9KiB	17.43KiB	0.0352199	False	False
file_to_blackhole	-37.42KiB	-0.04	31.27%	95.34MiB	2.97MiB	61.61KiB	0.0311652	95.3MiB	3.34MiB	69.59KiB	0.0350785	False	False
http_to_http_json	-45.29KiB	-0.19	99.91%	23.84MiB	345.33KiB	7.05KiB	0.0141407	23.8MiB	571.52KiB	11.66KiB	0.0234462	False	False
fluent_elasticsearch	-161.72KiB	-0.2	100.00%	79.47MiB	53.02KiB	1.07KiB	0.00065142	79.32MiB	1.39MiB	28.66KiB	0.0175611	False	False
datadog_agent_remap_blackhole	-280.29KiB	-0.46	98.93%	59.82MiB	4.07MiB	84.79KiB	0.0679963	59.55MiB	3.34MiB	69.76KiB	0.0561419	False	False
datadog_agent_remap_datadog_logs	-301.46KiB	-0.47	99.92%	62.62MiB	323.54KiB	6.62KiB	0.00504466	62.32MiB	4.3MiB	89.6KiB	0.0690382	False	False
http_text_to_http_json	-206.59KiB	-0.52	100.00%	38.77MiB	834.34KiB	17.03KiB	0.0210133	38.56MiB	824.88KiB	16.84KiB	0.0208839	False	False
http_to_http_noack	-129.02KiB	-0.53	100.00%	23.84MiB	404.06KiB	8.26KiB	0.0165499	23.71MiB	1.25MiB	26.04KiB	0.0527018	False	False
http_pipelines_blackhole	-12.71KiB	-0.76	100.00%	1.64MiB	17.12KiB	358.31B	0.0102038	1.63MiB	101.61KiB	2.07KiB	0.0610218	False	False
syslog_log2metric_humio_metrics	-142.84KiB	-1.04	100.00%	13.38MiB	271.74KiB	5.55KiB	0.0198236	13.24MiB	601.2KiB	12.23KiB	0.0443197	False	False
syslog_loki	-167.38KiB	-1.12	100.00%	14.65MiB	382.74KiB	7.84KiB	0.0255018	14.49MiB	761.94KiB	15.49KiB	0.0513413	False	False
http_pipelines_no_grok_blackhole	-164.53KiB	-1.43	100.00%	11.26MiB	210.81KiB	4.3KiB	0.0182818	11.1MiB	1.14MiB	23.82KiB	0.103082	False	False
http_pipelines_blackhole_acks	-20.46KiB	-1.76	100.00%	1.14MiB	101.26KiB	2.06KiB	0.0871029	1.12MiB	82.39KiB	1.68KiB	0.0721416	False	False

caibirdme · 2022-07-16T02:03:49Z

Glad to see this merged,we can continue to support log sink and otel trace

julealgon · 2022-07-18T20:25:04Z

Could someone elaborate which version of Vector will contain this enhancement?

I want to experiment with it ASAP as a means to workaround the lack of a Datadog log exporter in OTel collector.

michalsteyn · 2022-07-18T22:25:46Z

You can find it under the Nightly Builds: https://github.com/vectordotdev/vector/actions/workflows/nightly.yml

caibirdme · 2022-07-19T01:23:19Z

Could someone elaborate which version of Vector will contain this enhancement?

I want to experiment with it ASAP as a means to workaround the lack of a Datadog log exporter in OTel collector.

Or You can try docker image(latest, nightly) directly from: https://hub.docker.com/r/timberio/vector/tags,

spencergilbert · 2022-07-19T12:35:59Z

@julealgon - I'd also expect this to be released on a stable release with 0.24 but that's not confirmed, keep an eye on the release notes.

If you're using a nightly version, the docs are available here.

feat: support opentelemetry log source

401b5e9

github-actions bot added the domain: sources Anything related to the Vector's sources label Jun 24, 2022

caibirdme and others added 2 commits June 27, 2022 13:09

fix fmt

a5d4770

Merge branch 'vectordotdev:master' into opentelemetry-add-log

99a486c

This comment was marked as outdated.

Sign in to view

prognant suggested changes Jun 28, 2022

View reviewed changes

lib/opentelemetry/src/convert.rs Outdated Show resolved Hide resolved

src/sources/opentelemetry/log.rs Outdated Show resolved Hide resolved

src/sources/opentelemetry/mod.rs Outdated Show resolved Hide resolved

caibirdme added 3 commits June 29, 2022 14:59

Merge branch 'master' of https://github.com/vectordotdev/vector into …

b83c5d4

…opentelemetry-add-log

add opentelemetry.cue & add multi_output config & rename

94c842c

Merge branch 'opentelemetry-add-log' of github.com:caibirdme/vector i…

be3ef05

…nto opentelemetry-add-log

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Jun 29, 2022

This comment was marked as outdated.

Sign in to view

Merge branch 'vectordotdev:master' into opentelemetry-add-log

22a567b

This comment was marked as outdated.

Sign in to view

prognant reviewed Jun 30, 2022

View reviewed changes

src/sources/opentelemetry/log.rs Outdated Show resolved Hide resolved

Merge branch 'vectordotdev:master' into opentelemetry-add-log

e0e1a80

This comment was marked as outdated.

Sign in to view

add test for otel log

b26a0d3

This comment was marked as outdated.

Sign in to view

caibirdme added 2 commits July 2, 2022 21:42

Merge branch 'opentelemetry-add-log' of github.com:caibirdme/vector i…

b4fdbdf

…nto opentelemetry-add-log

fix dep

034dd06

github-actions bot added domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: sinks Anything related to the Vector's sinks labels Jul 2, 2022

This comment was marked as outdated.

Sign in to view

feat: remove deprecated pb field

93ae0e6

Merge branch 'master' of https://github.com/vectordotdev/vector into …

1927594

…opentelemetry-add-log

convert timestamp&fix unit test

e8cffd0

spencergilbert added 2 commits July 15, 2022 10:59

include generate_config test

ea91b37

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

convert unix timestamps to Value::Timestamp

9df6343

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

spencergilbert requested review from StephenWakely and jszwedko July 15, 2022 15:12

use configurable trait

3aefaa2

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

prognant approved these changes Jul 15, 2022

View reviewed changes

Cargo.toml Show resolved Hide resolved

update features

695873f

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

StephenWakely approved these changes Jul 15, 2022

View reviewed changes

update docs for timestamp conversion

c6b0ce7

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

spencergilbert enabled auto-merge (squash) July 15, 2022 16:01

still needs vector features, need to refactor

b77dab6

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

spencergilbert approved these changes Jul 15, 2022

View reviewed changes

spencergilbert merged commit 2dd6a20 into vectordotdev:master Jul 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sources): opentelemetry log #13320

feat(sources): opentelemetry log #13320

caibirdme commented Jun 24, 2022 •

edited

bits-bot commented Jun 24, 2022 •

edited

netlify bot commented Jun 24, 2022 •

edited

caibirdme commented Jun 24, 2022

This comment was marked as outdated.

This comment was marked as outdated.

prognant left a comment •

edited

This comment was marked as outdated.

caibirdme commented Jun 30, 2022

This comment was marked as outdated.

prognant left a comment

This comment was marked as outdated.

This comment was marked as outdated.

caibirdme commented Jul 2, 2022

This comment was marked as outdated.

github-actions bot commented Jul 14, 2022

caibirdme commented Jul 15, 2022

prognant commented Jul 15, 2022 •

edited

spencergilbert commented Jul 15, 2022

caibirdme commented Jul 15, 2022

prognant left a comment

github-actions bot commented Jul 15, 2022

StephenWakely left a comment

github-actions bot commented Jul 15, 2022

github-actions bot commented Jul 15, 2022

github-actions bot commented Jul 15, 2022

github-actions bot commented Jul 15, 2022

caibirdme commented Jul 16, 2022

julealgon commented Jul 18, 2022

michalsteyn commented Jul 18, 2022

caibirdme commented Jul 19, 2022

spencergilbert commented Jul 19, 2022

feat(sources): opentelemetry log #13320

feat(sources): opentelemetry log #13320

Conversation

caibirdme commented Jun 24, 2022 • edited

bits-bot commented Jun 24, 2022 • edited

netlify bot commented Jun 24, 2022 • edited

✅ Deploy Preview for vector-project canceled.

caibirdme commented Jun 24, 2022

This comment was marked as outdated.

This comment was marked as outdated.

prognant left a comment • edited

Choose a reason for hiding this comment

This comment was marked as outdated.

caibirdme commented Jun 30, 2022

This comment was marked as outdated.

prognant left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

caibirdme commented Jul 2, 2022

This comment was marked as outdated.

github-actions bot commented Jul 14, 2022

Soak Test Results

caibirdme commented Jul 15, 2022

prognant commented Jul 15, 2022 • edited

spencergilbert commented Jul 15, 2022

caibirdme commented Jul 15, 2022

prognant left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 15, 2022

Soak Test Results

StephenWakely left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 15, 2022

Soak Test Results

github-actions bot commented Jul 15, 2022

Soak Test Results

github-actions bot commented Jul 15, 2022

Soak Test Results

github-actions bot commented Jul 15, 2022

Soak Test Results

caibirdme commented Jul 16, 2022

julealgon commented Jul 18, 2022

michalsteyn commented Jul 18, 2022

caibirdme commented Jul 19, 2022

spencergilbert commented Jul 19, 2022

caibirdme commented Jun 24, 2022 •

edited

bits-bot commented Jun 24, 2022 •

edited

netlify bot commented Jun 24, 2022 •

edited

prognant left a comment •

edited

prognant commented Jul 15, 2022 •

edited