Track payload size for Kafka integration #6045

vandonr · 2023-10-13T09:51:11Z

This is mirroring part of the features that were added to the .net tracer in DataDog/dd-trace-dotnet#4520, on DSM team's request.

tested locally with a sample producer and consumer, I can see the metric being emited in my DD org

Jira ticket: AIT-8557

pr-commenter · 2023-10-13T10:33:47Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.23.0-SNAPSHOT~c2cfdeaf5e	1.23.0-SNAPSHOT~6d34af9fa9
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 52 cases.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:petclinic:iast:IAST	worse [+2.647ms; +7.341ms] or [+18.575%; +51.520%]	19.243ms	14.249ms
scenario:petclinic:iast:Telemetry	better [-6.871ms; -1.672ms] or [-66.070%; -16.076%]	6.128ms	10.399ms

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.23.0-SNAPSHOT~6d34af9fa9, baseline=1.23.0-SNAPSHOT~c2cfdeaf5e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1025617
Total [baseline] (9.277 s) : 0, 9277462
Agent [candidate] (1.034 s) : 0, 1033857
Total [candidate] (9.307 s) : 0, 9306940
section appsec
Agent [baseline] (1.113 s) : 0, 1113255
Total [baseline] (9.342 s) : 0, 9342114
Agent [candidate] (1.124 s) : 0, 1124468
Total [candidate] (9.427 s) : 0, 9426787
section iast
Agent [baseline] (1.152 s) : 0, 1152012
Total [baseline] (9.521 s) : 0, 9520896
Agent [candidate] (1.154 s) : 0, 1154490
Total [candidate] (9.534 s) : 0, 9534348
section profiling
Agent [baseline] (1.21 s) : 0, 1209865
Total [baseline] (9.546 s) : 0, 9546165
Agent [candidate] (1.211 s) : 0, 1210952
Total [candidate] (9.504 s) : 0, 9503697

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	appsec	1.113 s	87.638 ms (8.5%)
Agent	iast	1.152 s	126.395 ms (12.3%)
Agent	profiling	1.21 s	184.248 ms (18.0%)
Total	tracing	9.277 s	-
Total	appsec	9.342 s	64.652 ms (0.7%)
Total	iast	9.521 s	243.434 ms (2.6%)
Total	profiling	9.546 s	268.704 ms (2.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.034 s	-
Agent	appsec	1.124 s	90.611 ms (8.8%)
Agent	iast	1.154 s	120.633 ms (11.7%)
Agent	profiling	1.211 s	177.096 ms (17.1%)
Total	tracing	9.307 s	-
Total	appsec	9.427 s	119.847 ms (1.3%)
Total	iast	9.534 s	227.408 ms (2.4%)
Total	profiling	9.504 s	196.757 ms (2.1%)

gantt
    title petclinic - break down per module: candidate=1.23.0-SNAPSHOT~6d34af9fa9, baseline=1.23.0-SNAPSHOT~c2cfdeaf5e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (641.377 ms) : 0, 641377
BytebuddyAgent [candidate] (646.418 ms) : 0, 646418
GlobalTracer [baseline] (293.975 ms) : 0, 293975
GlobalTracer [candidate] (296.354 ms) : 0, 296354
AppSec [baseline] (48.994 ms) : 0, 48994
AppSec [candidate] (49.475 ms) : 0, 49475
Remote Config [baseline] (692.694 µs) : 0, 693
Remote Config [candidate] (700.01 µs) : 0, 700
Telemetry [baseline] (6.094 ms) : 0, 6094
Telemetry [candidate] (6.198 ms) : 0, 6198
section appsec
BytebuddyAgent [baseline] (641.055 ms) : 0, 641055
BytebuddyAgent [candidate] (648.118 ms) : 0, 648118
GlobalTracer [baseline] (293.146 ms) : 0, 293146
GlobalTracer [candidate] (296.096 ms) : 0, 296096
AppSec [baseline] (138.36 ms) : 0, 138360
AppSec [candidate] (139.061 ms) : 0, 139061
Remote Config [baseline] (642.432 µs) : 0, 642
Remote Config [candidate] (645.938 µs) : 0, 646
Telemetry [baseline] (5.717 ms) : 0, 5717
Telemetry [candidate] (5.824 ms) : 0, 5824
section iast
BytebuddyAgent [baseline] (771.489 ms) : 0, 771489
BytebuddyAgent [candidate] (772.682 ms) : 0, 772682
GlobalTracer [baseline] (273.315 ms) : 0, 273315
GlobalTracer [candidate] (274.193 ms) : 0, 274193
AppSec [baseline] (47.397 ms) : 0, 47397
AppSec [candidate] (47.165 ms) : 0, 47165
Remote Config [baseline] (581.661 µs) : 0, 582
Remote Config [candidate] (584.065 µs) : 0, 584
Telemetry [baseline] (10.399 ms) : 0, 10399
Telemetry [candidate] (6.128 ms) : 0, 6128
IAST [baseline] (14.249 ms) : 0, 14249
IAST [candidate] (19.243 ms) : 0, 19243
section profiling
BytebuddyAgent [baseline] (656.976 ms) : 0, 656976
BytebuddyAgent [candidate] (658.664 ms) : 0, 658664
GlobalTracer [baseline] (360.93 ms) : 0, 360930
GlobalTracer [candidate] (360.705 ms) : 0, 360705
AppSec [baseline] (50.199 ms) : 0, 50199
AppSec [candidate] (50.004 ms) : 0, 50004
Remote Config [baseline] (646.77 µs) : 0, 647
Remote Config [candidate] (656.504 µs) : 0, 657
Telemetry [baseline] (6.084 ms) : 0, 6084
Telemetry [candidate] (6.059 ms) : 0, 6059
ProfilingAgent [baseline] (81.211 ms) : 0, 81211
ProfilingAgent [candidate] (81.081 ms) : 0, 81081
Profiling [baseline] (81.235 ms) : 0, 81235
Profiling [candidate] (81.105 ms) : 0, 81105

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.23.0-SNAPSHOT~6d34af9fa9, baseline=1.23.0-SNAPSHOT~c2cfdeaf5e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.025 s) : 0, 1025453
Total [baseline] (8.736 s) : 0, 8735925
Agent [candidate] (1.026 s) : 0, 1026355
Total [candidate] (8.758 s) : 0, 8758233
section iast
Agent [baseline] (1.161 s) : 0, 1161257
Total [baseline] (9.313 s) : 0, 9313417
Agent [candidate] (1.155 s) : 0, 1155194
Total [candidate] (9.267 s) : 0, 9267378
section iast_TELEMETRY_OFF
Agent [baseline] (1.145 s) : 0, 1145485
Total [baseline] (9.256 s) : 0, 9256469
Agent [candidate] (1.145 s) : 0, 1145263
Total [candidate] (9.271 s) : 0, 9270511

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.025 s	-
Agent	iast	1.161 s	135.804 ms (13.2%)
Agent	iast_TELEMETRY_OFF	1.145 s	120.032 ms (11.7%)
Total	tracing	8.736 s	-
Total	iast	9.313 s	577.491 ms (6.6%)
Total	iast_TELEMETRY_OFF	9.256 s	520.544 ms (6.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	iast	1.155 s	128.839 ms (12.6%)
Agent	iast_TELEMETRY_OFF	1.145 s	118.909 ms (11.6%)
Total	tracing	8.758 s	-
Total	iast	9.267 s	509.145 ms (5.8%)
Total	iast_TELEMETRY_OFF	9.271 s	512.278 ms (5.8%)

gantt
    title insecure-bank - break down per module: candidate=1.23.0-SNAPSHOT~6d34af9fa9, baseline=1.23.0-SNAPSHOT~c2cfdeaf5e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (641.492 ms) : 0, 641492
BytebuddyAgent [candidate] (641.584 ms) : 0, 641584
GlobalTracer [baseline] (293.542 ms) : 0, 293542
GlobalTracer [candidate] (294.396 ms) : 0, 294396
AppSec [baseline] (49.21 ms) : 0, 49210
AppSec [candidate] (49.144 ms) : 0, 49144
Remote Config [baseline] (677.745 µs) : 0, 678
Remote Config [candidate] (686.053 µs) : 0, 686
Telemetry [baseline] (6.042 ms) : 0, 6042
Telemetry [candidate] (6.096 ms) : 0, 6096
section iast
BytebuddyAgent [baseline] (777.584 ms) : 0, 777584
BytebuddyAgent [candidate] (773.809 ms) : 0, 773809
GlobalTracer [baseline] (275.905 ms) : 0, 275905
GlobalTracer [candidate] (274.104 ms) : 0, 274104
AppSec [baseline] (47.669 ms) : 0, 47669
AppSec [candidate] (46.187 ms) : 0, 46187
IAST [baseline] (17.12 ms) : 0, 17120
IAST [candidate] (16.995 ms) : 0, 16995
Remote Config [baseline] (588.105 µs) : 0, 588
Remote Config [candidate] (597.845 µs) : 0, 598
Telemetry [baseline] (7.646 ms) : 0, 7646
Telemetry [candidate] (8.864 ms) : 0, 8864
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (765.04 ms) : 0, 765040
BytebuddyAgent [candidate] (765.065 ms) : 0, 765065
GlobalTracer [baseline] (274.039 ms) : 0, 274039
GlobalTracer [candidate] (274.144 ms) : 0, 274144
AppSec [baseline] (46.605 ms) : 0, 46605
AppSec [candidate] (46.641 ms) : 0, 46641
IAST [baseline] (16.432 ms) : 0, 16432
IAST [candidate] (17.638 ms) : 0, 17638
Remote Config [baseline] (653.366 µs) : 0, 653
Remote Config [candidate] (602.247 µs) : 0, 602
Telemetry [baseline] (8.179 ms) : 0, 8179
Telemetry [candidate] (6.785 ms) : 0, 6785

Load

Parameters

	Baseline	Candidate
commit	1.23.0-SNAPSHOT~c2cfdeaf5e	1.23.0-SNAPSHOT~6d34af9fa9
config	baseline	candidate
end_time	2023-10-27T13:55:12	2023-10-27T14:11:33
start_time	2023-10-27T13:55:00	2023-10-27T14:11:21

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 22 cases.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.23.0-SNAPSHOT~6d34af9fa9, baseline=1.23.0-SNAPSHOT~c2cfdeaf5e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.324 ms) : 1305, 1343
.   : milestone, 1324,
appsec (1.712 ms) : 1688, 1737
.   : milestone, 1712,
iast (1.457 ms) : 1433, 1482
.   : milestone, 1457,
profiling (1.463 ms) : 1438, 1487
.   : milestone, 1463,
tracing (1.46 ms) : 1435, 1485
.   : milestone, 1460,
section candidate
no_agent (1.344 ms) : 1326, 1363
.   : milestone, 1344,
appsec (1.701 ms) : 1676, 1726
.   : milestone, 1701,
iast (1.454 ms) : 1429, 1478
.   : milestone, 1454,
profiling (1.499 ms) : 1473, 1525
.   : milestone, 1499,
tracing (1.461 ms) : 1436, 1485
.   : milestone, 1461,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.324 ms [1.305 ms, 1.343 ms]	-
appsec	1.712 ms [1.688 ms, 1.737 ms]	388.832 µs (29.4%)
iast	1.457 ms [1.433 ms, 1.482 ms]	133.764 µs (10.1%)
profiling	1.463 ms [1.438 ms, 1.487 ms]	138.982 µs (10.5%)
tracing	1.46 ms [1.435 ms, 1.485 ms]	136.478 µs (10.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.344 ms [1.326 ms, 1.363 ms]	-
appsec	1.701 ms [1.676 ms, 1.726 ms]	356.548 µs (26.5%)
iast	1.454 ms [1.429 ms, 1.478 ms]	109.329 µs (8.1%)
profiling	1.499 ms [1.473 ms, 1.525 ms]	154.764 µs (11.5%)
tracing	1.461 ms [1.436 ms, 1.485 ms]	116.41 µs (8.7%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.23.0-SNAPSHOT~6d34af9fa9, baseline=1.23.0-SNAPSHOT~c2cfdeaf5e
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.248 µs) : 344, 386
.   : milestone, 365,
iast (459.553 µs) : 439, 480
.   : milestone, 460,
iast_FULL (523.134 µs) : 502, 544
.   : milestone, 523,
iast_INACTIVE (431.706 µs) : 411, 453
.   : milestone, 432,
iast_TELEMETRY_OFF (457.224 µs) : 436, 478
.   : milestone, 457,
tracing (431.785 µs) : 410, 453
.   : milestone, 432,
section candidate
no_agent (366.769 µs) : 344, 389
.   : milestone, 367,
iast (462.248 µs) : 441, 483
.   : milestone, 462,
iast_FULL (527.019 µs) : 506, 548
.   : milestone, 527,
iast_INACTIVE (429.705 µs) : 409, 451
.   : milestone, 430,
iast_TELEMETRY_OFF (457.531 µs) : 436, 479
.   : milestone, 458,
tracing (429.761 µs) : 409, 450
.   : milestone, 430,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	365.248 µs [344.408 µs, 386.089 µs]	-
iast	459.553 µs [438.899 µs, 480.206 µs]	94.304 µs (25.8%)
iast_FULL	523.134 µs [502.475 µs, 543.793 µs]	157.885 µs (43.2%)
iast_INACTIVE	431.706 µs [410.824 µs, 452.589 µs]	66.458 µs (18.2%)
iast_TELEMETRY_OFF	457.224 µs [436.116 µs, 478.332 µs]	91.976 µs (25.2%)
tracing	431.785 µs [410.076 µs, 453.493 µs]	66.536 µs (18.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.769 µs [344.302 µs, 389.236 µs]	-
iast	462.248 µs [441.29 µs, 483.205 µs]	95.479 µs (26.0%)
iast_FULL	527.019 µs [506.273 µs, 547.765 µs]	160.25 µs (43.7%)
iast_INACTIVE	429.705 µs [408.755 µs, 450.655 µs]	62.937 µs (17.2%)
iast_TELEMETRY_OFF	457.531 µs [436.274 µs, 478.788 µs]	90.763 µs (24.7%)
tracing	429.761 µs [409.227 µs, 450.296 µs]	62.993 µs (17.2%)

...rc/main/java/datadog/trace/instrumentation/armeria/grpc/server/TracingServerInterceptor.java

...aws-java-sqs-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sqs/TracingIterator.java

...pc-1.5/src/main/java/datadog/trace/instrumentation/grpc/server/TracingServerInterceptor.java

...rc/main/java/datadog/trace/instrumentation/kafka_streams/KafkaStreamTaskInstrumentation.java

...itmq-amqp-2.7/src/main/java/datadog/trace/instrumentation/rabbitmq/amqp/RabbitDecorator.java

dd-trace-core/src/main/java/datadog/trace/core/datastreams/DefaultDataStreamsMonitoring.java

...rc/main/java/datadog/trace/instrumentation/kafka_streams/KafkaStreamTaskInstrumentation.java

internal-api/src/main/java/datadog/trace/bootstrap/instrumentation/api/StatsPoint.java

dd-trace-core/src/main/java/datadog/trace/core/datastreams/DefaultPathwayContext.java

...-clients-0.11/src/main/java/datadog/trace/instrumentation/kafka_clients/TracingIterator.java

PerfectSlayer · 2023-10-19T11:21:35Z

Context? Goal? Description?

vandonr · 2023-10-19T11:45:16Z

Context? Goal? Description?

I added a little paragraph in the description, but I don't have a whole lot of info to add.

PerfectSlayer

One change request for internal tracer API.
I will let the two other default reviewers review the PR.

...pi/src/main/java/datadog/trace/bootstrap/instrumentation/api/AgentDataStreamsMonitoring.java

...tion/kafka-clients-0.11/src/main/java/datadog/trace/instrumentation/kafka_clients/Utils.java

PerfectSlayer

Thanks for the API edition

Author fixed the API concerns

kr-igor

LGTM

amarziali · 2023-10-30T08:50:27Z

...itmq-amqp-2.7/src/main/java/datadog/trace/instrumentation/rabbitmq/amqp/RabbitDecorator.java

-      AgentTracer.get().getDataStreamsMonitoring().setCheckpoint(span, sortedTags, produceMillis);
+      AgentTracer.get()
+          .getDataStreamsMonitoring()
+          .setCheckpoint(span, sortedTags, produceMillis, 0);


Nit: since you have the signature without the payload size you can just use it (and leave that class unchanged). It will generate a smaller diff (the same applied for all the other decorators)

no I don't have it anymore, @PerfectSlayer asked in #6045 (comment) that I remove it !

amarziali

Looks ok. I left a comment. Please let apm-java also review that PR

PerfectSlayer

LGTM

PerfectSlayer · 2023-10-30T09:38:10Z

dd-trace-core/src/main/java/datadog/trace/core/datastreams/StatsGroup.java

    pathwayLatency.accept(((double) pathwayLatencyNano) / NANOSECONDS_TO_SECOND);
    edgeLatency.accept(((double) edgeLatencyNano) / NANOSECONDS_TO_SECOND);
+    // payload size is set to zero when we cannot compute it
+    // in that case, it's probably better to have an empty histogram than filling it with zeros
+    if (payloadSizeBytes != 0) payloadSize.accept((double) payloadSizeBytes);


I don't even know spotless would agree to use such kind of one-liner... 🤔

vandonr commented Oct 13, 2023

View reviewed changes

vandonr requested a review from tabgok October 13, 2023 11:13

vandonr commented Oct 13, 2023

View reviewed changes

...rc/main/java/datadog/trace/instrumentation/kafka_streams/KafkaStreamTaskInstrumentation.java Outdated Show resolved Hide resolved

vandonr requested a review from amarziali October 13, 2023 11:17

vandonr commented Oct 13, 2023

View reviewed changes

internal-api/src/main/java/datadog/trace/bootstrap/instrumentation/api/StatsPoint.java Show resolved Hide resolved

amarziali reviewed Oct 13, 2023

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/datastreams/DefaultPathwayContext.java Show resolved Hide resolved

amarziali reviewed Oct 13, 2023

View reviewed changes

...-clients-0.11/src/main/java/datadog/trace/instrumentation/kafka_clients/TracingIterator.java Outdated Show resolved Hide resolved

track payload size for kafka integration

53081cc

vandonr force-pushed the vandonr/payload branch from 17eacbc to 53081cc Compare October 19, 2023 09:20

vandonr changed the title ~~track payload size for messaging integrations~~ track payload size for kafka integration Oct 19, 2023

vandonr marked this pull request as ready for review October 19, 2023 09:56

vandonr requested a review from a team as a code owner October 19, 2023 09:56

vandonr requested review from ygree and am312 October 19, 2023 09:56

PerfectSlayer added type: enhancement inst: kafka Kafka instrumentation labels Oct 19, 2023

Merge remote-tracking branch 'origin/master' into vandonr/payload

2c9896e

vandonr mentioned this pull request Oct 19, 2023

Track payload size in Rabbit & SQS integrations #6067

Closed

PerfectSlayer previously requested changes Oct 24, 2023

View reviewed changes

...pi/src/main/java/datadog/trace/bootstrap/instrumentation/api/AgentDataStreamsMonitoring.java Outdated Show resolved Hide resolved

...tion/kafka-clients-0.11/src/main/java/datadog/trace/instrumentation/kafka_clients/Utils.java Outdated Show resolved Hide resolved

PerfectSlayer changed the title ~~track payload size for kafka integration~~ Track payload size for Kafka integration Oct 24, 2023

vandonr added 2 commits October 24, 2023 13:53

protect utils class

a362038

remove function overload

6551110

vandonr requested a review from PerfectSlayer October 25, 2023 09:53

Merge branch 'master' into vandonr/payload

375fdff

PerfectSlayer reviewed Oct 27, 2023

View reviewed changes

Merge branch 'master' into vandonr/payload

6d34af9

kr-igor approved these changes Oct 27, 2023

View reviewed changes

amarziali reviewed Oct 30, 2023

View reviewed changes

amarziali approved these changes Oct 30, 2023

View reviewed changes

PerfectSlayer approved these changes Oct 30, 2023

View reviewed changes

vandonr merged commit 17c097d into master Oct 31, 2023
67 of 69 checks passed

vandonr deleted the vandonr/payload branch October 31, 2023 10:17

github-actions bot added this to the 1.23.0 milestone Oct 31, 2023

vandonr mentioned this pull request Nov 15, 2023

add instrumentation to allow sending kafka payload size on produce #6228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track payload size for Kafka integration #6045

Track payload size for Kafka integration #6045

vandonr commented Oct 13, 2023 •

edited

Loading

pr-commenter bot commented Oct 13, 2023 •

edited

Loading

PerfectSlayer commented Oct 19, 2023

vandonr commented Oct 19, 2023

PerfectSlayer left a comment •

edited

Loading

PerfectSlayer left a comment

kr-igor left a comment

amarziali Oct 30, 2023

vandonr Oct 30, 2023

amarziali left a comment •

edited

Loading

PerfectSlayer left a comment

PerfectSlayer Oct 30, 2023

Track payload size for Kafka integration #6045

Track payload size for Kafka integration #6045

Conversation

vandonr commented Oct 13, 2023 • edited Loading

pr-commenter bot commented Oct 13, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

PerfectSlayer commented Oct 19, 2023

vandonr commented Oct 19, 2023

PerfectSlayer left a comment • edited Loading

Choose a reason for hiding this comment

PerfectSlayer left a comment

Choose a reason for hiding this comment

kr-igor left a comment

Choose a reason for hiding this comment

amarziali Oct 30, 2023

Choose a reason for hiding this comment

vandonr Oct 30, 2023

Choose a reason for hiding this comment

amarziali left a comment • edited Loading

Choose a reason for hiding this comment

PerfectSlayer left a comment

Choose a reason for hiding this comment

PerfectSlayer Oct 30, 2023

Choose a reason for hiding this comment

vandonr commented Oct 13, 2023 •

edited

Loading

pr-commenter bot commented Oct 13, 2023 •

edited

Loading

PerfectSlayer left a comment •

edited

Loading

amarziali left a comment •

edited

Loading