Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial flush interval for long running spans #6540

Merged
merged 1 commit into from
Feb 1, 2024
Merged

Initial flush interval for long running spans #6540

merged 1 commit into from
Feb 1, 2024

Conversation

paul-laffon-dd
Copy link
Contributor

What Does This Do

Add the parameter trace.experimental.long-running.initial.flush.interval defaulting to 20s to control how long before flushing long running traces for the first time

Motivation

Reduce the delay to see long running spans in datadog, without having to set a frequent flush interval

Additional Notes

@pr-commenter
Copy link

pr-commenter bot commented Jan 23, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master paul.laffon/initial-flush-long-running-spans
git_commit_date 1706091852 1706092801
git_commit_sha 98e7a43 9c13266
release_version 1.29.0-SNAPSHOT~98e7a4379b 1.29.0-SNAPSHOT~9c132668b9
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1706095370 1706095370
ci_job_id 417114320 417114320
ci_pipeline_id 27084065 27084065
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 6 performance improvements and 1 performance regressions! Performance is the same for 41 metrics, 6 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:AppSec better
[-7.117ms; -1.870ms] or [-12.497%; -3.284%]		 52.458ms 56.952ms
scenario:startup:insecure-bank:tracing:Telemetry better
[-499.048µs; -245.737µs] or [-6.573%; -3.237%]		 7.220ms 7.592ms
scenario:startup:petclinic:appsec:Remote Config better
[-68.000µs; -39.892µs] or [-9.705%; -5.693%]		 646.736µs 700.682µs
scenario:startup:petclinic:profiling:Remote Config worse
[+296.894µs; +342.907µs] or [+44.323%; +51.192%]		 989.745µs 669.844µs
scenario:startup:petclinic:profiling:Telemetry better
[-408.609µs; -169.174µs] or [-5.428%; -2.247%]		 7.239ms 7.528ms
scenario:startup:petclinic:tracing:Remote Config better
[-66.850µs; -31.005µs] or [-9.459%; -4.387%]		 657.844µs 706.772µs
scenario:startup:petclinic:tracing:Telemetry better
[-604.516µs; -276.369µs] or [-7.912%; -3.617%]		 7.200ms 7.641ms
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.29.0-SNAPSHOT~9c132668b9, baseline=1.29.0-SNAPSHOT~98e7a4379b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064060
Total [baseline] (9.368 s) : 0, 9367575
Agent [candidate] (1.057 s) : 0, 1057393
Total [candidate] (9.352 s) : 0, 9351998
section appsec
Agent [baseline] (1.161 s) : 0, 1161454
Total [baseline] (9.494 s) : 0, 9494031
Agent [candidate] (1.151 s) : 0, 1151500
Total [candidate] (9.461 s) : 0, 9461491
section iast
Agent [baseline] (1.193 s) : 0, 1192747
Total [baseline] (9.659 s) : 0, 9659490
Agent [candidate] (1.178 s) : 0, 1177832
Total [candidate] (9.635 s) : 0, 9635122
section profiling
Agent [baseline] (1.281 s) : 0, 1281047
Total [baseline] (9.579 s) : 0, 9579018
Agent [candidate] (1.28 s) : 0, 1280049
Total [candidate] (9.605 s) : 0, 9604952
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent appsec 1.161 s 97.394 ms (9.2%)
Agent iast 1.193 s 128.687 ms (12.1%)
Agent profiling 1.281 s 216.987 ms (20.4%)
Total tracing 9.368 s -
Total appsec 9.494 s 126.456 ms (1.3%)
Total iast 9.659 s 291.916 ms (3.1%)
Total profiling 9.579 s 211.443 ms (2.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent appsec 1.151 s 94.106 ms (8.9%)
Agent iast 1.178 s 120.438 ms (11.4%)
Agent profiling 1.28 s 222.655 ms (21.1%)
Total tracing 9.352 s -
Total appsec 9.461 s 109.493 ms (1.2%)
Total iast 9.635 s 283.124 ms (3.0%)
Total profiling 9.605 s 252.954 ms (2.7%) 
gantt
    title petclinic - break down per module: candidate=1.29.0-SNAPSHOT~9c132668b9, baseline=1.29.0-SNAPSHOT~98e7a4379b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (671.043 ms) : 0, 671043
BytebuddyAgent [candidate] (667.995 ms) : 0, 667995
GlobalTracer [baseline] (298.054 ms) : 0, 298054
GlobalTracer [candidate] (296.334 ms) : 0, 296334
AppSec [baseline] (52.124 ms) : 0, 52124
AppSec [candidate] (50.869 ms) : 0, 50869
Remote Config [baseline] (706.772 µs) : 0, 707
Remote Config [candidate] (657.844 µs) : 0, 658
Telemetry [baseline] (7.641 ms) : 0, 7641
Telemetry [candidate] (7.2 ms) : 0, 7200
section appsec
BytebuddyAgent [baseline] (670.41 ms) : 0, 670410
BytebuddyAgent [candidate] (665.712 ms) : 0, 665712
GlobalTracer [baseline] (297.857 ms) : 0, 297857
GlobalTracer [candidate] (295.569 ms) : 0, 295569
AppSec [baseline] (151.15 ms) : 0, 151150
AppSec [candidate] (148.462 ms) : 0, 148462
Remote Config [baseline] (700.682 µs) : 0, 701
Remote Config [candidate] (646.736 µs) : 0, 647
Telemetry [baseline] (6.844 ms) : 0, 6844
Telemetry [candidate] (6.872 ms) : 0, 6872
section iast
BytebuddyAgent [baseline] (786.761 ms) : 0, 786761
BytebuddyAgent [candidate] (775.983 ms) : 0, 775983
GlobalTracer [baseline] (289.316 ms) : 0, 289316
GlobalTracer [candidate] (286.597 ms) : 0, 286597
AppSec [baseline] (55.99 ms) : 0, 55990
AppSec [candidate] (53.454 ms) : 0, 53454
Remote Config [baseline] (596.585 µs) : 0, 597
Remote Config [candidate] (606.951 µs) : 0, 607
Telemetry [baseline] (6.69 ms) : 0, 6690
Telemetry [candidate] (6.538 ms) : 0, 6538
IAST [baseline] (18.622 ms) : 0, 18622
IAST [candidate] (20.294 ms) : 0, 20294
section profiling
BytebuddyAgent [baseline] (665.023 ms) : 0, 665023
BytebuddyAgent [candidate] (664.979 ms) : 0, 664979
GlobalTracer [baseline] (377.566 ms) : 0, 377566
GlobalTracer [candidate] (377.673 ms) : 0, 377673
AppSec [baseline] (52.261 ms) : 0, 52261
AppSec [candidate] (50.972 ms) : 0, 50972
Remote Config [baseline] (669.844 µs) : 0, 670
Remote Config [candidate] (989.745 µs) : 0, 990
Telemetry [baseline] (7.528 ms) : 0, 7528
Telemetry [candidate] (7.239 ms) : 0, 7239
ProfilingAgent [baseline] (123.729 ms) : 0, 123729
ProfilingAgent [candidate] (123.826 ms) : 0, 123826
Profiling [baseline] (123.754 ms) : 0, 123754
Profiling [candidate] (123.851 ms) : 0, 123851
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.29.0-SNAPSHOT~9c132668b9, baseline=1.29.0-SNAPSHOT~98e7a4379b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054524
Total [baseline] (8.737 s) : 0, 8736702
Agent [candidate] (1.056 s) : 0, 1056319
Total [candidate] (8.751 s) : 0, 8751460
section iast
Agent [baseline] (1.187 s) : 0, 1186613
Total [baseline] (9.303 s) : 0, 9303019
Agent [candidate] (1.189 s) : 0, 1189256
Total [candidate] (9.34 s) : 0, 9339720
section iast_TELEMETRY_OFF
Agent [baseline] (1.168 s) : 0, 1168297
Total [baseline] (9.259 s) : 0, 9258609
Agent [candidate] (1.177 s) : 0, 1177186
Total [candidate] (9.292 s) : 0, 9291687
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.055 s -
Agent iast 1.187 s 132.088 ms (12.5%)
Agent iast_TELEMETRY_OFF 1.168 s 113.773 ms (10.8%)
Total tracing 8.737 s -
Total iast 9.303 s 566.317 ms (6.5%)
Total iast_TELEMETRY_OFF 9.259 s 521.907 ms (6.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent iast 1.189 s 132.936 ms (12.6%)
Agent iast_TELEMETRY_OFF 1.177 s 120.867 ms (11.4%)
Total tracing 8.751 s -
Total iast 9.34 s 588.26 ms (6.7%)
Total iast_TELEMETRY_OFF 9.292 s 540.228 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.29.0-SNAPSHOT~9c132668b9, baseline=1.29.0-SNAPSHOT~98e7a4379b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (664.83 ms) : 0, 664830
BytebuddyAgent [candidate] (666.816 ms) : 0, 666816
GlobalTracer [baseline] (295.388 ms) : 0, 295388
GlobalTracer [candidate] (296.338 ms) : 0, 296338
AppSec [baseline] (51.839 ms) : 0, 51839
AppSec [candidate] (50.977 ms) : 0, 50977
Remote Config [baseline] (709.71 µs) : 0, 710
Remote Config [candidate] (666.947 µs) : 0, 667
Telemetry [baseline] (7.592 ms) : 0, 7592
Telemetry [candidate] (7.22 ms) : 0, 7220
section iast
BytebuddyAgent [baseline] (781.267 ms) : 0, 781267
BytebuddyAgent [candidate] (783.194 ms) : 0, 783194
GlobalTracer [baseline] (288.527 ms) : 0, 288527
GlobalTracer [candidate] (290.225 ms) : 0, 290225
AppSec [baseline] (53.094 ms) : 0, 53094
AppSec [candidate] (56.693 ms) : 0, 56693
IAST [baseline] (20.412 ms) : 0, 20412
IAST [candidate] (17.241 ms) : 0, 17241
Remote Config [baseline] (615.892 µs) : 0, 616
Remote Config [candidate] (600.443 µs) : 0, 600
Telemetry [baseline] (8.137 ms) : 0, 8137
Telemetry [candidate] (6.719 ms) : 0, 6719
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (767.5 ms) : 0, 767500
BytebuddyAgent [candidate] (774.147 ms) : 0, 774147
GlobalTracer [baseline] (285.954 ms) : 0, 285954
GlobalTracer [candidate] (288.49 ms) : 0, 288490
AppSec [baseline] (56.952 ms) : 0, 56952
AppSec [candidate] (52.458 ms) : 0, 52458
IAST [baseline] (16.524 ms) : 0, 16524
IAST [candidate] (20.571 ms) : 0, 20571
Remote Config [baseline] (632.23 µs) : 0, 632
Remote Config [candidate] (631.302 µs) : 0, 631
Telemetry [baseline] (6.377 ms) : 0, 6377
Telemetry [candidate] (6.384 ms) : 0, 6384
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-01-24T11:02:04 2024-01-24T11:18:36
git_branch master paul.laffon/initial-flush-long-running-spans
git_commit_date 1706091852 1706092801
git_commit_sha 98e7a43 9c13266
release_version 1.29.0-SNAPSHOT~98e7a4379b 1.29.0-SNAPSHOT~9c132668b9
start_time 2024-01-24T11:01:51 2024-01-24T11:18:23
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1706095370 1706095370
ci_job_id 417114320 417114320
ci_pipeline_id 27084065 27084065
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.29.0-SNAPSHOT~9c132668b9, baseline=1.29.0-SNAPSHOT~98e7a4379b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.373 ms) : 1354, 1392
.   : milestone, 1373,
appsec (1.802 ms) : 1776, 1827
.   : milestone, 1802,
iast (1.528 ms) : 1503, 1553
.   : milestone, 1528,
profiling (1.55 ms) : 1524, 1576
.   : milestone, 1550,
tracing (1.524 ms) : 1500, 1549
.   : milestone, 1524,
section candidate
no_agent (1.36 ms) : 1341, 1379
.   : milestone, 1360,
appsec (1.792 ms) : 1767, 1817
.   : milestone, 1792,
iast (1.53 ms) : 1506, 1555
.   : milestone, 1530,
profiling (1.525 ms) : 1500, 1551
.   : milestone, 1525,
tracing (1.504 ms) : 1479, 1529
.   : milestone, 1504,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.373 ms [1.354 ms, 1.392 ms] -
appsec 1.802 ms [1.776 ms, 1.827 ms] 428.681 µs (31.2%)
iast 1.528 ms [1.503 ms, 1.553 ms] 155.203 µs (11.3%)
profiling 1.55 ms [1.524 ms, 1.576 ms] 177.162 µs (12.9%)
tracing 1.524 ms [1.5 ms, 1.549 ms] 151.617 µs (11.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.36 ms [1.341 ms, 1.379 ms] -
appsec 1.792 ms [1.767 ms, 1.817 ms] 431.817 µs (31.7%)
iast 1.53 ms [1.506 ms, 1.555 ms] 170.274 µs (12.5%)
profiling 1.525 ms [1.5 ms, 1.551 ms] 165.387 µs (12.2%)
tracing 1.504 ms [1.479 ms, 1.529 ms] 143.543 µs (10.6%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.29.0-SNAPSHOT~9c132668b9, baseline=1.29.0-SNAPSHOT~98e7a4379b
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.902 µs) : 350, 392
.   : milestone, 371,
iast (482.972 µs) : 462, 503
.   : milestone, 483,
iast_FULL (557.25 µs) : 537, 578
.   : milestone, 557,
iast_INACTIVE (452.611 µs) : 432, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (482.86 µs) : 462, 504
.   : milestone, 483,
tracing (453.336 µs) : 432, 474
.   : milestone, 453,
section candidate
no_agent (377.379 µs) : 357, 398
.   : milestone, 377,
iast (486.041 µs) : 465, 507
.   : milestone, 486,
iast_FULL (547.448 µs) : 527, 568
.   : milestone, 547,
iast_INACTIVE (454.462 µs) : 434, 475
.   : milestone, 454,
iast_TELEMETRY_OFF (470.464 µs) : 450, 491
.   : milestone, 470,
tracing (455.906 µs) : 435, 477
.   : milestone, 456,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 370.902 µs [349.787 µs, 392.016 µs] -
iast 482.972 µs [462.453 µs, 503.492 µs] 112.071 µs (30.2%)
iast_FULL 557.25 µs [536.557 µs, 577.943 µs] 186.348 µs (50.2%)
iast_INACTIVE 452.611 µs [431.631 µs, 473.59 µs] 81.709 µs (22.0%)
iast_TELEMETRY_OFF 482.86 µs [462.163 µs, 503.557 µs] 111.958 µs (30.2%)
tracing 453.336 µs [432.388 µs, 474.284 µs] 82.434 µs (22.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 377.379 µs [356.979 µs, 397.779 µs] -
iast 486.041 µs [465.323 µs, 506.76 µs] 108.662 µs (28.8%)
iast_FULL 547.448 µs [526.957 µs, 567.94 µs] 170.069 µs (45.1%)
iast_INACTIVE 454.462 µs [433.672 µs, 475.253 µs] 77.083 µs (20.4%)
iast_TELEMETRY_OFF 470.464 µs [449.991 µs, 490.937 µs] 93.085 µs (24.7%)
tracing 455.906 µs [434.982 µs, 476.83 µs] 78.526 µs (20.8%)

@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/initial-flush-long-running-spans branch 3 times, most recently from 47ecb07 to e7e6d6a Compare January 24, 2024 10:38
@paul-laffon-dd paul-laffon-dd force-pushed the paul.laffon/initial-flush-long-running-spans branch from e7e6d6a to 9c13266 Compare January 24, 2024 10:40
@paul-laffon-dd paul-laffon-dd added the tag: experimental Experimental changes label Jan 25, 2024
@paul-laffon-dd paul-laffon-dd marked this pull request as ready for review January 25, 2024 09:12
@paul-laffon-dd paul-laffon-dd requested a review from a team as a code owner January 25, 2024 09:12
dougqh
dougqh approved these changes Jan 30, 2024
View reviewed changes
Copy link
Contributor

@dougqh dougqh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I recall correctly, this interval is currently only used by data jobs monitoring, so I'm fine with the change.

@paul-laffon-dd paul-laffon-dd merged commit d3e9d55 into master Feb 1, 2024
79 checks passed
@paul-laffon-dd paul-laffon-dd deleted the paul.laffon/initial-flush-long-running-spans branch February 1, 2024 14:20
@github-actions github-actions bot added this to the 1.29.0 milestone Feb 1, 2024
@PerfectSlayer PerfectSlayer added the inst: apache spark Apache Spark instrumentation label Feb 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dougqh dougqh dougqh approved these changes

@am312 am312 Awaiting requested review from am312 am312 is a code owner automatically assigned from DataDog/apm-java

@raphaelgavache raphaelgavache Awaiting requested review from raphaelgavache

Assignees
No one assigned
Labels
inst: apache spark Apache Spark instrumentation tag: experimental Experimental changes
Projects
None yet
Milestone
1.29.0
Development

Successfully merging this pull request may close these issues.
3 participants
@paul-laffon-dd @dougqh @PerfectSlayer