Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Scalatest support #6331

Merged
merged 3 commits into from
Dec 13, 2023
Merged

Implement Scalatest support #6331

merged 3 commits into from
Dec 13, 2023

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Dec 7, 2023 •
edited by jira bot
Loading

What Does This Do

Adds Scalatest framework support to CI Visibility.

Motivation

There has been a request to support it from one of Datadog's internal projects.

Additional Notes

Support is implemented both for tests tracing and for Intelligent Test Runner.

Jira ticket: CIVIS-8238

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 7, 2023
@pr-commenter
Copy link

pr-commenter bot commented Dec 7, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/scalatest-support
git_commit_date 1702461697 1702463737
git_commit_sha e26f1a4 ca864c5
release_version 1.26.0-SNAPSHOT~e26f1a4abf 1.26.0-SNAPSHOT~ca864c5a9a
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1702466540 1702466540
ci_job_id 388654437 388654437
ci_pipeline_id 25007241 25007241
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 8 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~ca864c5a9a, baseline=1.26.0-SNAPSHOT~e26f1a4abf

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1042603
Total [baseline] (9.331 s) : 0, 9330626
Agent [candidate] (1.062 s) : 0, 1062027
Total [candidate] (9.358 s) : 0, 9358129
section appsec
Agent [baseline] (1.133 s) : 0, 1132773
Total [baseline] (9.436 s) : 0, 9436307
Agent [candidate] (1.13 s) : 0, 1129709
Total [candidate] (9.411 s) : 0, 9411375
section iast
Agent [baseline] (1.166 s) : 0, 1166203
Total [baseline] (9.599 s) : 0, 9599488
Agent [candidate] (1.161 s) : 0, 1161072
Total [candidate] (9.566 s) : 0, 9566475
section profiling
Agent [baseline] (1.232 s) : 0, 1232205
Total [baseline] (9.637 s) : 0, 9637497
Agent [candidate] (1.236 s) : 0, 1235621
Total [candidate] (9.644 s) : 0, 9643664
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.043 s -
Agent appsec 1.133 s 90.171 ms (8.6%)
Agent iast 1.166 s 123.6 ms (11.9%)
Agent profiling 1.232 s 189.603 ms (18.2%)
Total tracing 9.331 s -
Total appsec 9.436 s 105.68 ms (1.1%)
Total iast 9.599 s 268.862 ms (2.9%)
Total profiling 9.637 s 306.87 ms (3.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent appsec 1.13 s 67.681 ms (6.4%)
Agent iast 1.161 s 99.044 ms (9.3%)
Agent profiling 1.236 s 173.594 ms (16.3%)
Total tracing 9.358 s -
Total appsec 9.411 s 53.245 ms (0.6%)
Total iast 9.566 s 208.346 ms (2.2%)
Total profiling 9.644 s 285.535 ms (3.1%) 
gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~ca864c5a9a, baseline=1.26.0-SNAPSHOT~e26f1a4abf

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (645.116 ms) : 0, 645116
BytebuddyAgent [candidate] (657.217 ms) : 0, 657217
GlobalTracer [baseline] (306.647 ms) : 0, 306647
GlobalTracer [candidate] (312.551 ms) : 0, 312551
AppSec [baseline] (48.751 ms) : 0, 48751
AppSec [candidate] (49.417 ms) : 0, 49417
Remote Config [baseline] (668.886 µs) : 0, 669
Remote Config [candidate] (681.113 µs) : 0, 681
Telemetry [baseline] (7.116 ms) : 0, 7116
Telemetry [candidate] (7.253 ms) : 0, 7253
section appsec
BytebuddyAgent [baseline] (646.643 ms) : 0, 646643
BytebuddyAgent [candidate] (645.103 ms) : 0, 645103
GlobalTracer [baseline] (307.521 ms) : 0, 307521
GlobalTracer [candidate] (306.484 ms) : 0, 306484
AppSec [baseline] (136.738 ms) : 0, 136738
AppSec [candidate] (136.508 ms) : 0, 136508
Remote Config [baseline] (642.28 µs) : 0, 642
Remote Config [candidate] (645.745 µs) : 0, 646
Telemetry [baseline] (6.843 ms) : 0, 6843
Telemetry [candidate] (6.846 ms) : 0, 6846
section iast
BytebuddyAgent [baseline] (769.614 ms) : 0, 769614
BytebuddyAgent [candidate] (766.071 ms) : 0, 766071
GlobalTracer [baseline] (286.439 ms) : 0, 286439
GlobalTracer [candidate] (285.606 ms) : 0, 285606
AppSec [baseline] (46.964 ms) : 0, 46964
AppSec [candidate] (46.817 ms) : 0, 46817
Remote Config [baseline] (622.699 µs) : 0, 623
Remote Config [candidate] (640.106 µs) : 0, 640
Telemetry [baseline] (7.325 ms) : 0, 7325
Telemetry [candidate] (8.118 ms) : 0, 8118
IAST [baseline] (21.031 ms) : 0, 21031
IAST [candidate] (19.695 ms) : 0, 19695
section profiling
BytebuddyAgent [baseline] (655.469 ms) : 0, 655469
BytebuddyAgent [candidate] (658.376 ms) : 0, 658376
GlobalTracer [baseline] (377.245 ms) : 0, 377245
GlobalTracer [candidate] (378.314 ms) : 0, 378314
AppSec [baseline] (48.708 ms) : 0, 48708
AppSec [candidate] (48.664 ms) : 0, 48664
Remote Config [baseline] (653.135 µs) : 0, 653
Remote Config [candidate] (667.64 µs) : 0, 668
Telemetry [baseline] (7.377 ms) : 0, 7377
Telemetry [candidate] (7.441 ms) : 0, 7441
ProfilingAgent [baseline] (88.638 ms) : 0, 88638
ProfilingAgent [candidate] (88.038 ms) : 0, 88038
Profiling [baseline] (88.663 ms) : 0, 88663
Profiling [candidate] (88.063 ms) : 0, 88063
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~ca864c5a9a, baseline=1.26.0-SNAPSHOT~e26f1a4abf

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.052 s) : 0, 1051667
Total [baseline] (8.726 s) : 0, 8726493
Agent [candidate] (1.046 s) : 0, 1045557
Total [candidate] (8.738 s) : 0, 8738276
section iast
Agent [baseline] (1.165 s) : 0, 1164632
Total [baseline] (9.255 s) : 0, 9254781
Agent [candidate] (1.169 s) : 0, 1168827
Total [candidate] (9.311 s) : 0, 9310913
section iast_TELEMETRY_OFF
Agent [baseline] (1.161 s) : 0, 1160909
Total [baseline] (9.272 s) : 0, 9271870
Agent [candidate] (1.164 s) : 0, 1163846
Total [candidate] (9.316 s) : 0, 9316028
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.052 s -
Agent iast 1.165 s 112.965 ms (10.7%)
Agent iast_TELEMETRY_OFF 1.161 s 109.242 ms (10.4%)
Total tracing 8.726 s -
Total iast 9.255 s 528.288 ms (6.1%)
Total iast_TELEMETRY_OFF 9.272 s 545.378 ms (6.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.046 s -
Agent iast 1.169 s 123.269 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.164 s 118.288 ms (11.3%)
Total tracing 8.738 s -
Total iast 9.311 s 572.637 ms (6.6%)
Total iast_TELEMETRY_OFF 9.316 s 577.753 ms (6.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~ca864c5a9a, baseline=1.26.0-SNAPSHOT~e26f1a4abf

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (651.68 ms) : 0, 651680
BytebuddyAgent [candidate] (647.221 ms) : 0, 647221
GlobalTracer [baseline] (308.805 ms) : 0, 308805
GlobalTracer [candidate] (307.491 ms) : 0, 307491
AppSec [baseline] (48.606 ms) : 0, 48606
AppSec [candidate] (48.729 ms) : 0, 48729
Remote Config [baseline] (685.029 µs) : 0, 685
Remote Config [candidate] (675.271 µs) : 0, 675
Telemetry [baseline] (7.222 ms) : 0, 7222
Telemetry [candidate] (7.123 ms) : 0, 7123
section iast
BytebuddyAgent [baseline] (769.095 ms) : 0, 769095
BytebuddyAgent [candidate] (771.925 ms) : 0, 771925
GlobalTracer [baseline] (285.278 ms) : 0, 285278
GlobalTracer [candidate] (287.175 ms) : 0, 287175
AppSec [baseline] (47.004 ms) : 0, 47004
AppSec [candidate] (46.952 ms) : 0, 46952
Remote Config [baseline] (608.695 µs) : 0, 609
Remote Config [candidate] (612.251 µs) : 0, 612
Telemetry [baseline] (7.993 ms) : 0, 7993
Telemetry [candidate] (6.591 ms) : 0, 6591
IAST [baseline] (20.246 ms) : 0, 20246
IAST [candidate] (21.064 ms) : 0, 21064
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (764.656 ms) : 0, 764656
BytebuddyAgent [candidate] (766.857 ms) : 0, 766857
GlobalTracer [baseline] (287.292 ms) : 0, 287292
GlobalTracer [candidate] (288.789 ms) : 0, 288789
AppSec [baseline] (48.105 ms) : 0, 48105
AppSec [candidate] (46.316 ms) : 0, 46316
Remote Config [baseline] (595.807 µs) : 0, 596
Remote Config [candidate] (608.271 µs) : 0, 608
Telemetry [baseline] (7.963 ms) : 0, 7963
Telemetry [candidate] (8.058 ms) : 0, 8058
IAST [baseline] (17.736 ms) : 0, 17736
IAST [candidate] (18.624 ms) : 0, 18624
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2023-12-13T11:01:33 2023-12-13T11:18:07
git_branch master nikita-tkachenko/scalatest-support
git_commit_date 1702461697 1702463737
git_commit_sha e26f1a4 ca864c5
release_version 1.26.0-SNAPSHOT~e26f1a4abf 1.26.0-SNAPSHOT~ca864c5a9a
start_time 2023-12-13T11:01:20 2023-12-13T11:17:54
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1702466540 1702466540
ci_job_id 388654437 388654437
ci_pipeline_id 25007241 25007241
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 13 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~ca864c5a9a, baseline=1.26.0-SNAPSHOT~e26f1a4abf
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.354 ms) : 1335, 1374
.   : milestone, 1354,
appsec (1.774 ms) : 1748, 1799
.   : milestone, 1774,
iast (1.523 ms) : 1498, 1547
.   : milestone, 1523,
profiling (1.528 ms) : 1502, 1553
.   : milestone, 1528,
tracing (1.501 ms) : 1476, 1525
.   : milestone, 1501,
section candidate
no_agent (1.356 ms) : 1337, 1375
.   : milestone, 1356,
appsec (1.754 ms) : 1729, 1779
.   : milestone, 1754,
iast (1.525 ms) : 1501, 1550
.   : milestone, 1525,
profiling (1.513 ms) : 1488, 1538
.   : milestone, 1513,
tracing (1.502 ms) : 1478, 1527
.   : milestone, 1502,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.354 ms [1.335 ms, 1.374 ms] -
appsec 1.774 ms [1.748 ms, 1.799 ms] 419.058 µs (30.9%)
iast 1.523 ms [1.498 ms, 1.547 ms] 168.031 µs (12.4%)
profiling 1.528 ms [1.502 ms, 1.553 ms] 173.074 µs (12.8%)
tracing 1.501 ms [1.476 ms, 1.525 ms] 146.085 µs (10.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.356 ms [1.337 ms, 1.375 ms] -
appsec 1.754 ms [1.729 ms, 1.779 ms] 398.208 µs (29.4%)
iast 1.525 ms [1.501 ms, 1.55 ms] 169.474 µs (12.5%)
profiling 1.513 ms [1.488 ms, 1.538 ms] 157.287 µs (11.6%)
tracing 1.502 ms [1.478 ms, 1.527 ms] 146.177 µs (10.8%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~ca864c5a9a, baseline=1.26.0-SNAPSHOT~e26f1a4abf
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.379 µs) : 346, 385
.   : milestone, 365,
iast (477.666 µs) : 457, 498
.   : milestone, 478,
iast_FULL (539.25 µs) : 519, 560
.   : milestone, 539,
iast_INACTIVE (446.387 µs) : 425, 467
.   : milestone, 446,
iast_TELEMETRY_OFF (469.62 µs) : 449, 490
.   : milestone, 470,
tracing (439.931 µs) : 419, 461
.   : milestone, 440,
section candidate
no_agent (361.865 µs) : 342, 381
.   : milestone, 362,
iast (471.896 µs) : 451, 492
.   : milestone, 472,
iast_FULL (537.755 µs) : 517, 558
.   : milestone, 538,
iast_INACTIVE (440.762 µs) : 420, 461
.   : milestone, 441,
iast_TELEMETRY_OFF (464.454 µs) : 444, 485
.   : milestone, 464,
tracing (447.299 µs) : 426, 468
.   : milestone, 447,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.379 µs [345.63 µs, 385.128 µs] -
iast 477.666 µs [456.884 µs, 498.448 µs] 112.287 µs (30.7%)
iast_FULL 539.25 µs [518.724 µs, 559.776 µs] 173.871 µs (47.6%)
iast_INACTIVE 446.387 µs [425.4 µs, 467.374 µs] 81.008 µs (22.2%)
iast_TELEMETRY_OFF 469.62 µs [448.789 µs, 490.45 µs] 104.241 µs (28.5%)
tracing 439.931 µs [419.175 µs, 460.687 µs] 74.552 µs (20.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 361.865 µs [342.236 µs, 381.493 µs] -
iast 471.896 µs [451.378 µs, 492.415 µs] 110.032 µs (30.4%)
iast_FULL 537.755 µs [517.343 µs, 558.167 µs] 175.89 µs (48.6%)
iast_INACTIVE 440.762 µs [420.498 µs, 461.025 µs] 78.897 µs (21.8%)
iast_TELEMETRY_OFF 464.454 µs [443.849 µs, 485.06 µs] 102.59 µs (28.4%)
tracing 447.299 µs [426.114 µs, 468.485 µs] 85.435 µs (23.6%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review December 9, 2023 19:11
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as draft December 12, 2023 09:56
@nikita-tkachenko-datadog nikita-tkachenko-datadog changed the base branch from master to nikita-tkachenko/test-module-name-fix December 12, 2023 20:00
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review December 12, 2023 22:53
Base automatically changed from nikita-tkachenko/test-module-name-fix to master December 13, 2023 10:01
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit 6e9eadb into master Dec 13, 2023
72 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/scalatest-support branch December 13, 2023 11:58
@github-actions github-actions bot added this to the 1.26.0 milestone Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes

@tonyredondo tonyredondo Awaiting requested review from tonyredondo tonyredondo is a code owner automatically assigned from DataDog/ci-app-libraries-java

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer PerfectSlayer is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility
Projects
None yet
Milestone
1.26.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@nikita-tkachenko-datadog @anmarchenko