Fix module name calculation for test executions whose working dir is outside of project repository #6235

nikita-tkachenko-datadog · 2023-11-16T13:28:32Z

What Does This Do

Changes logic that calculates module names for CI Visibility executions that run without build system instrumentation.
A check is added: if current folder of the JVM running the tests is outside of the project's repository, service name is used to populate module name. Otherwise, module name is the path to the current folder relative to the project root (previous behaviour).

Motivation

Bazel, whose instrumentation is not supported, runs tests in temporary folders with randomly generated names.
As the result, with the previous logic module name is randomised.
Since module name is a part of the test fingerprint, the fingerprints change with every run, which breaks certain features (execution history, ITR, etc.).

The motivation to use service name as the module name is to avoid adding another configuration parameter that the customers would have to set.
If the build system is not instrumented, then each individual module must be instrumented manually, which means that a different service name can be manually set for each module.

Jira ticket: CIVIS-8038

…outside of project repo

pr-commenter · 2023-11-16T14:12:04Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.25.0-SNAPSHOT~1b0471a383	1.25.0-SNAPSHOT~36307442fe
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 52 cases.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:insecure-bank:iast:Telemetry	worse [+0.481ms; +5.183ms] or [+7.470%; +80.447%]	9.276ms	6.443ms
scenario:petclinic:iast:Remote Config	worse [+16.066µs; +77.269µs] or [+2.841%; +13.666%]	612.063µs	565.396µs

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.25.0-SNAPSHOT~36307442fe, baseline=1.25.0-SNAPSHOT~1b0471a383

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1035828
Total [baseline] (8.822 s) : 0, 8821900
Agent [candidate] (1.037 s) : 0, 1037368
Total [candidate] (8.798 s) : 0, 8797679
section iast
Agent [baseline] (1.15 s) : 0, 1150008
Total [baseline] (9.324 s) : 0, 9324028
Agent [candidate] (1.159 s) : 0, 1159424
Total [candidate] (9.393 s) : 0, 9392693
section iast_TELEMETRY_OFF
Agent [baseline] (1.142 s) : 0, 1142461
Total [baseline] (9.323 s) : 0, 9323278
Agent [candidate] (1.154 s) : 0, 1153717
Total [candidate] (9.321 s) : 0, 9320849

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.036 s	-
Agent	iast	1.15 s	114.18 ms (11.0%)
Agent	iast_TELEMETRY_OFF	1.142 s	106.633 ms (10.3%)
Total	tracing	8.822 s	-
Total	iast	9.324 s	502.128 ms (5.7%)
Total	iast_TELEMETRY_OFF	9.323 s	501.377 ms (5.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	iast	1.159 s	122.057 ms (11.8%)
Agent	iast_TELEMETRY_OFF	1.154 s	116.35 ms (11.2%)
Total	tracing	8.798 s	-
Total	iast	9.393 s	595.013 ms (6.8%)
Total	iast_TELEMETRY_OFF	9.321 s	523.17 ms (5.9%)

gantt
    title insecure-bank - break down per module: candidate=1.25.0-SNAPSHOT~36307442fe, baseline=1.25.0-SNAPSHOT~1b0471a383

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (648.477 ms) : 0, 648477
BytebuddyAgent [candidate] (648.712 ms) : 0, 648712
GlobalTracer [baseline] (296.117 ms) : 0, 296117
GlobalTracer [candidate] (297.121 ms) : 0, 297121
AppSec [baseline] (48.728 ms) : 0, 48728
AppSec [candidate] (49.029 ms) : 0, 49029
Remote Config [baseline] (678.41 µs) : 0, 678
Remote Config [candidate] (658.199 µs) : 0, 658
Telemetry [baseline] (7.223 ms) : 0, 7223
Telemetry [candidate] (7.205 ms) : 0, 7205
section iast
BytebuddyAgent [baseline] (768.321 ms) : 0, 768321
BytebuddyAgent [candidate] (773.628 ms) : 0, 773628
GlobalTracer [baseline] (275.996 ms) : 0, 275996
GlobalTracer [candidate] (277.829 ms) : 0, 277829
AppSec [baseline] (46.777 ms) : 0, 46777
AppSec [candidate] (46.894 ms) : 0, 46894
IAST [baseline] (17.475 ms) : 0, 17475
IAST [candidate] (16.525 ms) : 0, 16525
Remote Config [baseline] (575.432 µs) : 0, 575
Remote Config [candidate] (585.282 µs) : 0, 585
Telemetry [baseline] (6.443 ms) : 0, 6443
Telemetry [candidate] (9.276 ms) : 0, 9276
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (760.581 ms) : 0, 760581
BytebuddyAgent [candidate] (768.265 ms) : 0, 768265
GlobalTracer [baseline] (275.761 ms) : 0, 275761
GlobalTracer [candidate] (277.985 ms) : 0, 277985
AppSec [baseline] (46.205 ms) : 0, 46205
AppSec [candidate] (47.292 ms) : 0, 47292
IAST [baseline] (15.204 ms) : 0, 15204
IAST [candidate] (16.956 ms) : 0, 16956
Remote Config [baseline] (580.61 µs) : 0, 581
Remote Config [candidate] (592.491 µs) : 0, 592
Telemetry [baseline] (9.737 ms) : 0, 9737
Telemetry [candidate] (7.864 ms) : 0, 7864

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.25.0-SNAPSHOT~36307442fe, baseline=1.25.0-SNAPSHOT~1b0471a383

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1042767
Total [baseline] (9.317 s) : 0, 9317310
Agent [candidate] (1.045 s) : 0, 1044594
Total [candidate] (9.375 s) : 0, 9374765
section appsec
Agent [baseline] (1.124 s) : 0, 1123580
Total [baseline] (9.459 s) : 0, 9458778
Agent [candidate] (1.125 s) : 0, 1124630
Total [candidate] (9.352 s) : 0, 9351502
section iast
Agent [baseline] (1.152 s) : 0, 1151540
Total [baseline] (9.481 s) : 0, 9480953
Agent [candidate] (1.166 s) : 0, 1165533
Total [candidate] (9.62 s) : 0, 9619599
section profiling
Agent [baseline] (1.221 s) : 0, 1221276
Total [baseline] (9.547 s) : 0, 9547125
Agent [candidate] (1.225 s) : 0, 1225449
Total [candidate] (9.575 s) : 0, 9574872

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	appsec	1.124 s	80.813 ms (7.7%)
Agent	iast	1.152 s	108.773 ms (10.4%)
Agent	profiling	1.221 s	178.509 ms (17.1%)
Total	tracing	9.317 s	-
Total	appsec	9.459 s	141.468 ms (1.5%)
Total	iast	9.481 s	163.643 ms (1.8%)
Total	profiling	9.547 s	229.815 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	appsec	1.125 s	80.036 ms (7.7%)
Agent	iast	1.166 s	120.938 ms (11.6%)
Agent	profiling	1.225 s	180.855 ms (17.3%)
Total	tracing	9.375 s	-
Total	appsec	9.352 s	-23.263 ms (-0.2%)
Total	iast	9.62 s	244.834 ms (2.6%)
Total	profiling	9.575 s	200.107 ms (2.1%)

gantt
    title petclinic - break down per module: candidate=1.25.0-SNAPSHOT~36307442fe, baseline=1.25.0-SNAPSHOT~1b0471a383

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (653.014 ms) : 0, 653014
BytebuddyAgent [candidate] (653.761 ms) : 0, 653761
GlobalTracer [baseline] (297.695 ms) : 0, 297695
GlobalTracer [candidate] (298.647 ms) : 0, 298647
AppSec [baseline] (49.285 ms) : 0, 49285
AppSec [candidate] (49.475 ms) : 0, 49475
Remote Config [baseline] (667.142 µs) : 0, 667
Remote Config [candidate] (665.472 µs) : 0, 665
Telemetry [baseline] (7.367 ms) : 0, 7367
Telemetry [candidate] (7.308 ms) : 0, 7308
section appsec
BytebuddyAgent [baseline] (647.692 ms) : 0, 647692
BytebuddyAgent [candidate] (647.908 ms) : 0, 647908
GlobalTracer [baseline] (295.492 ms) : 0, 295492
GlobalTracer [candidate] (296.305 ms) : 0, 296305
AppSec [baseline] (138.543 ms) : 0, 138543
AppSec [candidate] (138.632 ms) : 0, 138632
Remote Config [baseline] (635.529 µs) : 0, 636
Remote Config [candidate] (644.366 µs) : 0, 644
Telemetry [baseline] (6.75 ms) : 0, 6750
Telemetry [candidate] (6.741 ms) : 0, 6741
section iast
BytebuddyAgent [baseline] (767.932 ms) : 0, 767932
BytebuddyAgent [candidate] (778.915 ms) : 0, 778915
GlobalTracer [baseline] (275.236 ms) : 0, 275236
GlobalTracer [candidate] (280.559 ms) : 0, 280559
AppSec [baseline] (46.717 ms) : 0, 46717
AppSec [candidate] (47.619 ms) : 0, 47619
Remote Config [baseline] (565.396 µs) : 0, 565
Remote Config [candidate] (612.063 µs) : 0, 612
Telemetry [baseline] (8.391 ms) : 0, 8391
Telemetry [candidate] (6.766 ms) : 0, 6766
IAST [baseline] (18.253 ms) : 0, 18253
IAST [candidate] (16.127 ms) : 0, 16127
section profiling
BytebuddyAgent [baseline] (659.736 ms) : 0, 659736
BytebuddyAgent [candidate] (663.163 ms) : 0, 663163
GlobalTracer [baseline] (362.078 ms) : 0, 362078
GlobalTracer [candidate] (362.143 ms) : 0, 362143
AppSec [baseline] (49.117 ms) : 0, 49117
AppSec [candidate] (48.788 ms) : 0, 48788
Remote Config [baseline] (690.298 µs) : 0, 690
Remote Config [candidate] (650.198 µs) : 0, 650
Telemetry [baseline] (7.402 ms) : 0, 7402
Telemetry [candidate] (7.48 ms) : 0, 7480
ProfilingAgent [baseline] (87.61 ms) : 0, 87610
ProfilingAgent [candidate] (88.265 ms) : 0, 88265
Profiling [baseline] (87.634 ms) : 0, 87634
Profiling [candidate] (88.288 ms) : 0, 88288

Load

Parameters

	Baseline	Candidate
commit	1.25.0-SNAPSHOT~1b0471a383	1.25.0-SNAPSHOT~36307442fe
config	baseline	candidate
end_time	2023-11-16T13:50:14	2023-11-16T14:06:47
start_time	2023-11-16T13:50:01	2023-11-16T14:06:34

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 22 cases.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~36307442fe, baseline=1.25.0-SNAPSHOT~1b0471a383
    dateFormat X
    axisFormat %s
section baseline
no_agent (375.382 µs) : 355, 395
.   : milestone, 375,
iast (469.851 µs) : 449, 491
.   : milestone, 470,
iast_FULL (529.629 µs) : 509, 550
.   : milestone, 530,
iast_INACTIVE (441.201 µs) : 420, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (470.239 µs) : 450, 491
.   : milestone, 470,
tracing (438.344 µs) : 417, 459
.   : milestone, 438,
section candidate
no_agent (364.972 µs) : 344, 386
.   : milestone, 365,
iast (470.857 µs) : 450, 492
.   : milestone, 471,
iast_FULL (527.56 µs) : 507, 548
.   : milestone, 528,
iast_INACTIVE (444.011 µs) : 422, 466
.   : milestone, 444,
iast_TELEMETRY_OFF (472.424 µs) : 451, 493
.   : milestone, 472,
tracing (452.491 µs) : 431, 474
.   : milestone, 452,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	375.382 µs [355.424 µs, 395.341 µs]	-
iast	469.851 µs [448.665 µs, 491.037 µs]	94.469 µs (25.2%)
iast_FULL	529.629 µs [508.958 µs, 550.299 µs]	154.246 µs (41.1%)
iast_INACTIVE	441.201 µs [420.252 µs, 462.151 µs]	65.819 µs (17.5%)
iast_TELEMETRY_OFF	470.239 µs [449.567 µs, 490.911 µs]	94.857 µs (25.3%)
tracing	438.344 µs [417.313 µs, 459.375 µs]	62.961 µs (16.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.972 µs [344.398 µs, 385.546 µs]	-
iast	470.857 µs [450.13 µs, 491.584 µs]	105.885 µs (29.0%)
iast_FULL	527.56 µs [506.963 µs, 548.157 µs]	162.588 µs (44.5%)
iast_INACTIVE	444.011 µs [421.634 µs, 466.388 µs]	79.039 µs (21.7%)
iast_TELEMETRY_OFF	472.424 µs [451.362 µs, 493.486 µs]	107.452 µs (29.4%)
tracing	452.491 µs [430.775 µs, 474.207 µs]	87.519 µs (24.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~36307442fe, baseline=1.25.0-SNAPSHOT~1b0471a383
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.364 ms) : 1345, 1384
.   : milestone, 1364,
appsec (1.725 ms) : 1700, 1750
.   : milestone, 1725,
iast (1.458 ms) : 1433, 1482
.   : milestone, 1458,
profiling (1.516 ms) : 1489, 1542
.   : milestone, 1516,
tracing (1.477 ms) : 1452, 1501
.   : milestone, 1477,
section candidate
no_agent (1.347 ms) : 1328, 1367
.   : milestone, 1347,
appsec (1.76 ms) : 1734, 1785
.   : milestone, 1760,
iast (1.492 ms) : 1468, 1516
.   : milestone, 1492,
profiling (1.483 ms) : 1458, 1507
.   : milestone, 1483,
tracing (1.464 ms) : 1439, 1489
.   : milestone, 1464,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.364 ms [1.345 ms, 1.384 ms]	-
appsec	1.725 ms [1.7 ms, 1.75 ms]	361.068 µs (26.5%)
iast	1.458 ms [1.433 ms, 1.482 ms]	93.325 µs (6.8%)
profiling	1.516 ms [1.489 ms, 1.542 ms]	151.179 µs (11.1%)
tracing	1.477 ms [1.452 ms, 1.501 ms]	112.545 µs (8.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.347 ms [1.328 ms, 1.367 ms]	-
appsec	1.76 ms [1.734 ms, 1.785 ms]	412.251 µs (30.6%)
iast	1.492 ms [1.468 ms, 1.516 ms]	144.422 µs (10.7%)
profiling	1.483 ms [1.458 ms, 1.507 ms]	135.507 µs (10.1%)
tracing	1.464 ms [1.439 ms, 1.489 ms]	116.655 µs (8.7%)

Fix module name calculation for test executions whose working dir is …

3630744

…outside of project repo

nikita-tkachenko-datadog added type: bug comp: ci visibility Continuous Integration Visibility labels Nov 16, 2023

nikita-tkachenko-datadog marked this pull request as ready for review November 16, 2023 13:57

nikita-tkachenko-datadog requested a review from a team as a code owner November 16, 2023 13:57

nikita-tkachenko-datadog requested a review from tonyredondo November 16, 2023 13:57

romainkomorndatadog approved these changes Nov 16, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit 6067ac3 into release/v1.24.x Nov 16, 2023
62 of 64 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/bazel-module-name-fix-cherry-pick branch November 16, 2023 14:17

nikita-tkachenko-datadog added this to the 1.24.1 milestone Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix module name calculation for test executions whose working dir is outside of project repository #6235

Fix module name calculation for test executions whose working dir is outside of project repository #6235

nikita-tkachenko-datadog commented Nov 16, 2023 •

edited by jira bot

Loading

pr-commenter bot commented Nov 16, 2023

Fix module name calculation for test executions whose working dir is outside of project repository #6235

Fix module name calculation for test executions whose working dir is outside of project repository #6235

Conversation

nikita-tkachenko-datadog commented Nov 16, 2023 • edited by jira bot Loading

What Does This Do

Motivation

pr-commenter bot commented Nov 16, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

nikita-tkachenko-datadog commented Nov 16, 2023 •

edited by jira bot

Loading