Implement coverage for Cucumber .feature files #6251

nikita-tkachenko-datadog · 2023-11-20T15:07:42Z

What Does This Do

When running instrumented Cucumber scenarios, their corresponding .feature files are added to the list of covered files.

Motivation

Needed for ITR to work in Cucumber projects: if a .feature file is modified, ITR should not be skipping tests that are defined in that file.
Therefore the files should be included in the tests' coverage sets.

Additional Notes

Jira ticket: CIVIS-7993

pr-commenter · 2023-11-20T15:52:34Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/cucumber-feature-coverage
git_commit_date	1700735892	1700744858
git_commit_sha	`48850c7`	`725b2ad`
release_version	1.25.0-SNAPSHOT~48850c78e8	1.25.0-SNAPSHOT~725b2ad92f

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1700747557	1700747557
ci_job_id	375606909	375606909
ci_pipeline_id	23756495	23756495
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.25.0-SNAPSHOT~725b2ad92f, baseline=1.25.0-SNAPSHOT~48850c78e8

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1038065
Total [baseline] (9.326 s) : 0, 9325649
Agent [candidate] (1.04 s) : 0, 1040265
Total [candidate] (9.373 s) : 0, 9372645
section appsec
Agent [baseline] (1.118 s) : 0, 1118267
Total [baseline] (9.409 s) : 0, 9408680
Agent [candidate] (1.122 s) : 0, 1122219
Total [candidate] (9.402 s) : 0, 9402304
section iast
Agent [baseline] (1.16 s) : 0, 1160493
Total [baseline] (9.626 s) : 0, 9625607
Agent [candidate] (1.164 s) : 0, 1163972
Total [candidate] (9.641 s) : 0, 9641280
section profiling
Agent [baseline] (1.229 s) : 0, 1228734
Total [baseline] (9.629 s) : 0, 9628910
Agent [candidate] (1.222 s) : 0, 1221654
Total [candidate] (9.635 s) : 0, 9634614

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.038 s	-
Agent	appsec	1.118 s	80.202 ms (7.7%)
Agent	iast	1.16 s	122.428 ms (11.8%)
Agent	profiling	1.229 s	190.669 ms (18.4%)
Total	tracing	9.326 s	-
Total	appsec	9.409 s	83.032 ms (0.9%)
Total	iast	9.626 s	299.959 ms (3.2%)
Total	profiling	9.629 s	303.262 ms (3.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.04 s	-
Agent	appsec	1.122 s	81.954 ms (7.9%)
Agent	iast	1.164 s	123.707 ms (11.9%)
Agent	profiling	1.222 s	181.388 ms (17.4%)
Total	tracing	9.373 s	-
Total	appsec	9.402 s	29.659 ms (0.3%)
Total	iast	9.641 s	268.635 ms (2.9%)
Total	profiling	9.635 s	261.969 ms (2.8%)

gantt
    title petclinic - break down per module: candidate=1.25.0-SNAPSHOT~725b2ad92f, baseline=1.25.0-SNAPSHOT~48850c78e8

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (647.598 ms) : 0, 647598
BytebuddyAgent [candidate] (648.363 ms) : 0, 648363
GlobalTracer [baseline] (299.309 ms) : 0, 299309
GlobalTracer [candidate] (300.745 ms) : 0, 300745
AppSec [baseline] (48.543 ms) : 0, 48543
AppSec [candidate] (48.617 ms) : 0, 48617
Remote Config [baseline] (692.753 µs) : 0, 693
Remote Config [candidate] (685.635 µs) : 0, 686
Telemetry [baseline] (7.344 ms) : 0, 7344
Telemetry [candidate] (7.26 ms) : 0, 7260
section appsec
BytebuddyAgent [baseline] (643.191 ms) : 0, 643191
BytebuddyAgent [candidate] (645.234 ms) : 0, 645234
GlobalTracer [baseline] (297.012 ms) : 0, 297012
GlobalTracer [candidate] (299.102 ms) : 0, 299102
AppSec [baseline] (136.415 ms) : 0, 136415
AppSec [candidate] (136.232 ms) : 0, 136232
Remote Config [baseline] (651.706 µs) : 0, 652
Remote Config [candidate] (651.288 µs) : 0, 651
Telemetry [baseline] (6.766 ms) : 0, 6766
Telemetry [candidate] (6.737 ms) : 0, 6737
section iast
BytebuddyAgent [baseline] (769.973 ms) : 0, 769973
BytebuddyAgent [candidate] (774.555 ms) : 0, 774555
GlobalTracer [baseline] (280.099 ms) : 0, 280099
GlobalTracer [candidate] (279.605 ms) : 0, 279605
AppSec [baseline] (51.066 ms) : 0, 51066
AppSec [candidate] (51.077 ms) : 0, 51077
Remote Config [baseline] (576.441 µs) : 0, 576
Remote Config [candidate] (572.022 µs) : 0, 572
Telemetry [baseline] (10.029 ms) : 0, 10029
Telemetry [candidate] (9.315 ms) : 0, 9315
IAST [baseline] (14.181 ms) : 0, 14181
IAST [candidate] (14.081 ms) : 0, 14081
section profiling
BytebuddyAgent [baseline] (658.355 ms) : 0, 658355
BytebuddyAgent [candidate] (655.265 ms) : 0, 655265
GlobalTracer [baseline] (370.077 ms) : 0, 370077
GlobalTracer [candidate] (367.699 ms) : 0, 367699
AppSec [baseline] (49.002 ms) : 0, 49002
AppSec [candidate] (48.613 ms) : 0, 48613
Remote Config [baseline] (708.503 µs) : 0, 709
Remote Config [candidate] (705.985 µs) : 0, 706
Telemetry [baseline] (7.496 ms) : 0, 7496
Telemetry [candidate] (7.369 ms) : 0, 7369
ProfilingAgent [baseline] (88.608 ms) : 0, 88608
ProfilingAgent [candidate] (87.905 ms) : 0, 87905
Profiling [baseline] (88.632 ms) : 0, 88632
Profiling [candidate] (87.929 ms) : 0, 87929

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.25.0-SNAPSHOT~725b2ad92f, baseline=1.25.0-SNAPSHOT~48850c78e8

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.032 s) : 0, 1031933
Total [baseline] (8.668 s) : 0, 8668380
Agent [candidate] (1.037 s) : 0, 1036890
Total [candidate] (8.686 s) : 0, 8686075
section iast
Agent [baseline] (1.155 s) : 0, 1155093
Total [baseline] (9.216 s) : 0, 9215929
Agent [candidate] (1.151 s) : 0, 1151267
Total [candidate] (9.24 s) : 0, 9239757
section iast_TELEMETRY_OFF
Agent [baseline] (1.14 s) : 0, 1139716
Total [baseline] (9.207 s) : 0, 9206692
Agent [candidate] (1.142 s) : 0, 1142113
Total [candidate] (9.22 s) : 0, 9219667

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.032 s	-
Agent	iast	1.155 s	123.16 ms (11.9%)
Agent	iast_TELEMETRY_OFF	1.14 s	107.784 ms (10.4%)
Total	tracing	8.668 s	-
Total	iast	9.216 s	547.549 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.207 s	538.312 ms (6.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	iast	1.151 s	114.377 ms (11.0%)
Agent	iast_TELEMETRY_OFF	1.142 s	105.223 ms (10.1%)
Total	tracing	8.686 s	-
Total	iast	9.24 s	553.682 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.22 s	533.592 ms (6.1%)

gantt
    title insecure-bank - break down per module: candidate=1.25.0-SNAPSHOT~725b2ad92f, baseline=1.25.0-SNAPSHOT~48850c78e8

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (643.699 ms) : 0, 643699
BytebuddyAgent [candidate] (646.469 ms) : 0, 646469
GlobalTracer [baseline] (297.908 ms) : 0, 297908
GlobalTracer [candidate] (299.654 ms) : 0, 299654
AppSec [baseline] (48.148 ms) : 0, 48148
AppSec [candidate] (48.504 ms) : 0, 48504
Remote Config [baseline] (683.597 µs) : 0, 684
Remote Config [candidate] (675.392 µs) : 0, 675
Telemetry [baseline] (7.254 ms) : 0, 7254
Telemetry [candidate] (7.296 ms) : 0, 7296
section iast
BytebuddyAgent [baseline] (768.018 ms) : 0, 768018
BytebuddyAgent [candidate] (764.689 ms) : 0, 764689
GlobalTracer [baseline] (279.09 ms) : 0, 279090
GlobalTracer [candidate] (278.706 ms) : 0, 278706
AppSec [baseline] (48.914 ms) : 0, 48914
AppSec [candidate] (47.219 ms) : 0, 47219
Remote Config [baseline] (569.877 µs) : 0, 570
Remote Config [candidate] (583.125 µs) : 0, 583
Telemetry [baseline] (8.635 ms) : 0, 8635
Telemetry [candidate] (9.495 ms) : 0, 9495
IAST [baseline] (15.443 ms) : 0, 15443
IAST [candidate] (16.182 ms) : 0, 16182
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (755.077 ms) : 0, 755077
BytebuddyAgent [candidate] (757.746 ms) : 0, 757746
GlobalTracer [baseline] (278.216 ms) : 0, 278216
GlobalTracer [candidate] (278.542 ms) : 0, 278542
AppSec [baseline] (46.712 ms) : 0, 46712
AppSec [candidate] (46.456 ms) : 0, 46456
Remote Config [baseline] (565.258 µs) : 0, 565
Remote Config [candidate] (571.172 µs) : 0, 571
Telemetry [baseline] (7.025 ms) : 0, 7025
Telemetry [candidate] (7.136 ms) : 0, 7136
IAST [baseline] (17.982 ms) : 0, 17982
IAST [candidate] (17.276 ms) : 0, 17276

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-11-23T13:31:53	2023-11-23T13:48:25
git_branch	master	nikita-tkachenko/cucumber-feature-coverage
git_commit_date	1700735892	1700744858
git_commit_sha	`48850c7`	`725b2ad`
release_version	1.25.0-SNAPSHOT~48850c78e8	1.25.0-SNAPSHOT~725b2ad92f
start_time	2023-11-23T13:31:40	2023-11-23T13:48:12

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1700747557	1700747557
ci_job_id	375606909	375606909
ci_pipeline_id	23756495	23756495
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~725b2ad92f, baseline=1.25.0-SNAPSHOT~48850c78e8
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.335 ms) : 1316, 1353
.   : milestone, 1335,
appsec (1.77 ms) : 1745, 1796
.   : milestone, 1770,
iast (1.519 ms) : 1495, 1544
.   : milestone, 1519,
profiling (1.513 ms) : 1486, 1540
.   : milestone, 1513,
tracing (1.5 ms) : 1475, 1525
.   : milestone, 1500,
section candidate
no_agent (1.337 ms) : 1318, 1356
.   : milestone, 1337,
appsec (1.754 ms) : 1729, 1780
.   : milestone, 1754,
iast (1.534 ms) : 1510, 1558
.   : milestone, 1534,
profiling (1.515 ms) : 1491, 1540
.   : milestone, 1515,
tracing (1.494 ms) : 1469, 1519
.   : milestone, 1494,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.335 ms [1.316 ms, 1.353 ms]	-
appsec	1.77 ms [1.745 ms, 1.796 ms]	435.546 µs (32.6%)
iast	1.519 ms [1.495 ms, 1.544 ms]	184.729 µs (13.8%)
profiling	1.513 ms [1.486 ms, 1.54 ms]	178.502 µs (13.4%)
tracing	1.5 ms [1.475 ms, 1.525 ms]	165.156 µs (12.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.337 ms [1.318 ms, 1.356 ms]	-
appsec	1.754 ms [1.729 ms, 1.78 ms]	417.571 µs (31.2%)
iast	1.534 ms [1.51 ms, 1.558 ms]	197.302 µs (14.8%)
profiling	1.515 ms [1.491 ms, 1.54 ms]	178.595 µs (13.4%)
tracing	1.494 ms [1.469 ms, 1.519 ms]	157.304 µs (11.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~725b2ad92f, baseline=1.25.0-SNAPSHOT~48850c78e8
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.061 µs) : 345, 385
.   : milestone, 365,
iast (467.566 µs) : 447, 488
.   : milestone, 468,
iast_FULL (532.475 µs) : 512, 553
.   : milestone, 532,
iast_INACTIVE (444.447 µs) : 422, 466
.   : milestone, 444,
iast_TELEMETRY_OFF (456.54 µs) : 436, 477
.   : milestone, 457,
tracing (450.546 µs) : 429, 472
.   : milestone, 451,
section candidate
no_agent (366.303 µs) : 347, 386
.   : milestone, 366,
iast (464.395 µs) : 444, 485
.   : milestone, 464,
iast_FULL (527.245 µs) : 507, 548
.   : milestone, 527,
iast_INACTIVE (439.224 µs) : 419, 460
.   : milestone, 439,
iast_TELEMETRY_OFF (470.054 µs) : 449, 491
.   : milestone, 470,
tracing (446.721 µs) : 425, 468
.   : milestone, 447,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	365.061 µs [344.897 µs, 385.224 µs]	-
iast	467.566 µs [447.162 µs, 487.969 µs]	102.505 µs (28.1%)
iast_FULL	532.475 µs [512.292 µs, 552.657 µs]	167.414 µs (45.9%)
iast_INACTIVE	444.447 µs [422.471 µs, 466.423 µs]	79.387 µs (21.7%)
iast_TELEMETRY_OFF	456.54 µs [436.117 µs, 476.962 µs]	91.479 µs (25.1%)
tracing	450.546 µs [429.484 µs, 471.608 µs]	85.486 µs (23.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.303 µs [346.5 µs, 386.105 µs]	-
iast	464.395 µs [443.995 µs, 484.794 µs]	98.092 µs (26.8%)
iast_FULL	527.245 µs [506.817 µs, 547.673 µs]	160.943 µs (43.9%)
iast_INACTIVE	439.224 µs [418.715 µs, 459.733 µs]	72.921 µs (19.9%)
iast_TELEMETRY_OFF	470.054 µs [448.97 µs, 491.139 µs]	103.752 µs (28.3%)
tracing	446.721 µs [425.164 µs, 468.278 µs]	80.418 µs (22.0%)

romainkomorndatadog · 2023-11-24T15:26:31Z

...nt/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/SourceType.java


  private static final SourceType[] UNIVERSE = SourceType.values();

  private final String extension;
+  private final boolean resource;


very nitpicky but I find the mismatch between resource and isResource disturbing.

This is one of the standard naming conventions in Java: if there's a boolean field named something then its getter method is usually named isSomething().

nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Nov 20, 2023

nikita-tkachenko-datadog changed the base branch from master to nikita-tkachenko/fix-conflicting-ci-vis-names November 21, 2023 17:56

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/cucumber-feature-coverage branch from cccd833 to 7b8f591 Compare November 21, 2023 18:00

nikita-tkachenko-datadog changed the title ~~Nikita tkachenko/cucumber feature coverage~~ Implement coverage for Cucumber .feature files Nov 21, 2023

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/cucumber-feature-coverage branch from 764d2fb to 2490794 Compare November 21, 2023 18:27

Base automatically changed from nikita-tkachenko/fix-conflicting-ci-vis-names to master November 22, 2023 10:08

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/cucumber-feature-coverage branch from 0023e61 to 673f888 Compare November 22, 2023 15:03

Implement coverage for Cucumber .feature files

725b2ad

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/cucumber-feature-coverage branch from c69d024 to 725b2ad Compare November 23, 2023 13:08

nikita-tkachenko-datadog marked this pull request as ready for review November 23, 2023 13:46

nikita-tkachenko-datadog requested review from a team as code owners November 23, 2023 13:46

nikita-tkachenko-datadog requested review from romainkomorndatadog, bantonsson and nayeem-kamal November 23, 2023 13:46

Merge branch 'master' into nikita-tkachenko/cucumber-feature-coverage

cc886eb

romainkomorndatadog approved these changes Nov 24, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit c2b3e05 into master Nov 24, 2023
68 of 71 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/cucumber-feature-coverage branch November 24, 2023 15:45

github-actions bot added this to the 1.25.0 milestone Nov 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement coverage for Cucumber .feature files #6251

Implement coverage for Cucumber .feature files #6251

nikita-tkachenko-datadog commented Nov 20, 2023 •

edited

Loading

pr-commenter bot commented Nov 20, 2023 •

edited

Loading

romainkomorndatadog Nov 24, 2023

nikita-tkachenko-datadog Nov 24, 2023

Implement coverage for Cucumber .feature files #6251

Implement coverage for Cucumber .feature files #6251

Conversation

nikita-tkachenko-datadog commented Nov 20, 2023 • edited Loading

What Does This Do

Motivation

Additional Notes

pr-commenter bot commented Nov 20, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

romainkomorndatadog Nov 24, 2023

Choose a reason for hiding this comment

nikita-tkachenko-datadog Nov 24, 2023

Choose a reason for hiding this comment

nikita-tkachenko-datadog commented Nov 20, 2023 •

edited

Loading

pr-commenter bot commented Nov 20, 2023 •

edited

Loading