Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix handling skipped features in Karate #6870

Merged
merged 1 commit into from Apr 2, 2024
Merged

Fix handling skipped features in Karate #6870

merged 1 commit into from Apr 2, 2024

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Apr 2, 2024 •
edited

What Does This Do

Fixes handling of skipped features in Karate testing framework instrumentation.

Additional Notes

Normally Karate calls beforeFeature and afterFeature callbacks of the listeners that registered themselves with the framework.
For skipped features the afterFeature callback is not called.
This resulted in an error in the tracer code.

Jira ticket: CIVIS-9570

@nikita-tkachenko-datadog nikita-tkachenko-datadog added type: bug comp: ci visibility Continuous Integration Visibility labels Apr 2, 2024
@pr-commenter
Copy link

pr-commenter bot commented Apr 2, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/karate-skipped-feature-fix
git_commit_date 1712064681 1712077959
git_commit_sha 329acac a638e84
release_version 1.32.0-SNAPSHOT~329acac1da 1.32.0-SNAPSHOT~a638e84cb3
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1712081426 1712081426
ci_job_id 475770589 475770589
ci_pipeline_id 31286135 31286135
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 13 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~a638e84cb3, baseline=1.32.0-SNAPSHOT~329acac1da

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.086 s) : 0, 1085752
Total [baseline] (10.429 s) : 0, 10429444
Agent [candidate] (1.076 s) : 0, 1075806
Total [candidate] (10.427 s) : 0, 10426663
section appsec
Agent [baseline] (1.2 s) : 0, 1200177
Total [baseline] (10.592 s) : 0, 10591774
Agent [candidate] (1.195 s) : 0, 1195407
Total [candidate] (10.465 s) : 0, 10465161
section iast
Agent [baseline] (1.198 s) : 0, 1197638
Total [baseline] (10.873 s) : 0, 10873361
Agent [candidate] (1.199 s) : 0, 1199306
Total [candidate] (10.82 s) : 0, 10819732
section profiling
Agent [baseline] (1.267 s) : 0, 1266513
Total [baseline] (10.602 s) : 0, 10601699
Agent [candidate] (1.277 s) : 0, 1277013
Total [candidate] (10.654 s) : 0, 10654178
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.086 s -
Agent appsec 1.2 s 114.425 ms (10.5%)
Agent iast 1.198 s 111.886 ms (10.3%)
Agent profiling 1.267 s 180.762 ms (16.6%)
Total tracing 10.429 s -
Total appsec 10.592 s 162.331 ms (1.6%)
Total iast 10.873 s 443.917 ms (4.3%)
Total profiling 10.602 s 172.256 ms (1.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.076 s -
Agent appsec 1.195 s 119.601 ms (11.1%)
Agent iast 1.199 s 123.5 ms (11.5%)
Agent profiling 1.277 s 201.207 ms (18.7%)
Total tracing 10.427 s -
Total appsec 10.465 s 38.498 ms (0.4%)
Total iast 10.82 s 393.069 ms (3.8%)
Total profiling 10.654 s 227.515 ms (2.2%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~a638e84cb3, baseline=1.32.0-SNAPSHOT~329acac1da

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (679.273 ms) : 0, 679273
BytebuddyAgent [candidate] (672.975 ms) : 0, 672975
GlobalTracer [baseline] (313.11 ms) : 0, 313110
GlobalTracer [candidate] (310.481 ms) : 0, 310481
AppSec [baseline] (50.307 ms) : 0, 50307
AppSec [candidate] (49.728 ms) : 0, 49728
Remote Config [baseline] (673.863 µs) : 0, 674
Remote Config [candidate] (654.872 µs) : 0, 655
Telemetry [baseline] (7.706 ms) : 0, 7706
Telemetry [candidate] (7.544 ms) : 0, 7544
section appsec
BytebuddyAgent [baseline] (697.476 ms) : 0, 697476
BytebuddyAgent [candidate] (693.748 ms) : 0, 693748
GlobalTracer [baseline] (292.313 ms) : 0, 292313
GlobalTracer [candidate] (291.659 ms) : 0, 291659
AppSec [baseline] (149.581 ms) : 0, 149581
AppSec [candidate] (149.485 ms) : 0, 149485
Remote Config [baseline] (606.879 µs) : 0, 607
Remote Config [candidate] (603.384 µs) : 0, 603
Telemetry [baseline] (6.797 ms) : 0, 6797
Telemetry [candidate] (6.765 ms) : 0, 6765
IAST [baseline] (18.852 ms) : 0, 18852
IAST [candidate] (18.771 ms) : 0, 18771
section iast
BytebuddyAgent [baseline] (793.537 ms) : 0, 793537
BytebuddyAgent [candidate] (794.445 ms) : 0, 794445
GlobalTracer [baseline] (288.376 ms) : 0, 288376
GlobalTracer [candidate] (288.846 ms) : 0, 288846
AppSec [baseline] (51.598 ms) : 0, 51598
AppSec [candidate] (51.596 ms) : 0, 51596
Remote Config [baseline] (576.38 µs) : 0, 576
Remote Config [candidate] (586.907 µs) : 0, 587
Telemetry [baseline] (6.607 ms) : 0, 6607
Telemetry [candidate] (6.567 ms) : 0, 6567
IAST [baseline] (22.66 ms) : 0, 22660
IAST [candidate] (22.943 ms) : 0, 22943
section profiling
BytebuddyAgent [baseline] (676.674 ms) : 0, 676674
BytebuddyAgent [candidate] (682.531 ms) : 0, 682531
GlobalTracer [baseline] (379.912 ms) : 0, 379912
GlobalTracer [candidate] (383.682 ms) : 0, 383682
AppSec [baseline] (50.106 ms) : 0, 50106
AppSec [candidate] (50.264 ms) : 0, 50264
Remote Config [baseline] (698.019 µs) : 0, 698
Remote Config [candidate] (708.203 µs) : 0, 708
Telemetry [baseline] (7.507 ms) : 0, 7507
Telemetry [candidate] (7.564 ms) : 0, 7564
ProfilingAgent [baseline] (95.486 ms) : 0, 95486
ProfilingAgent [candidate] (95.531 ms) : 0, 95531
Profiling [baseline] (95.51 ms) : 0, 95510
Profiling [candidate] (95.556 ms) : 0, 95556
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~a638e84cb3, baseline=1.32.0-SNAPSHOT~329acac1da

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.071 s) : 0, 1071117
Total [baseline] (8.538 s) : 0, 8538436
Agent [candidate] (1.092 s) : 0, 1091548
Total [candidate] (8.626 s) : 0, 8625616
section iast
Agent [baseline] (1.198 s) : 0, 1198356
Total [baseline] (9.114 s) : 0, 9114223
Agent [candidate] (1.198 s) : 0, 1197926
Total [candidate] (9.065 s) : 0, 9064663
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.209 s) : 0, 1208706
Total [baseline] (9.097 s) : 0, 9096814
Agent [candidate] (1.213 s) : 0, 1212574
Total [candidate] (9.098 s) : 0, 9098148
section iast_TELEMETRY_OFF
Agent [baseline] (1.21 s) : 0, 1210402
Total [baseline] (9.164 s) : 0, 9163935
Agent [candidate] (1.197 s) : 0, 1197135
Total [candidate] (9.127 s) : 0, 9126511
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.071 s -
Agent iast 1.198 s 127.239 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.209 s 137.588 ms (12.8%)
Agent iast_TELEMETRY_OFF 1.21 s 139.285 ms (13.0%)
Total tracing 8.538 s -
Total iast 9.114 s 575.787 ms (6.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.097 s 558.378 ms (6.5%)
Total iast_TELEMETRY_OFF 9.164 s 625.498 ms (7.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.092 s -
Agent iast 1.198 s 106.378 ms (9.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.213 s 121.025 ms (11.1%)
Agent iast_TELEMETRY_OFF 1.197 s 105.587 ms (9.7%)
Total tracing 8.626 s -
Total iast 9.065 s 439.047 ms (5.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.098 s 472.532 ms (5.5%)
Total iast_TELEMETRY_OFF 9.127 s 500.895 ms (5.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~a638e84cb3, baseline=1.32.0-SNAPSHOT~329acac1da

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (670.821 ms) : 0, 670821
BytebuddyAgent [candidate] (682.866 ms) : 0, 682866
GlobalTracer [baseline] (308.42 ms) : 0, 308420
GlobalTracer [candidate] (314.782 ms) : 0, 314782
AppSec [baseline] (49.323 ms) : 0, 49323
AppSec [candidate] (50.425 ms) : 0, 50425
Remote Config [baseline] (657.235 µs) : 0, 657
Remote Config [candidate] (686.529 µs) : 0, 687
Telemetry [baseline] (7.547 ms) : 0, 7547
Telemetry [candidate] (7.776 ms) : 0, 7776
section iast
BytebuddyAgent [baseline] (794.888 ms) : 0, 794888
BytebuddyAgent [candidate] (793.294 ms) : 0, 793294
GlobalTracer [baseline] (287.851 ms) : 0, 287851
GlobalTracer [candidate] (288.69 ms) : 0, 288690
AppSec [baseline] (49.614 ms) : 0, 49614
AppSec [candidate] (51.0 ms) : 0, 51000
IAST [baseline] (22.869 ms) : 0, 22869
IAST [candidate] (22.006 ms) : 0, 22006
Remote Config [baseline] (584.134 µs) : 0, 584
Remote Config [candidate] (593.271 µs) : 0, 593
Telemetry [baseline] (8.09 ms) : 0, 8090
Telemetry [candidate] (8.088 ms) : 0, 8088
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (799.799 ms) : 0, 799799
BytebuddyAgent [candidate] (803.534 ms) : 0, 803534
GlobalTracer [baseline] (291.506 ms) : 0, 291506
GlobalTracer [candidate] (291.266 ms) : 0, 291266
AppSec [baseline] (49.119 ms) : 0, 49119
AppSec [candidate] (48.541 ms) : 0, 48541
IAST [baseline] (25.622 ms) : 0, 25622
IAST [candidate] (25.564 ms) : 0, 25564
Remote Config [baseline] (597.964 µs) : 0, 598
Remote Config [candidate] (589.399 µs) : 0, 589
Telemetry [baseline] (7.447 ms) : 0, 7447
Telemetry [candidate] (8.133 ms) : 0, 8133
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (801.419 ms) : 0, 801419
BytebuddyAgent [candidate] (792.548 ms) : 0, 792548
GlobalTracer [baseline] (291.486 ms) : 0, 291486
GlobalTracer [candidate] (289.026 ms) : 0, 289026
AppSec [baseline] (51.708 ms) : 0, 51708
AppSec [candidate] (51.006 ms) : 0, 51006
IAST [baseline] (22.086 ms) : 0, 22086
IAST [candidate] (23.084 ms) : 0, 23084
Remote Config [baseline] (601.043 µs) : 0, 601
Remote Config [candidate] (572.035 µs) : 0, 572
Telemetry [baseline] (8.278 ms) : 0, 8278
Telemetry [candidate] (6.577 ms) : 0, 6577

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-02T17:42:37 2024-04-02T18:04:39
git_branch master nikita-tkachenko/karate-skipped-feature-fix
git_commit_date 1712064681 1712077959
git_commit_sha 329acac a638e84
release_version 1.32.0-SNAPSHOT~329acac1da 1.32.0-SNAPSHOT~a638e84cb3
start_time 2024-04-02T17:42:24 2024-04-02T18:04:26
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1712081426 1712081426
ci_job_id 475770589 475770589
ci_pipeline_id 31286135 31286135
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 11 metrics, 16 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling worse
[+30.606µs; +84.515µs] or [+2.052%; +5.667%]		 unstable
[-562.595op/s; +562.595op/s] or [-18.284%; +18.284%]		 1.549ms 3076.923op/s 1.491ms 3076.923op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~a638e84cb3, baseline=1.32.0-SNAPSHOT~329acac1da
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.345 ms) : 1326, 1364
.   : milestone, 1345,
appsec (1.718 ms) : 1693, 1743
.   : milestone, 1718,
appsec_no_iast (1.747 ms) : 1723, 1771
.   : milestone, 1747,
iast (1.478 ms) : 1455, 1500
.   : milestone, 1478,
profiling (1.491 ms) : 1467, 1516
.   : milestone, 1491,
tracing (1.462 ms) : 1438, 1487
.   : milestone, 1462,
section candidate
no_agent (1.343 ms) : 1324, 1363
.   : milestone, 1343,
appsec (1.709 ms) : 1685, 1734
.   : milestone, 1709,
appsec_no_iast (1.74 ms) : 1716, 1763
.   : milestone, 1740,
iast (1.506 ms) : 1483, 1530
.   : milestone, 1506,
profiling (1.549 ms) : 1524, 1574
.   : milestone, 1549,
tracing (1.48 ms) : 1456, 1504
.   : milestone, 1480,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.345 ms [1.326 ms, 1.364 ms] -
appsec 1.718 ms [1.693 ms, 1.743 ms] 372.396 µs (27.7%)
appsec_no_iast 1.747 ms [1.723 ms, 1.771 ms] 401.71 µs (29.9%)
iast 1.478 ms [1.455 ms, 1.5 ms] 132.369 µs (9.8%)
profiling 1.491 ms [1.467 ms, 1.516 ms] 146.184 µs (10.9%)
tracing 1.462 ms [1.438 ms, 1.487 ms] 117.241 µs (8.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.343 ms [1.324 ms, 1.363 ms] -
appsec 1.709 ms [1.685 ms, 1.734 ms] 366.116 µs (27.3%)
appsec_no_iast 1.74 ms [1.716 ms, 1.763 ms] 396.291 µs (29.5%)
iast 1.506 ms [1.483 ms, 1.53 ms] 163.093 µs (12.1%)
profiling 1.549 ms [1.524 ms, 1.574 ms] 205.641 µs (15.3%)
tracing 1.48 ms [1.456 ms, 1.504 ms] 136.974 µs (10.2%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~a638e84cb3, baseline=1.32.0-SNAPSHOT~329acac1da
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.054 µs) : 351, 393
.   : milestone, 372,
iast (474.897 µs) : 455, 495
.   : milestone, 475,
iast_FULL (542.63 µs) : 522, 563
.   : milestone, 543,
iast_GLOBAL (495.923 µs) : 475, 516
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (475.367 µs) : 454, 496
.   : milestone, 475,
iast_INACTIVE (452.535 µs) : 431, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (471.159 µs) : 450, 492
.   : milestone, 471,
tracing (451.508 µs) : 430, 473
.   : milestone, 452,
section candidate
no_agent (373.58 µs) : 354, 393
.   : milestone, 374,
iast (476.425 µs) : 456, 497
.   : milestone, 476,
iast_FULL (543.763 µs) : 523, 565
.   : milestone, 544,
iast_GLOBAL (500.221 µs) : 479, 521
.   : milestone, 500,
iast_HARDCODED_SECRET_DISABLED (478.956 µs) : 458, 500
.   : milestone, 479,
iast_INACTIVE (452.876 µs) : 431, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (477.138 µs) : 456, 498
.   : milestone, 477,
tracing (445.117 µs) : 425, 466
.   : milestone, 445,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.054 µs [351.284 µs, 392.825 µs] -
iast 474.897 µs [454.571 µs, 495.222 µs] 102.842 µs (27.6%)
iast_FULL 542.63 µs [522.025 µs, 563.236 µs] 170.576 µs (45.8%)
iast_GLOBAL 495.923 µs [475.492 µs, 516.354 µs] 123.869 µs (33.3%)
iast_HARDCODED_SECRET_DISABLED 475.367 µs [454.375 µs, 496.359 µs] 103.313 µs (27.8%)
iast_INACTIVE 452.535 µs [430.899 µs, 474.17 µs] 80.48 µs (21.6%)
iast_TELEMETRY_OFF 471.159 µs [450.407 µs, 491.91 µs] 99.105 µs (26.6%)
tracing 451.508 µs [430.08 µs, 472.937 µs] 79.454 µs (21.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 373.58 µs [353.844 µs, 393.317 µs] -
iast 476.425 µs [455.804 µs, 497.046 µs] 102.845 µs (27.5%)
iast_FULL 543.763 µs [522.971 µs, 564.554 µs] 170.182 µs (45.6%)
iast_GLOBAL 500.221 µs [479.226 µs, 521.216 µs] 126.641 µs (33.9%)
iast_HARDCODED_SECRET_DISABLED 478.956 µs [458.367 µs, 499.545 µs] 105.376 µs (28.2%)
iast_INACTIVE 452.876 µs [431.328 µs, 474.423 µs] 79.295 µs (21.2%)
iast_TELEMETRY_OFF 477.138 µs [455.886 µs, 498.39 µs] 103.558 µs (27.7%)
tracing 445.117 µs [424.643 µs, 465.592 µs] 71.537 µs (19.1%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review April 2, 2024 18:22
@nikita-tkachenko-datadog nikita-tkachenko-datadog requested a review from a team as a code owner April 2, 2024 18:22
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit 16f22b4 into master Apr 2, 2024
77 of 79 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/karate-skipped-feature-fix branch April 2, 2024 18:42
@github-actions github-actions bot added this to the 1.32.0 milestone Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@romainkomorndatadog romainkomorndatadog romainkomorndatadog approved these changes

@tonyredondo tonyredondo Awaiting requested review from tonyredondo tonyredondo is a code owner automatically assigned from DataDog/ci-app-libraries-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility type: bug
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@nikita-tkachenko-datadog @romainkomorndatadog