Update Git unshallow logic to handle cases when HEAD points to a local-only commit #5934

nikita-tkachenko-datadog · 2023-09-25T11:00:39Z

What Does This Do

Updates Git repository unshallowing logic in CI Visibility:

if HEAD points to a commit that does not exist in the remote repo, an attempt is made to calculate the HEAD of the tracked upstream branch and unshallow from that
if current local branch does not track any remote branch, or if HEAD is detached, no SHA is specified for unshallowing, that is all the commit data is checked out from the remote repo

Motivation

The command that is used for unshallowing currently has a drawback: if the local HEAD is a commit that has not been pushed to the remote, it will fail.
If that happens, the tracer should fall back to another command that will attempt to use the tracked branch for the current branch in order to unshallow.
It could be that the CI is working on a detached HEAD or maybe branch tracking hasn’t been set up. In that case, this command will also fail, and we will finally fallback to the behaviour where we just unshallow all the things.

…l-only commit

pr-commenter · 2023-09-25T11:44:39Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.22.0-SNAPSHOT~e7bf753b08	1.22.0-SNAPSHOT~031ea213a2
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.22.0-SNAPSHOT~031ea213a2, baseline=1.22.0-SNAPSHOT~e7bf753b08

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.006 s) : 0, 1006257
Total [baseline] (9.215 s) : 0, 9214991
Agent [candidate] (1.007 s) : 0, 1006775
Total [candidate] (9.207 s) : 0, 9207490
section appsec
Agent [baseline] (1.098 s) : 0, 1098365
Total [baseline] (9.317 s) : 0, 9316582
Agent [candidate] (1.093 s) : 0, 1092953
Total [candidate] (9.293 s) : 0, 9293363
section iast
Agent [baseline] (1.118 s) : 0, 1117713
Total [baseline] (9.397 s) : 0, 9396859
Agent [candidate] (1.118 s) : 0, 1117841
Total [candidate] (9.46 s) : 0, 9459831
section profiling
Agent [baseline] (1.179 s) : 0, 1179011
Total [baseline] (9.389 s) : 0, 9389071
Agent [candidate] (1.176 s) : 0, 1176245
Total [candidate] (9.365 s) : 0, 9364960

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.006 s	-
Agent	appsec	1.098 s	92.107 ms (9.2%)
Agent	iast	1.118 s	111.456 ms (11.1%)
Agent	profiling	1.179 s	172.754 ms (17.2%)
Total	tracing	9.215 s	-
Total	appsec	9.317 s	101.59 ms (1.1%)
Total	iast	9.397 s	181.868 ms (2.0%)
Total	profiling	9.389 s	174.08 ms (1.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.007 s	-
Agent	appsec	1.093 s	86.177 ms (8.6%)
Agent	iast	1.118 s	111.066 ms (11.0%)
Agent	profiling	1.176 s	169.469 ms (16.8%)
Total	tracing	9.207 s	-
Total	appsec	9.293 s	85.873 ms (0.9%)
Total	iast	9.46 s	252.341 ms (2.7%)
Total	profiling	9.365 s	157.47 ms (1.7%)

gantt
    title petclinic - break down per module: candidate=1.22.0-SNAPSHOT~031ea213a2, baseline=1.22.0-SNAPSHOT~e7bf753b08

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (623.551 ms) : 0, 623551
BytebuddyAgent [candidate] (623.449 ms) : 0, 623449
GlobalTracer [baseline] (293.122 ms) : 0, 293122
GlobalTracer [candidate] (293.939 ms) : 0, 293939
AppSec [baseline] (48.66 ms) : 0, 48660
AppSec [candidate] (48.481 ms) : 0, 48481
Remote Config [baseline] (663.832 µs) : 0, 664
Remote Config [candidate] (668.637 µs) : 0, 669
Telemetry [baseline] (5.964 ms) : 0, 5964
Telemetry [candidate] (5.956 ms) : 0, 5956
section appsec
BytebuddyAgent [baseline] (627.689 ms) : 0, 627689
BytebuddyAgent [candidate] (624.099 ms) : 0, 624099
GlobalTracer [baseline] (294.222 ms) : 0, 294222
GlobalTracer [candidate] (293.217 ms) : 0, 293217
AppSec [baseline] (135.426 ms) : 0, 135426
AppSec [candidate] (134.88 ms) : 0, 134880
Remote Config [baseline] (646.256 µs) : 0, 646
Remote Config [candidate] (637.073 µs) : 0, 637
Telemetry [baseline] (5.911 ms) : 0, 5911
Telemetry [candidate] (5.829 ms) : 0, 5829
section iast
BytebuddyAgent [baseline] (738.041 ms) : 0, 738041
BytebuddyAgent [candidate] (738.544 ms) : 0, 738544
GlobalTracer [baseline] (278.576 ms) : 0, 278576
GlobalTracer [candidate] (278.418 ms) : 0, 278418
AppSec [baseline] (45.971 ms) : 0, 45971
AppSec [candidate] (45.68 ms) : 0, 45680
IAST [baseline] (14.429 ms) : 0, 14429
IAST [candidate] (14.459 ms) : 0, 14459
Remote Config [baseline] (555.383 µs) : 0, 555
Remote Config [candidate] (560.242 µs) : 0, 560
Telemetry [baseline] (5.754 ms) : 0, 5754
Telemetry [candidate] (5.794 ms) : 0, 5794
section profiling
BytebuddyAgent [baseline] (635.758 ms) : 0, 635758
BytebuddyAgent [candidate] (633.861 ms) : 0, 633861
GlobalTracer [baseline] (354.336 ms) : 0, 354336
GlobalTracer [candidate] (352.917 ms) : 0, 352917
AppSec [baseline] (49.092 ms) : 0, 49092
AppSec [candidate] (48.712 ms) : 0, 48712
Remote Config [baseline] (648.307 µs) : 0, 648
Remote Config [candidate] (642.737 µs) : 0, 643
Telemetry [baseline] (5.998 ms) : 0, 5998
Telemetry [candidate] (6.019 ms) : 0, 6019
ProfilingAgent [baseline] (80.087 ms) : 0, 80087
ProfilingAgent [candidate] (81.288 ms) : 0, 81288
Profiling [baseline] (80.112 ms) : 0, 80112
Profiling [candidate] (81.314 ms) : 0, 81314

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.22.0-SNAPSHOT~031ea213a2, baseline=1.22.0-SNAPSHOT~e7bf753b08

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.014 s) : 0, 1014466
Total [baseline] (8.698 s) : 0, 8698455
Agent [candidate] (1.009 s) : 0, 1009281
Total [candidate] (8.661 s) : 0, 8661314
section appsec
Agent [baseline] (1.101 s) : 0, 1101017
Total [baseline] (8.738 s) : 0, 8737686
Agent [candidate] (1.092 s) : 0, 1092472
Total [candidate] (8.743 s) : 0, 8742830
section iast
Agent [baseline] (1.115 s) : 0, 1114741
Total [baseline] (9.216 s) : 0, 9216424
Agent [candidate] (1.13 s) : 0, 1130422
Total [candidate] (9.234 s) : 0, 9234131
section profiling
Agent [baseline] (1.179 s) : 0, 1179065
Total [baseline] (8.915 s) : 0, 8915028
Agent [candidate] (1.179 s) : 0, 1178528
Total [candidate] (8.872 s) : 0, 8871982

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.014 s	-
Agent	appsec	1.101 s	86.552 ms (8.5%)
Agent	iast	1.115 s	100.275 ms (9.9%)
Agent	profiling	1.179 s	164.599 ms (16.2%)
Total	tracing	8.698 s	-
Total	appsec	8.738 s	39.231 ms (0.5%)
Total	iast	9.216 s	517.969 ms (6.0%)
Total	profiling	8.915 s	216.573 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.009 s	-
Agent	appsec	1.092 s	83.191 ms (8.2%)
Agent	iast	1.13 s	121.141 ms (12.0%)
Agent	profiling	1.179 s	169.247 ms (16.8%)
Total	tracing	8.661 s	-
Total	appsec	8.743 s	81.516 ms (0.9%)
Total	iast	9.234 s	572.816 ms (6.6%)
Total	profiling	8.872 s	210.668 ms (2.4%)

gantt
    title insecure-bank - break down per module: candidate=1.22.0-SNAPSHOT~031ea213a2, baseline=1.22.0-SNAPSHOT~e7bf753b08

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (628.835 ms) : 0, 628835
BytebuddyAgent [candidate] (624.657 ms) : 0, 624657
GlobalTracer [baseline] (295.381 ms) : 0, 295381
GlobalTracer [candidate] (294.687 ms) : 0, 294687
AppSec [baseline] (48.901 ms) : 0, 48901
AppSec [candidate] (48.936 ms) : 0, 48936
Remote Config [baseline] (677.298 µs) : 0, 677
Remote Config [candidate] (677.68 µs) : 0, 678
Telemetry [baseline] (6.053 ms) : 0, 6053
Telemetry [candidate] (6.017 ms) : 0, 6017
section appsec
BytebuddyAgent [baseline] (628.639 ms) : 0, 628639
BytebuddyAgent [candidate] (623.254 ms) : 0, 623254
GlobalTracer [baseline] (295.395 ms) : 0, 295395
GlobalTracer [candidate] (293.595 ms) : 0, 293595
AppSec [baseline] (135.793 ms) : 0, 135793
AppSec [candidate] (134.891 ms) : 0, 134891
Remote Config [baseline] (642.895 µs) : 0, 643
Remote Config [candidate] (635.126 µs) : 0, 635
Telemetry [baseline] (5.887 ms) : 0, 5887
Telemetry [candidate] (5.834 ms) : 0, 5834
section iast
BytebuddyAgent [baseline] (736.208 ms) : 0, 736208
BytebuddyAgent [candidate] (748.551 ms) : 0, 748551
GlobalTracer [baseline] (277.65 ms) : 0, 277650
GlobalTracer [candidate] (280.137 ms) : 0, 280137
AppSec [baseline] (45.965 ms) : 0, 45965
AppSec [candidate] (45.967 ms) : 0, 45967
IAST [baseline] (14.432 ms) : 0, 14432
IAST [candidate] (14.557 ms) : 0, 14557
Remote Config [baseline] (555.816 µs) : 0, 556
Remote Config [candidate] (566.288 µs) : 0, 566
Telemetry [baseline] (5.748 ms) : 0, 5748
Telemetry [candidate] (5.827 ms) : 0, 5827
section profiling
BytebuddyAgent [baseline] (635.004 ms) : 0, 635004
BytebuddyAgent [candidate] (635.374 ms) : 0, 635374
GlobalTracer [baseline] (355.036 ms) : 0, 355036
GlobalTracer [candidate] (354.441 ms) : 0, 354441
AppSec [baseline] (48.645 ms) : 0, 48645
AppSec [candidate] (48.963 ms) : 0, 48963
Remote Config [baseline] (650.944 µs) : 0, 651
Remote Config [candidate] (650.762 µs) : 0, 651
Telemetry [baseline] (6.024 ms) : 0, 6024
Telemetry [candidate] (6.043 ms) : 0, 6043
ProfilingAgent [baseline] (80.529 ms) : 0, 80529
ProfilingAgent [candidate] (80.222 ms) : 0, 80222
Profiling [baseline] (80.553 ms) : 0, 80553
Profiling [candidate] (80.246 ms) : 0, 80246

Load

Parameters

	Baseline	Candidate
commit	1.22.0-SNAPSHOT~e7bf753b08	1.22.0-SNAPSHOT~031ea213a2
config	baseline	candidate
end_time	2023-09-25T11:21:21	2023-09-25T11:39:26
start_time	2023-09-25T11:21:03	2023-09-25T11:39:08

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.22.0-SNAPSHOT~031ea213a2, baseline=1.22.0-SNAPSHOT~e7bf753b08
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.338 ms) : 1318, 1357
.   : milestone, 1338,
appsec (1.723 ms) : 1699, 1748
.   : milestone, 1723,
iast (1.427 ms) : 1403, 1452
.   : milestone, 1427,
profiling (1.47 ms) : 1444, 1496
.   : milestone, 1470,
tracing (1.444 ms) : 1419, 1468
.   : milestone, 1444,
section candidate
no_agent (1.351 ms) : 1331, 1370
.   : milestone, 1351,
appsec (1.691 ms) : 1666, 1716
.   : milestone, 1691,
iast (1.446 ms) : 1422, 1471
.   : milestone, 1446,
profiling (1.478 ms) : 1453, 1504
.   : milestone, 1478,
tracing (1.45 ms) : 1426, 1475
.   : milestone, 1450,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.338 ms [1.318 ms, 1.357 ms]	-
appsec	1.723 ms [1.699 ms, 1.748 ms]	385.47 µs (28.8%)
iast	1.427 ms [1.403 ms, 1.452 ms]	89.431 µs (6.7%)
profiling	1.47 ms [1.444 ms, 1.496 ms]	131.796 µs (9.9%)
tracing	1.444 ms [1.419 ms, 1.468 ms]	105.685 µs (7.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.351 ms [1.331 ms, 1.37 ms]	-
appsec	1.691 ms [1.666 ms, 1.716 ms]	340.354 µs (25.2%)
iast	1.446 ms [1.422 ms, 1.471 ms]	95.462 µs (7.1%)
profiling	1.478 ms [1.453 ms, 1.504 ms]	127.752 µs (9.5%)
tracing	1.45 ms [1.426 ms, 1.475 ms]	99.908 µs (7.4%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.22.0-SNAPSHOT~031ea213a2, baseline=1.22.0-SNAPSHOT~e7bf753b08
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.853 µs) : 345, 385
.   : milestone, 365,
appsec (671.964 µs) : 651, 693
.   : milestone, 672,
iast (462.365 µs) : 441, 483
.   : milestone, 462,
iast_FULL (519.843 µs) : 499, 541
.   : milestone, 520,
iast_INACTIVE (424.436 µs) : 404, 445
.   : milestone, 424,
profiling (435.452 µs) : 414, 457
.   : milestone, 435,
tracing (428.921 µs) : 409, 449
.   : milestone, 429,
section candidate
no_agent (361.202 µs) : 341, 382
.   : milestone, 361,
appsec (667.896 µs) : 647, 689
.   : milestone, 668,
iast (459.317 µs) : 439, 480
.   : milestone, 459,
iast_FULL (519.417 µs) : 499, 540
.   : milestone, 519,
iast_INACTIVE (430.525 µs) : 410, 451
.   : milestone, 431,
profiling (440.971 µs) : 420, 462
.   : milestone, 441,
tracing (428.991 µs) : 408, 450
.   : milestone, 429,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.853 µs [344.943 µs, 384.762 µs]	-
appsec	671.964 µs [651.127 µs, 692.801 µs]	307.111 µs (84.2%)
iast	462.365 µs [441.301 µs, 483.43 µs]	97.513 µs (26.7%)
iast_FULL	519.843 µs [499.102 µs, 540.584 µs]	154.99 µs (42.5%)
iast_INACTIVE	424.436 µs [404.184 µs, 444.688 µs]	59.583 µs (16.3%)
profiling	435.452 µs [413.963 µs, 456.941 µs]	70.6 µs (19.4%)
tracing	428.921 µs [408.61 µs, 449.232 µs]	64.068 µs (17.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	361.202 µs [340.615 µs, 381.789 µs]	-
appsec	667.896 µs [647.165 µs, 688.627 µs]	306.694 µs (84.9%)
iast	459.317 µs [438.503 µs, 480.13 µs]	98.114 µs (27.2%)
iast_FULL	519.417 µs [498.588 µs, 540.246 µs]	158.215 µs (43.8%)
iast_INACTIVE	430.525 µs [409.701 µs, 451.348 µs]	69.323 µs (19.2%)
profiling	440.971 µs [419.635 µs, 462.308 µs]	79.769 µs (22.1%)
tracing	428.991 µs [408.371 µs, 449.611 µs]	67.789 µs (18.8%)

Update Git unshallow logic to handle cases when HEAD points to a loca…

031ea21

…l-only commit

nikita-tkachenko-datadog added type: bug comp: ci visibility Continuous Integration Visibility labels Sep 25, 2023

nikita-tkachenko-datadog marked this pull request as ready for review September 25, 2023 15:23

nikita-tkachenko-datadog requested a review from a team as a code owner September 25, 2023 15:23

juan-fernandez approved these changes Sep 26, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit 45b43b1 into master Sep 26, 2023
68 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/git-unshallow-local-commit branch September 26, 2023 08:52

github-actions bot added this to the 1.22.0 milestone Sep 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Git unshallow logic to handle cases when HEAD points to a local-only commit #5934

Update Git unshallow logic to handle cases when HEAD points to a local-only commit #5934

nikita-tkachenko-datadog commented Sep 25, 2023

pr-commenter bot commented Sep 25, 2023

Update Git unshallow logic to handle cases when HEAD points to a local-only commit #5934

Update Git unshallow logic to handle cases when HEAD points to a local-only commit #5934

Conversation

nikita-tkachenko-datadog commented Sep 25, 2023

What Does This Do

Motivation

pr-commenter bot commented Sep 25, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary