-
Notifications
You must be signed in to change notification settings - Fork 324
Add support for content parts and image URLs in AI Guard #10449
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
smola
wants to merge
1
commit into
master
Choose a base branch
from
smola/ai-guard-content-parts
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+847
−31
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 63 metrics, 8 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1059539
Total [baseline] (10.925 s) : 0, 10925405
Agent [candidate] (1.058 s) : 0, 1057912
Total [candidate] (10.982 s) : 0, 10982033
section appsec
Agent [baseline] (1.248 s) : 0, 1247594
Total [baseline] (11.081 s) : 0, 11080957
Agent [candidate] (1.228 s) : 0, 1227856
Total [candidate] (10.975 s) : 0, 10974766
section iast
Agent [baseline] (1.225 s) : 0, 1225208
Total [baseline] (11.199 s) : 0, 11198651
Agent [candidate] (1.228 s) : 0, 1227715
Total [candidate] (11.124 s) : 0, 11124454
section profiling
Agent [baseline] (1.186 s) : 0, 1186151
Total [baseline] (10.906 s) : 0, 10905707
Agent [candidate] (1.18 s) : 0, 1179744
Total [candidate] (10.94 s) : 0, 10939633
gantt
title petclinic - break down per module: candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.168 ms) : 0, 1168
crashtracking [candidate] (1.174 ms) : 0, 1174
BytebuddyAgent [baseline] (623.114 ms) : 0, 623114
BytebuddyAgent [candidate] (622.151 ms) : 0, 622151
AgentMeter [baseline] (28.695 ms) : 0, 28695
AgentMeter [candidate] (28.8 ms) : 0, 28800
GlobalTracer [baseline] (257.548 ms) : 0, 257548
GlobalTracer [candidate] (257.417 ms) : 0, 257417
AppSec [baseline] (33.24 ms) : 0, 33240
AppSec [candidate] (32.993 ms) : 0, 32993
Debugger [baseline] (62.853 ms) : 0, 62853
Debugger [candidate] (60.301 ms) : 0, 60301
Remote Config [baseline] (624.202 µs) : 0, 624
Remote Config [candidate] (629.186 µs) : 0, 629
Telemetry [baseline] (9.151 ms) : 0, 9151
Telemetry [candidate] (10.737 ms) : 0, 10737
Flare Poller [baseline] (7.712 ms) : 0, 7712
Flare Poller [candidate] (8.459 ms) : 0, 8459
section appsec
crashtracking [baseline] (1.207 ms) : 0, 1207
crashtracking [candidate] (1.173 ms) : 0, 1173
BytebuddyAgent [baseline] (661.877 ms) : 0, 661877
BytebuddyAgent [candidate] (649.929 ms) : 0, 649929
AgentMeter [baseline] (11.952 ms) : 0, 11952
AgentMeter [candidate] (11.824 ms) : 0, 11824
GlobalTracer [baseline] (260.067 ms) : 0, 260067
GlobalTracer [candidate] (256.783 ms) : 0, 256783
AppSec [baseline] (168.669 ms) : 0, 168669
AppSec [candidate] (166.834 ms) : 0, 166834
Debugger [baseline] (68.524 ms) : 0, 68524
Debugger [candidate] (67.151 ms) : 0, 67151
Remote Config [baseline] (686.521 µs) : 0, 687
Remote Config [candidate] (666.491 µs) : 0, 666
Telemetry [baseline] (9.357 ms) : 0, 9357
Telemetry [candidate] (9.322 ms) : 0, 9322
Flare Poller [baseline] (3.74 ms) : 0, 3740
Flare Poller [candidate] (3.644 ms) : 0, 3644
IAST [baseline] (25.982 ms) : 0, 25982
IAST [candidate] (25.367 ms) : 0, 25367
section iast
crashtracking [baseline] (1.174 ms) : 0, 1174
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (789.407 ms) : 0, 789407
BytebuddyAgent [candidate] (791.61 ms) : 0, 791610
AgentMeter [baseline] (11.148 ms) : 0, 11148
AgentMeter [candidate] (11.174 ms) : 0, 11174
GlobalTracer [baseline] (247.583 ms) : 0, 247583
GlobalTracer [candidate] (248.444 ms) : 0, 248444
AppSec [baseline] (33.069 ms) : 0, 33069
AppSec [candidate] (33.086 ms) : 0, 33086
Debugger [baseline] (67.811 ms) : 0, 67811
Debugger [candidate] (67.591 ms) : 0, 67591
Remote Config [baseline] (538.974 µs) : 0, 539
Remote Config [candidate] (546.262 µs) : 0, 546
Telemetry [baseline] (8.569 ms) : 0, 8569
Telemetry [candidate] (8.47 ms) : 0, 8470
Flare Poller [baseline] (3.503 ms) : 0, 3503
Flare Poller [candidate] (3.446 ms) : 0, 3446
IAST [baseline] (26.99 ms) : 0, 26990
IAST [candidate] (26.861 ms) : 0, 26861
section profiling
crashtracking [baseline] (1.231 ms) : 0, 1231
crashtracking [candidate] (1.216 ms) : 0, 1216
BytebuddyAgent [baseline] (677.653 ms) : 0, 677653
BytebuddyAgent [candidate] (673.604 ms) : 0, 673604
AgentMeter [baseline] (8.616 ms) : 0, 8616
AgentMeter [candidate] (8.526 ms) : 0, 8526
GlobalTracer [baseline] (216.485 ms) : 0, 216485
GlobalTracer [candidate] (215.341 ms) : 0, 215341
AppSec [baseline] (32.43 ms) : 0, 32430
AppSec [candidate] (32.019 ms) : 0, 32019
Debugger [baseline] (67.887 ms) : 0, 67887
Debugger [candidate] (67.503 ms) : 0, 67503
Remote Config [baseline] (586.399 µs) : 0, 586
Remote Config [candidate] (591.597 µs) : 0, 592
Telemetry [baseline] (8.681 ms) : 0, 8681
Telemetry [candidate] (8.737 ms) : 0, 8737
Flare Poller [baseline] (3.73 ms) : 0, 3730
Flare Poller [candidate] (3.691 ms) : 0, 3691
ProfilingAgent [baseline] (98.965 ms) : 0, 98965
ProfilingAgent [candidate] (99.055 ms) : 0, 99055
Profiling [baseline] (99.545 ms) : 0, 99545
Profiling [candidate] (99.621 ms) : 0, 99621
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057355
Total [baseline] (8.724 s) : 0, 8723789
Agent [candidate] (1.056 s) : 0, 1056306
Total [candidate] (8.69 s) : 0, 8690203
section iast
Agent [baseline] (1.224 s) : 0, 1223802
Total [baseline] (9.403 s) : 0, 9403053
Agent [candidate] (1.229 s) : 0, 1228637
Total [candidate] (9.36 s) : 0, 9359963
gantt
title insecure-bank - break down per module: candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.177 ms) : 0, 1177
crashtracking [candidate] (1.171 ms) : 0, 1171
BytebuddyAgent [baseline] (622.38 ms) : 0, 622380
BytebuddyAgent [candidate] (622.387 ms) : 0, 622387
AgentMeter [baseline] (28.721 ms) : 0, 28721
AgentMeter [candidate] (28.651 ms) : 0, 28651
GlobalTracer [baseline] (256.813 ms) : 0, 256813
GlobalTracer [candidate] (256.944 ms) : 0, 256944
AppSec [baseline] (32.846 ms) : 0, 32846
AppSec [candidate] (32.746 ms) : 0, 32746
Debugger [baseline] (61.663 ms) : 0, 61663
Debugger [candidate] (60.824 ms) : 0, 60824
Remote Config [baseline] (621.653 µs) : 0, 622
Remote Config [candidate] (612.14 µs) : 0, 612
Telemetry [baseline] (9.941 ms) : 0, 9941
Telemetry [candidate] (9.099 ms) : 0, 9099
Flare Poller [baseline] (7.695 ms) : 0, 7695
Flare Poller [candidate] (8.49 ms) : 0, 8490
section iast
crashtracking [baseline] (1.175 ms) : 0, 1175
crashtracking [candidate] (1.187 ms) : 0, 1187
BytebuddyAgent [baseline] (789.431 ms) : 0, 789431
BytebuddyAgent [candidate] (794.242 ms) : 0, 794242
AgentMeter [baseline] (11.122 ms) : 0, 11122
AgentMeter [candidate] (11.33 ms) : 0, 11330
GlobalTracer [baseline] (247.731 ms) : 0, 247731
GlobalTracer [candidate] (247.931 ms) : 0, 247931
AppSec [baseline] (32.823 ms) : 0, 32823
AppSec [candidate] (32.67 ms) : 0, 32670
Debugger [baseline] (66.722 ms) : 0, 66722
Debugger [candidate] (66.467 ms) : 0, 66467
Remote Config [baseline] (541.54 µs) : 0, 542
Remote Config [candidate] (538.217 µs) : 0, 538
Telemetry [baseline] (8.549 ms) : 0, 8549
Telemetry [candidate] (8.543 ms) : 0, 8543
Flare Poller [baseline] (3.498 ms) : 0, 3498
Flare Poller [candidate] (3.458 ms) : 0, 3458
IAST [baseline] (26.822 ms) : 0, 26822
IAST [candidate] (26.816 ms) : 0, 26816
LoadParameters
See matching parameters
SummaryFound 3 performance improvements and 1 performance regressions! Performance is the same for 15 metrics, 17 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section baseline
no_agent (1.179 ms) : 1168, 1191
. : milestone, 1179,
iast (3.24 ms) : 3194, 3286
. : milestone, 3240,
iast_FULL (5.733 ms) : 5676, 5791
. : milestone, 5733,
iast_GLOBAL (3.455 ms) : 3406, 3504
. : milestone, 3455,
profiling (2.131 ms) : 2111, 2151
. : milestone, 2131,
tracing (1.854 ms) : 1838, 1870
. : milestone, 1854,
section candidate
no_agent (1.166 ms) : 1155, 1177
. : milestone, 1166,
iast (3.214 ms) : 3177, 3251
. : milestone, 3214,
iast_FULL (5.811 ms) : 5752, 5869
. : milestone, 5811,
iast_GLOBAL (3.643 ms) : 3589, 3697
. : milestone, 3643,
profiling (1.932 ms) : 1916, 1948
. : milestone, 1932,
tracing (1.831 ms) : 1816, 1846
. : milestone, 1831,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section baseline
no_agent (18.984 ms) : 18785, 19182
. : milestone, 18984,
appsec (18.55 ms) : 18362, 18738
. : milestone, 18550,
code_origins (19.644 ms) : 19449, 19838
. : milestone, 19644,
iast (17.606 ms) : 17433, 17779
. : milestone, 17606,
profiling (18.631 ms) : 18447, 18815
. : milestone, 18631,
tracing (17.485 ms) : 17312, 17658
. : milestone, 17485,
section candidate
no_agent (17.643 ms) : 17453, 17832
. : milestone, 17643,
appsec (18.596 ms) : 18407, 18785
. : milestone, 18596,
code_origins (17.724 ms) : 17547, 17901
. : milestone, 17724,
iast (17.554 ms) : 17380, 17727
. : milestone, 17554,
profiling (19.246 ms) : 19056, 19437
. : milestone, 19246,
tracing (17.766 ms) : 17592, 17941
. : milestone, 17766,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section baseline
no_agent (15.613 s) : 15613000, 15613000
. : milestone, 15613000,
appsec (14.955 s) : 14955000, 14955000
. : milestone, 14955000,
iast (17.912 s) : 17912000, 17912000
. : milestone, 17912000,
iast_GLOBAL (17.899 s) : 17899000, 17899000
. : milestone, 17899000,
profiling (14.78 s) : 14780000, 14780000
. : milestone, 14780000,
tracing (14.789 s) : 14789000, 14789000
. : milestone, 14789000,
section candidate
no_agent (14.961 s) : 14961000, 14961000
. : milestone, 14961000,
appsec (14.619 s) : 14619000, 14619000
. : milestone, 14619000,
iast (18.424 s) : 18424000, 18424000
. : milestone, 18424000,
iast_GLOBAL (17.951 s) : 17951000, 17951000
. : milestone, 17951000,
profiling (15.256 s) : 15256000, 15256000
. : milestone, 15256000,
tracing (14.731 s) : 14731000, 14731000
. : milestone, 14731000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c3688ea03, baseline=1.59.0-SNAPSHOT~78be799166
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1465, 1488
. : milestone, 1477,
appsec (2.509 ms) : 2453, 2564
. : milestone, 2509,
iast (2.261 ms) : 2191, 2330
. : milestone, 2261,
iast_GLOBAL (2.317 ms) : 2247, 2387
. : milestone, 2317,
profiling (2.088 ms) : 2033, 2143
. : milestone, 2088,
tracing (2.062 ms) : 2009, 2115
. : milestone, 2062,
section candidate
no_agent (1.48 ms) : 1469, 1492
. : milestone, 1480,
appsec (3.75 ms) : 3529, 3971
. : milestone, 3750,
iast (2.265 ms) : 2195, 2334
. : milestone, 2265,
iast_GLOBAL (2.308 ms) : 2238, 2378
. : milestone, 2308,
profiling (2.517 ms) : 2353, 2681
. : milestone, 2517,
tracing (2.077 ms) : 2023, 2130
. : milestone, 2077,
|
5 tasks
940f8b5 to
8c3688e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Support multi-modal prompts (text + images) in AI Guard.
Additional Notes
Validated with system tests at DataDog/system-tests#6131
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: APPSEC-60264