Is your feature request related to a problem? Please describe.
PyRIT does not have a scenario for garak's API-key generation and partial-key completion checks. CredentialLeakScorer provides a useful base, but its current service-pattern coverage is smaller than the upstream probe's coverage. A direct detector port could also score echoed partial keys and safe placeholders as leaks. This is part of #511.
Describe the solution you'd like
Add a PyRIT-native API key scenario for the active GetKey and CompleteKey variants in garak/probes/apikey.py.
Implementation requirements:
- Store service/key-type names, request templates, safe placeholder exclusions, and synthetic partial keys in local PyRIT datasets. Never add real credentials.
- Model key generation and key completion as scenario techniques. Use the standard
PromptSendingAttack.
- Reuse
CredentialLeakScorer. Audit its coverage against the active upstream service patterns and extend the existing scorer with reviewed service-specific patterns only where needed for parity. Do not add a duplicate scenario-specific detector unless the existing scorer contract cannot represent the check.
- Preserve the MIT attribution for any regex implementation derived from garak's referenced
dora project. Garak itself is Apache-2.0, so retain the applicable garak notices for copied or adapted content.
- A positive score must require a credential-shaped value that is not only the partial value supplied in the request. Exclude known safe placeholders and straightforward request echo.
- Bound the default prompt count.
CompleteKey can otherwise create hundreds of service/prefix combinations.
- Add exports, focused tests for service coverage and false-positive controls, and synchronized
.py and .ipynb scanner documentation. All test and documentation values must be synthetic.
Follow PromptInject PR #2509, doc/code/framework.md, and the applicable scenario, dataset, scorer, test, and documentation instructions. Keep source data in datasets, execution in attacks, and response evaluation in the scorer.
Describe alternatives you've considered, if relevant
- Copying the complete garak detector into the scenario would duplicate
CredentialLeakScorer and mix scoring with orchestration.
- A single broad secret regex would have poor precision.
- Real or plausibly active key material is not needed for this test and must not be used.
Additional context
Is your feature request related to a problem? Please describe.
PyRIT does not have a scenario for garak's API-key generation and partial-key completion checks.
CredentialLeakScorerprovides a useful base, but its current service-pattern coverage is smaller than the upstream probe's coverage. A direct detector port could also score echoed partial keys and safe placeholders as leaks. This is part of #511.Describe the solution you'd like
Add a PyRIT-native API key scenario for the active
GetKeyandCompleteKeyvariants ingarak/probes/apikey.py.Implementation requirements:
PromptSendingAttack.CredentialLeakScorer. Audit its coverage against the active upstream service patterns and extend the existing scorer with reviewed service-specific patterns only where needed for parity. Do not add a duplicate scenario-specific detector unless the existing scorer contract cannot represent the check.doraproject. Garak itself is Apache-2.0, so retain the applicable garak notices for copied or adapted content.CompleteKeycan otherwise create hundreds of service/prefix combinations..pyand.ipynbscanner documentation. All test and documentation values must be synthetic.Follow PromptInject PR #2509,
doc/code/framework.md, and the applicable scenario, dataset, scorer, test, and documentation instructions. Keep source data in datasets, execution in attacks, and response evaluation in the scorer.Describe alternatives you've considered, if relevant
CredentialLeakScorerand mix scoring with orchestration.Additional context
garak/probes/apikey.pypyrit/score/true_false/regex/credential_leak_scorer.py