Repository navigation
Runs with zero prompts report SKIP 0/0 without showing the provider error." #2308
Replies: 3 comments
|
This is an artifact of recent support for The underlying |
|
I looked at this on current main. In OpenAICompatible._call_model, a 404 APIStatusError isn't in transient_retry_codes, so it isn't retried. It logs HTTP 404 from : and returns [None]. That matches the log in the report. The "40 attempts" look like 40 probe attempts each returning None, not backoff retries, and the run then finishes with no prompts scored, which shows as SKIP 0/0. |
|
Also the error is not silent, the exception is clearly logged in Note generators returning As noted in comment in #2141 a functionality to better indicate and surface unstable targets is a desired enhancement in need of a comprehensive design. |
Uh oh!
There was an error while loading. Please reload this page.
Command :- python -m garak --target_type openai --target_name llama-3.1-8b-instant --spec probes.policy.TemporalAuthority --generations 1
Terminal output :- policy.TemporalAuthority mitigation.MitigationBypass: SKIP ok on 0/ 0
garak run complete in 6.51s
The garak.log lines showing the 404 model_not_found :- HTTP 404 from https://api.groq.com/openai/v1/chat/completions: Error code: 404 - {'error': {'message': 'The model
llama-3.1-8b-instantdoes not exist or you do not have access to it.', 'type': 'invalid_request_error', 'code': 'model_not_found'}}probe return: ... with 40 attempts
The garak version (v0.17.1.pre1) and the Groq model name (llama-3.1-8b-instant)
All reactions