FEAT: HuggingFace reproducibility params, OpenAI-compatible integration tests, and endpoint target deprecation#1672
Merged
romanlutz merged 14 commits intomicrosoft:mainfrom May 2, 2026
Conversation
…tTarget Add top_k, do_sample, repetition_penalty, and seed parameters for exact reproducibility of single-turn attacks on open-weight models. Fix conversation handling to send full history (system + user + assistant) through the chat template instead of only the last user turn. Store effective generation config in prompt_metadata for audit trail. Warn when sampling params are set without do_sample=True. Seed RNG once at construction time (not per request) and expose set_seed() for re-seeding between conversations. Build static generation params dict once at init rather than per request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add top_k, do_sample, and repetition_penalty parameters to HuggingFaceEndpointTarget, matching the HuggingFaceChatTarget API. These are passed through to the HF Inference Endpoints HTTP payload. Seed is intentionally excluded as the HF API does not reliably support server-side seeding. Include new params in the component identifier (None values are filtered out automatically). Add do_sample validation warning. Fix Optional to str | None per style guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test both OpenAIChatTarget (/v1/chat/completions) and OpenAIResponseTarget (/v1/responses) against the HF Inference Providers router. Tests read HUGGINGFACE_ENDPOINT from .env (consistent with other targets). Add HUGGINGFACE_ENDPOINT to .env_example. Verified: Chat Completions and Responses APIs work. The legacy Completions API (/v1/completions) is not supported by the HF router. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add HuggingFace to the list of OpenAI-compatible endpoints in the notebook, configuration docs, index, and populating secrets guide. Add a HuggingFace tab in the secrets setup with endpoint, token, and model examples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ibility # Conflicts: # pyrit/prompt_target/hugging_face/hugging_face_chat_target.py
Deprecate HuggingFaceEndpointTarget in favor of OpenAIChatTarget with the HuggingFace router endpoint. The HF Inference Providers API is OpenAI-compatible, making the legacy target redundant. Switch integration test default model from katanemo/Arch-Router-1.5B to meta-llama/Llama-3.1-8B-Instruct for better real-world coverage. Add HuggingFace to OpenAI-compatible provider lists in docs, notebook, configuration guide, and populating secrets (with tabbed example). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
romanlutz
changed the title
…rage Add 6 new tests covering _send_prompt_to_target_async (list response, dict response, optional params in payload, None params omitted, metadata model_id) and _validate_request (multi-piece rejection). Endpoint target coverage: 57% -> 95%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ibility # Conflicts: # pyrit/prompt_target/hugging_face/hugging_face_chat_target.py
… into romanlutz/hf-reproducibility
hannahwestra25
approved these changes
May 1, 2026
Rename the seed parameter to random_seed across HuggingFaceChatTarget and its tests to avoid confusion with seed prompts/objectives. Rename set_seed() to set_random_seed(). Internal field self._seed becomes self._random_seed. Identifier and effective config keys updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improve reproducibility for HuggingFace targets, add integration tests demonstrating HuggingFace works via OpenAI-compatible targets, and deprecate the legacy
HuggingFaceEndpointTarget.HuggingFaceChatTarget (local inference)
top_k,do_sample,repetition_penalty, andseedparameters for exact reproducibility of single-turn attacks on open-weight modelsset_seed()for re-seeding between conversationsprompt_metadatafor audit traildo_sample=TrueHuggingFaceEndpointTarget (deprecated)
top_k,do_sample,repetition_penaltyto HTTP payload and identifierOpenAIChatTargetwithendpoint=https://router.huggingface.co/v1. The HF Inference Providers API is OpenAI-compatible, making this target redundant. Will be removed in v0.15.0.seedintentionally excluded — HF Inference API does not reliably support server-side seedingOptionaltostr | Noneper style guideHuggingFace integration tests (new)
OpenAIChatTargetandOpenAIResponseTargetwork directly withHUGGINGFACE_ENDPOINT+HUGGINGFACE_TOKEN— no new target needed/v1/chat/completions) and Responses API (/v1/responses) usingmeta-llama/Llama-3.1-8B-InstructHUGGINGFACE_ENDPOINTto.env_exampleDocumentation
Test Results