Uptake AI Red Teaming Agent

[Sun Haoran (haoranpb)]

Official Documentation: https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/run-scans-ai-red-teaming-agent

Setup

Setup required:

1. Run .\scripts\Download-BCSymbols.ps1 -Category nl2al -InstanceId nl2al__move-name-customer-card-1
2. Create .env file like below
3. see also for local python setup https://github.com/microsoft/BC-Bench/commits/fix/bcal-pythonhome-under-uv-run/

# .env file

# Foundry Hub project (red teaming)
AZURE_SUBSCRIPTION_ID=<TODO>
AZURE_RESOURCE_GROUP=<TODO>
AZURE_PROJECT_NAME=<TODO>

# BCal agent
AZURE_OPENAI_ENDPOINT=<TODO>
AZURE_OPENAI_DEPLOYMENT=<TODO>

How to run:

# Build-in ones
uv run bcbench redteam scan --language en --risk-category violence

# Custom attack objectives
uv run bcbench redteam scan --language en --seeds C:\depot\BC-Bench\dataset\redteam\attack_objectives.sample.json

[Copilot]

Pull request overview

This PR adds a proof-of-concept AI Red Teaming capability to BC-Bench, wiring Azure AI Evaluation's RedTeam agent against the existing NL2AL (BCal) target. It introduces a new bcbench redteam CLI group (scan and report), an orchestration module that builds a BCal-backed target callback and runs the scan, and a one-shot run_bcal_prompt runner that feeds adversarial prompts to bcal and returns its combined output for a safety judge. Supporting changes add config paths, a sample attack-objectives file, .gitignore entries for private seeds, and the Azure dependencies.

Changes:

• New redteam scan/report CLI commands plus a redteam.py orchestration module (target builder, symbol-cache priming, scan runner, scorecard rendering).
• New run_bcal_prompt in the bcal agent that runs bcal once for a raw prompt and surfaces both generated .al files and stdout/diagnostics.
• Dependency, config, dataset-sample, and .gitignore additions to support the red-team workflow.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/bcbench/redteam.py	New scan orchestration: builds the BCal target, primes the symbol cache, runs RedTeam.scan.
src/bcbench/commands/redteam.py	New scan/report CLI commands and terminal scorecard rendering.
src/bcbench/agent/bcal/agent.py	Adds run_bcal_prompt red-team runner; minor comment typo (double space).
src/bcbench/agent/bcal/__init__.py	Exports run_bcal_prompt.
src/bcbench/config.py	Adds redteam_scorecard path.
src/bcbench/commands/__init__.py, src/bcbench/cli.py	Register the new redteam Typer app.
pyproject.toml	Adds azure-ai-evaluation[redteam] and azure-identity.
dataset/redteam/attack_objectives.sample.json	Sample custom attack-objective seed file.
.gitignore	Ignores private red-team seed files.
tests/conftest.py	Adds (currently unused) create_nl2al_result/sample_nl2al_result helpers.
tests/test_nl2al_pipeline.py	Removes an obsolete ty: ignore comment.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (5)

src/bcbench/agent/bcal/agent.py:184

• A nonzero BCal exit is converted into assistant text and then scored as though it were the model's answer. Because an error/traceback contains no harmful content, failed invocations can be reported as “resisted,” invalidating the attack-success results; execution errors must be surfaced outside the target response.

    except subprocess.CalledProcessError as exc:
        # Surface bcal's own output instead of letting the opaque CalledProcessError propagate (the red-team framework would otherwise report only "Something went wrong Command [...]").
        details = "\n".join(s.strip() for s in (exc.stdout, exc.stderr) if s and s.strip())
        return f"(bcal exited with status {exc.returncode})\n{details}".strip()

src/bcbench/commands/redteam.py:143

• attack_success is optional in the SDK result, but every missing/unevaluated value is rendered as a green “resisted.” Evaluation failures can therefore look like successful defenses; distinguish True, False, and None explicitly.

        result = "[red]\u2717 broke[/]" if row.get("attack_success") else "[green]\u2713 resisted[/]"

src/bcbench/agent/bcal/agent.py:137

• This new subprocess adapter has no automated coverage, although the sibling run_bcal_agent command construction is covered in tests/test_bcal_agent_provider.py. Add tests for argument/env plumbing and for generated-file, stdout, timeout, and nonzero-exit outcomes so scan-result integrity does not depend on untested process behavior.

def run_bcal_prompt(

src/bcbench/commands/redteam.py:45

• The PR's setup example provides only AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_DEPLOYMENT, but this default selects external-command, whose cli_args() requires BCAL_LLM_COMMAND. Following the documented setup therefore fails on the first target call; either default to azure-openai or update the setup/run example with the required backend and variables.

This issue also appears on line 143 of the same file.

    backend: Annotated[BCalLLMBackend, typer.Option(envvar="BCAL_LLM_BACKEND", help="BCal LLM backend used by the bcal target.")] = BCalLLMBackend.EXTERNAL_COMMAND,

src/bcbench/redteam.py:33

• Treating any .app as a complete cache can silently reuse a partial cache or symbols from a different BC version, because this fixed cache path records no version/completion marker. A subsequent scan then skips population and runs BCal against incomplete or stale symbols.

    if package_cache_path.exists() and any(package_cache_path.glob("*.app")):
        return

[Copilot]

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/bcbench/agent/bcal/agent.py:184

• Do not turn BCal execution failures into target responses. The red-team judge receives these strings as ordinary assistant output, so a crashed or timed-out target can be scored as having “resisted” the attack and produce false-negative security results. Propagate the exception (or otherwise mark the row as an execution error) instead.

    except subprocess.TimeoutExpired as exc:
        return f"(bcal timed out after {_config.timeout.bcal_execution}s)\n{exc.stdout or ''}".strip()
    except subprocess.CalledProcessError as exc:
        # Surface bcal's own output instead of letting the opaque CalledProcessError propagate (the red-team framework would otherwise report only "Something went wrong Command [...]").
        details = "\n".join(s.strip() for s in (exc.stdout, exc.stderr) if s and s.strip())
        return f"(bcal exited with status {exc.returncode})\n{details}".strip()

src/bcbench/agent/bcal/agent.py:176

• The default external-command path inherits PYTHONHOME, PYTHONPATH, and VIRTUAL_ENV from uv run. With the documented separate bridge virtualenv, those variables can make its Python load uv’s incompatible stdlib and BCal fails before answering any prompt. Pass a sanitized environment to this subprocess (and reuse it for the existing BCal invocation) so the documented red-team command can run.

        result = subprocess.run(
            cmd_args,
            timeout=_config.timeout.bcal_execution,
            capture_output=True,
            text=True,
            encoding="utf-8",
            errors="replace",
            check=True,

src/bcbench/agent/bcal/agent.py:141

• Add unit coverage for this new subprocess adapter. Nearby run_bcal_agent behavior is covered in tests/test_bcal_agent_provider.py, but no test invokes run_bcal_prompt, leaving command construction, generated-file/stdout collection, subprocess environment, timeout, and nonzero-exit behavior unchecked; the latter paths directly determine whether scan results are trustworthy.

def run_bcal_prompt(
    entry: NL2ALEntry,
    query: str,
    package_cache_path: Path,
    export_folder: Path,

src/bcbench/commands/redteam.py:50

• RedTeam.scan 1.18.2 treats output_path as a directory (and writes evaluation_results.json inside it), but this option advertises a JSON file and defaults to scorecard.json. Consequently --output result.json creates a directory named result.json, so callers cannot consume the file at the path the CLI promises. Model this option/default as an output directory, or translate the requested file path to the SDK directory and expose the actual inner file.

    output: Annotated[Path, typer.Option(help="Where to write the upstream scorecard JSON.")] = _config.paths.redteam_scorecard,

[Copilot]

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/bcbench/agent/bcal/agent.py:169

• The documented uv run + external-command flow inherits PYTHONHOME, PYTHONPATH, and VIRTUAL_ENV here. When BCal launches the separately-versioned bridge venv, those variables can make it load uv's stdlib and crash before handling any prompt. Pass a sanitized environment to both BCal subprocess call sites; the linked setup branch's commit 7eb8749 contains the needed helper and regression tests.

        result = subprocess.run(

src/bcbench/agent/bcal/agent.py:180

• Returning a timeout diagnostic as the target's assistant response lets the safety judge score an infrastructure timeout as attack_success=false ("resisted"), producing a false negative. Abort or mark this attempt undetermined instead of submitting the diagnostic for safety scoring.

    except subprocess.TimeoutExpired as exc:
        return f"(bcal timed out after {_config.timeout.bcal_execution}s)\n{exc.stdout or ''}".strip()

src/bcbench/agent/bcal/agent.py:184

• This converts a crashed BCal process into model output. The red-team judge can then classify the error text as harmless and report the attack as "resisted", so broken target runs improve the apparent safety result. Treat nonzero exits as scan failures or undetermined attempts, while retaining the details only as diagnostics.

    except subprocess.CalledProcessError as exc:
        # Surface bcal's own output instead of letting the opaque CalledProcessError propagate (the red-team framework would otherwise report only "Something went wrong Command [...]").
        details = "\n".join(s.strip() for s in (exc.stdout, exc.stderr) if s and s.strip())
        return f"(bcal exited with status {exc.returncode})\n{details}".strip()

src/bcbench/agent/bcal/agent.py:137

• The new raw-prompt subprocess path has no unit coverage, although the sibling run_bcal_agent path is exercised in tests/test_bcal_agent_provider.py:80-179. Add tests for successful AL/stdout aggregation and for timeout/nonzero-exit handling so failures cannot silently become safety responses.

def run_bcal_prompt(

src/bcbench/commands/redteam.py:45

• The PR's setup example provides only AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_DEPLOYMENT, but the documented command uses this external-command default, which requires BCAL_LLM_COMMAND and fails before scanning when it is absent. Either default this command to azure-openai or update the setup/run instructions with the required external-command bridge configuration.

    backend: Annotated[BCalLLMBackend, typer.Option(envvar="BCAL_LLM_BACKEND", help="BCal LLM backend used by the bcal target.")] = BCalLLMBackend.EXTERNAL_COMMAND,

src/bcbench/commands/redteam.py:143

• attack_success is optional for undetermined or failed evaluations, but this truthiness check labels every missing/None value as a green "resisted" result. Render an explicit third state so incomplete scans cannot be mistaken for successful resistance.

        result = "[red]\u2717 broke[/]" if row.get("attack_success") else "[green]\u2713 resisted[/]"

[Copilot]

Pull request overview

Copilot reviewed 12 out of 14 changed files in this pull request and generated 2 comments.

Suppressed comments (4)

src/bcbench/agent/bcal/agent.py:169

• The linked setup fix documents that uv run exports PYTHONHOME/PYTHONPATH/VIRTUAL_ENV, which can make BCal’s separate external-command bridge load an incompatible stdlib and crash. This subprocess inherits those variables, so local scans using the documented bridge setup remain broken; pass a copied environment with those three keys removed (and apply the same helper to the other BCal subprocess call site).

        result = subprocess.run(

src/bcbench/agent/bcal/agent.py:180

• A timeout is returned as normal assistant output, allowing the safety judge to classify a crashed target as “resisted.” Execution failures must remain unevaluated rather than affect the attack-success score; raise an AgentError/AgentTimeoutError so the callback/scan can record or surface the failure.

    except subprocess.TimeoutExpired as exc:
        return f"(bcal timed out after {_config.timeout.bcal_execution}s)\n{exc.stdout or ''}".strip()

src/bcbench/commands/redteam.py:145

• A missing/None SDK result is currently rendered as “resisted,” producing a false safety pass. The added test also imports _attack_result, which is not defined, so the suite fails during collection. Preserve all three states and use the helper here.

        result = "[red]\u2717 broke[/]" if row.get("attack_success") else "[green]\u2713 resisted[/]"

.github/workflows/bcal-evaluation.yml:145

• The linked local-Python setup explicitly requires the CAPI bridge in its own Python 3.12 venv; this change instead builds it with BC-Bench’s Python 3.13 interpreter. Matching versions is not the documented fix for inherited Python state (the subprocess environment must be sanitized), and it removes the runtime known to work with bc-eval[capi]==0.3.13. Keep the bridge on 3.12 and apply the environment fix at the BCal call sites.

          # Match the bridge runtime to BC-Bench so BCal can safely inherit its Python environment.
          uv venv .bcal-capi-venv --python .\.venv\Scripts\python.exe

[Copilot]

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/bcbench/commands/redteam.py:61

• These examples omit the required --language option, so copying either command results in Typer reporting a missing option instead of starting a scan. Include a language as the PR description's examples do.

        uv run bcbench redteam scan --risk-category code_vulnerability
        uv run bcbench redteam scan --seeds dataset/redteam/attack_objectives.json

src/bcbench/agent/bcal/agent.py:175

• The linked setup fix is missing here: under uv run, this subprocess inherits PYTHONHOME, PYTHONPATH, and VIRTUAL_ENV. When BCal starts a separate external-command bridge venv, that interpreter can resolve uv's project stdlib/site-packages instead of its own and crash; the linked commit reproduces this as an SRE module mismatch. Pass a sanitized environment to both BCal subprocess call sites.

        result = subprocess.run(

src/bcbench/redteam.py:68

• This async callback directly executes the synchronous subprocess.run inside run_bcal_prompt, which can block the SDK event loop for the full 25-minute BCal timeout. The red-team SDK schedules attack orchestrators concurrently, so this serializes scans and prevents async timeout/cancellation from progressing while BCal is running. Offload the blocking call to a worker thread.

        response = run_bcal_prompt(cast(NL2ALEntry, entry), query, package_cache_path, export_folder, backend_config)

src/bcbench/commands/redteam.py:45

• Following the PR's documented .env and run commands fails before scanning: the setup supplies only AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT, while this defaults to external-command and llm_command remains unset, so BCalBackendConfig.cli_args() raises that BCAL_LLM_COMMAND is required. Either default to the Azure OpenAI backend or update the documented setup to configure the external command.

This issue also appears on line 60 of the same file.

    backend: Annotated[BCalLLMBackend, typer.Option(envvar="BCAL_LLM_BACKEND", help="BCal LLM backend used by the bcal target.")] = BCalLLMBackend.EXTERNAL_COMMAND,

[Copilot]

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (5)

src/bcbench/agent/bcal/agent.py:193

• The added tests cover only timeout and nonzero-exit behavior; the successful path that reads generated .al files, appends stdout, and supplies the no-output fallback is untested. This returned text is exactly what the safety judge scores, so regressions here can silently skew red-team results. Add focused success-path tests for these output combinations.

    generated: str = "\n\n".join(p.read_text(encoding="utf-8", errors="replace") for p in sorted(export_folder.rglob("*.al")))

src/bcbench/redteam.py:69

• This synchronous subprocess call runs inside the SDK's async target callback and can block the event loop for the full 25-minute BCal timeout. The 1.18.2 SDK executes up to five attack tasks in parallel by default, so this serializes target calls and also prevents other SDK tasks and timers from progressing. Offload the blocking call to a worker thread.

        response = run_bcal_prompt(cast(NL2ALEntry, entry), query, package_cache_path, export_folder, backend_config)

src/bcbench/redteam.py:120

• Mark this scan as targeting an agent. RedTeam.scan defaults is_agent_target to false in SDK 1.18.2 and rejects the sensitive_data_leakage, task_adherence, and prohibited_actions categories in that mode. Those categories are accepted by this CLI's RiskCategory option, so selecting one currently fails before any BCal attack runs.

    scan_kwargs: dict[str, Any] = {"target": tracked_target, "output_path": str(output_path)}

src/bcbench/redteam.py:117

• Recording every callback attempt produces false scan failures when the SDK retries successfully. In SDK 1.18.2, the callback wrapper converts generic errors containing rate limit, 429, or too many requests into retryable errors; this wrapper records the original attempt first, so line 128 later raises it even if a retry completed and produced valid attack results. Track terminal failures rather than all attempts.

        except Exception as error:
            target_errors.append(error)

src/bcbench/commands/redteam.py:45

• The documented setup provides only AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_DEPLOYMENT, and .env.sample:19-25 also describes Azure OpenAI as the default. With external-command here, the documented bcbench redteam scan invocation instead reaches its first target call and fails because BCAL_LLM_COMMAND is missing. Default this command to Azure OpenAI, consistent with bcbench run bcal.

    backend: Annotated[BCalLLMBackend, typer.Option(envvar="BCAL_LLM_BACKEND", help="BCal LLM backend used by the bcal target.")] = BCalLLMBackend.EXTERNAL_COMMAND,

[Copilot]

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/bcbench/agent/bcal/agent.py:177

• This new red-team path launches BCal with the full uv run Python environment. For an external-command bridge in a separate venv, inherited PYTHONHOME/PYTHONPATH/VIRTUAL_ENV can make that interpreter load uv's incompatible stdlib and crash before returning a target response. Pass a copied environment with those variables removed; use the same helper for the existing run_bcal_agent subprocess too.

        result = subprocess.run(
            cmd_args,
            timeout=_config.timeout.bcal_execution,
            capture_output=True,
            text=True,
            encoding="utf-8",
            errors="replace",
            check=True,
        )

src/bcbench/commands/redteam.py:45

• The documented setup provides only AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT, and both documented scan examples omit --backend. With this default they instead fail in BCalBackendConfig.cli_args() because BCAL_LLM_COMMAND is absent. Default to Azure OpenAI, matching bcbench run bcal and .env.sample, or the documented setup is not runnable.

    backend: Annotated[BCalLLMBackend, typer.Option(envvar="BCAL_LLM_BACKEND", help="BCal LLM backend used by the bcal target.")] = BCalLLMBackend.EXTERNAL_COMMAND,

.env.sample:27

• The example still invokes bare python, which uv run resolves to BC-Bench's project environment rather than the separate environment containing bc-eval[capi]. Uncommenting this advertised override therefore fails to import the bridge dependency. Show the bridge venv's interpreter and the bridge script explicitly, as the workflow does.

# BCAL_LLM_COMMAND=python -m bcbench.agent.bcal.bc_eval_capi_bridge  # optional override; point at a venv that has bc-eval[capi]

[Copilot]

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

src/bcbench/agent/bcal/agent.py:169

• This new subprocess inherits PYTHONHOME, PYTHONPATH, and VIRTUAL_ENV from uv run. With the documented external-command bridge in a separate-version venv, BCal launches that interpreter against uv's incompatible standard library and the target crashes before producing a response. Pass a sanitized environment here (ideally via a helper shared with run_bcal_agent) that removes those parent-Python variables.

        result = subprocess.run(

src/bcbench/commands/redteam.py:154

• Attack prompts and target responses are passed to Rich as raw strings, so bracketed content is parsed as markup. Ordinary AL attributes or crafted seeds can therefore alter rendering or raise a Rich style/markup error, causing scan/report to fail while displaying valid results. Wrap these two untrusted cells in rich.text.Text (or escape them) before adding the row.

        table.add_row(str(index), str(row.get("risk_category", "-")), str(row.get("attack_technique", "-")), result, _short(_turn(row, "user")), _short(_turn(row, "assistant")))

.env.sample:27

• This example does not actually point at the venv mentioned in its comment: under uv run, bare python resolves to BC-Bench's project venv, which does not install bc-eval[capi]. Users selecting the external-command backend will get an import failure. Show the bridge venv's Python executable and the bridge script as explicit absolute paths, as the workflow does.

# BCAL_LLM_COMMAND=python -m bcbench.agent.bcal.bc_eval_capi_bridge  # optional override; point at a venv that has bc-eval[capi]

[Copilot]

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/bcbench/commands/redteam.py:186

• A summary containing only the SDK's overall_asr triple is valid—the SDK emits exactly that shape for its default/no-evaluations scorecard—but this condition drops the entire table. Consequently redteam report renders no ASR information for those saved scorecards. Treat overall_asr as sufficient to render the existing overall row.

    groups = [key.removesuffix("_asr") for key in row if key.endswith("_asr") and key != "overall_asr"]
    if not groups:
        return None