Skip to content

Docs: Explain GITHUB_TOKEN forwarding for eval #3841

Description

@elgreco247

Overview

Docker Agent does not automatically forward GITHUB_TOKEN or GH_TOKEN into eval containers, even when defined in the shell or ~/.config/cagent/.env. This is intentional because GitHub tokens are broad credentials, but the current docs do not make it clear.

Add a warning to the evaluation and GitHub Copilot provider docs:

Eval cases run in isolated containers. For security reasons, GITHUB_TOKEN and GH_TOKEN are not forwarded automatically. Explicitly pass the required token by name:

   docker agent eval agent.yaml ./evals -e GITHUB_TOKEN

When using a custom env file, both flags are required:

   docker agent eval agent.yaml ./evals \
     --env-from-file /path/to/secrets.env \
     -e GITHUB_TOKEN

Also clarify that host-side judge validation may succeed while the evaluated agent fails if the token is not forwarded.

Motivation

Env vars for other providers are forwarded to eval containers automatically and it is confusing that this doesn't happen for github-copilot.

Use cases

No response

Proposed solution

No response

Alternatives

No response

Related issues

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/docsDocumentation changesarea/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/testingTest infrastructure, CI/CD, test runners, evaluation

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions