Add docs-site robots.txt AI crawler rules and a tracked llms.txt knowledge index by Copilot · Pull Request #30524 · github/gh-aw

Copilot · 2026-05-06T05:39:20Z

The docs site was missing AI-discoverability assets expected by the GEO audit: explicit crawler directives in robots.txt and a first-party llms.txt summary. This PR adds both under the docs static public path so they are served at /gh-aw/robots.txt and /gh-aw/llms.txt.

Robots policy updates
- Expanded docs/public/robots.txt from generic allow-only to explicit allow directives for major AI crawlers:
  - GPTBot
  - ClaudeBot
  - PerplexityBot
  - Google-Extended
- Preserved sitemap declaration for canonical crawl discovery.
New llms.txt document for model-oriented indexing
- Added docs/public/llms.txt with a 500+ word project summary.
- Content covers:
  - project purpose and architecture model
  - key concepts (markdown workflows, engines, MCP/tools, safety model)
  - direct links to core docs sections (intro/setup/reference/patterns/examples)
  - installation + quick-start flow for agent consumers
Repository tracking adjustment
- Updated .gitignore to explicitly allow docs/public/llms.txt (repo-wide *.txt ignore previously blocked it).

Example excerpt of the robots policy now shipped:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/graphql
- Triggering command: /usr/bin/gh gh repo view --json owner,name --jq .owner.login + "/" + .name rtcfg.link GOINSECURE GOMOD GOMODCACHE S4K5xSvh3iq54IRJtest@example.com (http block)
- Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linuremote.origin.url env SmQI/_0wLSN2hIggGQMUySmQI GO111MODULE util.test GOINSECURE GOMOD GOMODCACHE util.test (http block)
- Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile (http block)
https://api.github.com/orgs/test-owner/actions/secrets
- Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name go1.25.8 -c=4 -nolocalimports -importcfg /tmp/go-build374504578/b439/importcfg -pack /tmp/go-build374504578/b439/_testmain.go env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name h ../../../.pret--log-level=error gh 1/x64/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin/node /repos/actions/gnode --jq (http block)
https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1
- Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/x64/pkg/tool/linurepos/{owner}/{repo}/actions/runs/1/artifacts /usr/bin/git 504578/b431/embegit -buildtags /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git runs/20260506-05gh Test User 504578/b441/vet./repos/actions/github-script/git/ref/tags/v9 git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq [.object.sha, .object.type] | @tsv -L current (local changes) /usr/bin/git base (original) -L new (upstream) git rev-�� --show-toplevel /tmp/gh-aw-merge-765894969/new.md /usr/bin/git /ref/tags/v9 ../../../**/*.jsapi sv git (http block)
https://api.github.com/repos/actions/checkout/git/ref/tags/v3
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv remove remote2 /usr/bin/git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv --git-dir ad_handler (http block)
https://api.github.com/repos/actions/checkout/git/ref/tags/v5
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv ithub-script/git/ref/tags/v9.0.0 -buildtags bject.type] | @tsv -errorsas -ifaceassert -nilfunc /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linuTest User -uns�� -unreachable=false /tmp/go-build374504578/b228/vet.cfg 1/x64/bin/node "prettier" --cheinfocmp go 64/bin/go 1/x64/bin/node (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv ithub-script/git/ref/tags/v9 /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/link bject.type] | @tsv /tmp/go-build374git -importcfg /opt/hostedtoolc--show-toplevel git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/compile /usr/bin/git /tmp/go-build374git -trimpath /usr/bin/git git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel infocmp /usr/bin/infocmp xterm-color ache/go/1.25.8/xrev-parse /usr/bin/git infocmp -1 xterm-color git /usr/bin/git --show-toplevel util.test /usr/bin/gh git (http block)
https://api.github.com/repos/actions/checkout/git/ref/tags/v6
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv sistency_WithImports2142244740/001/main.md config /usr/lib/git-core/git-upload-pack remote.origin.urgit GO111MODULE 64/bin/go git-upload-pack /tmp�� 38324764 go /usr/bin/git -json GO111MODULE 64/bin/go git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv /tmp/go-build374504578/b445/_pkg_.a l /usr/bin/git -p main -lang=go1.25 git rev-�� --show-toplevel -dwarf=false /usr/bin/infocmp 01 -c=4 -nolocalimports infocmp (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -bool -buildtags /usr/bin/git -errorsas -ifaceassert -nilfunc git conf�� user.name Test User /usr/bin/git -json GO111MODULE At,event,headBra--show-toplevel git (http block)
https://api.github.com/repos/actions/github-script/git/ref/tags/v8
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv xterm-color docker /usr/bin/git test/race-image:git x_amd64/vet /usr/bin/git git rev-�� --show-toplevel git /usr/bin/infocmp .github/workflowgit x_amd64/vet /usr/bin/git infocmp (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv xterm-color git /usr/bin/git --get remote.origin.urrev-parse /usr/bin/git git rev-�� --show-toplevel git /usr/bin/infocmp --show-toplevel gh /usr/bin/git infocmp (http block)
https://api.github.com/repos/actions/github-script/git/ref/tags/v9
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE cNu6uh6Xsdg3 env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv -c=4 -nolocalimports -importcfg /tmp/go-build374504578/b417/importcfg -pack /home/REDACTED/work/gh-aw/gh-aw/pkg/envutil/envutil.go /home/REDACTED/work/gh-aw/gh-aw/pkg/envutil/envutil_test.go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
https://api.github.com/repos/actions/github-script/git/ref/tags/v9.0.0
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE tImIOC-NY5wN env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
https://api.github.com/repos/actions/setup-go/git/ref/tags/v4
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel /opt/hostedtoolcTest User /usr/bin/gh k/gh-aw/gh-aw /tmp/go-build374api 1/x64/bin/node gh api /repos/actions/github-script/git/ref/tags/v9 --jq /usr/bin/git ithub/workflows/git -buildtags /opt/hostedtoolc--show-toplevel git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel bash /usr/bin/infocmp --noprofile on 1/x64/bin/node infocmp -1 xterm-color 1/x64/bin/node /usr/bin/git || 'round-robin'git infocmp (http block)
https://api.github.com/repos/actions/setup-node/git/ref/tags/v4
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --get remote.origin.url /usr/bin/git -json GO111MODULE x_amd64/vet git rev-�� --git-dir resolved$ (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel 64/pkg/tool/linux_amd64/compile /usr/bin/git 2864663238 /tmp/go-build374api 504578/b361/vet./repos/actions/github-script/git/ref/tags/v9 git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linuremote.origin.url /usr/bin/git Z-LYbBHBh -buildtags /opt/hostedtoolc--show-toplevel git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linu5 /usr/bin/git ons-test57019896git -trimpath /opt/hostedtoolc--show-toplevel git rev-�� /ref/tags/v9 node sv ub/gh-aw.git url 64/pkg/tool/linuxterm-color gh (http block)
https://api.github.com/repos/actions/setup-node/git/ref/tags/v6
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv sistency_KeyOrdering944999471/001/test1.md show /usr/bin/git ntent.md GO111MODULE 64/bin/go 504578/b447/importcfg comm�� -aw/git/ref/tags/v1.2.3 k/gh-aw/gh-aw/pkg/stringutil/identifiers.go ache/node/24.14.1/x64/bin/node -json GO111MODULE 64/bin/go ache/node/24.14.1/x64/bin/node (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv /tmp/go-build374504578/b445/stats.test -importcfg /usr/bin/git -s -w -buildmode=exe git remo�� testing.testBinary=1 (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ansitiveImports3184368208/001 -test.v=true /usr/bin/git -test.timeout=10git -test.run=^Test -test.short=true--show-toplevel git conf�� --get remote.origin.url /usr/bin/git -json GO111MODULE x_amd64/vet git (http block)
https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v4
- Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv /ref/tags/v9 x_amd64/vet sv -json GO111MODULE ed } } git rev-�� --show-toplevel x_amd64/vet /usr/bin/git b/workflows GO111MODULE 64/pkg/tool/linu--show-toplevel git (http block)
- Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv ithub-script/git/ref/tags/v9 -tests bject.type] | @tsv e=false gh 1/x64/bin/sh git conf�� /ref/tags/v9 remote.origin.url sv Gitmain_branch78git Gitmain_branch78rev-parse layTitle git (http block)
https://api.github.com/repos/aws-actions/configure-aws-credentials/git/ref/tags/v4
- Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/node/24.14.1/x64/bin/node /usr/bin/git uts.branch config ps git rev-�� --show-toplevel ps /usr/bin/git git config pts.test git (http block)
- Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git /tmp/TestGuardPogh rev-parse clusion,workflow/repos/actions/github-script/git/ref/tags/v9 git rev-�� /ref/tags/v9 gh sv /ref/tags/v9 l sv infocmp (http block)
- Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel 8453316/b470/workflow.test /usr/bin/git t0 git "warnings":[]}] git rev-�� --show-toplevel git /usr/bin/git (http block)
https://api.github.com/repos/azure/login/git/ref/tags/v2
- Triggering command: /usr/bin/gh gh api /repos/azure/login/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/node/24.14.1/x64/bin/node /usr/bin/git orce_all)' -importcfg /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git /tmp/gh-aw-test-infocmp rev-parse /usr/bin/git git (http block)
- Triggering command: /usr/bin/gh gh api /repos/azure/login/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git ErrorFormatting8git remote (http block)
https://api.github.com/repos/docker/login-action/git/ref/tags/v3
- Triggering command: /usr/bin/gh gh api /repos/docker/login-action/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv --show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/compile /usr/bin/git /tmp/go-build374git -trimpath /usr/bin/git git rev-�� --show-toplevel git /usr/bin/infocmp 4-FU9HTvZ8owjYw_infocmp -dwarf=false om/myorg/repo.gixterm-color infocmp (http block)
- Triggering command: /usr/bin/gh gh api /repos/docker/login-action/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git /tmp/gh-aw-test-git rev-parse /usr/bin/gh git rev-�� --show-toplevel gh /usr/bin/infocmp 964369/001 --jq /opt/hostedtoolcxterm-color infocmp (http block)
https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.1.2
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq [.object.sha, .object.type] | @tsv /repos/actions/github-script/git/ref/tags/v9 --jq /usr/bin/git xfTG6WhB5 -importcfg 504578/b425/impo--show-toplevel git rev-�� --show-toplevel rtcfg /usr/bin/git FieldEnforcementinfocmp g/sliceutil/slic-1 ache/go/1.25.8/xxterm-color git (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq [.object.sha, .object.type] | @tsv --show-toplevel ex_logs.go /usr/bin/gh /ref/tags/v9 ex_playwright_teapi sv gh --ve�� --annotation=pla/repos/actions/github-script/git/ref/tags/v9 /opt/hostedtoolc--jq /usr/bin/git ithub-script/gitinfocmp pkg/workflow/sec-1 e/git git (http block)
https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.0.0
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv /v1.0.0 -extld=gcc sv -json GO111MODULE x_amd64/vet git rev-�� --show-toplevel x_amd64/vet r,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,disp--show-toplevel -json flow-12345 64/pkg/tool/linu--show-toplevel git (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv /ref/tags/v9 x_amd64/link sv iant-653409215 gh n-dir/bash git add /tmp/file-tracker-test1544025520/test1.md /tmp/file-tracker-test1544025520/test2.lock.yml /usr/bin/git te 'scripts/**/*git --jq es/.bin/node git (http block)
https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.2.3
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv --get remote.origin.url /usr/bin/git -json GO111MODULE nch,headSha,disp--show-toplevel git rev-�� --show-toplevel x_amd64/vet /usr/bin/git -json GO111MODULE 64/pkg/tool/linu--show-toplevel git (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv -v x_amd64/compile /usr/bin/git iant-653409215/.git infocmp ed } } git rev-�� --show-toplevel infocmp /usr/bin/git te 'scripts/**/*git git ad_handler git (http block)
https://api.github.com/repos/github/gh-aw/actions/runs
- Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-29 GOMOD GOMODCACHE 64/pkg/tool/linuremote.origin.url env -json cfg 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-06 GOMOD GOMODCACHE 64/pkg/tool/linuremote.origin.url env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-02-05 GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet env LsRemoteWithRealGitcustom_branch2444402576/001' LsRemoteWithRealGitcustom_branch2444402576/001' 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/1/artifacts --jq .artifacts[].name GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linuTest User (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/1/artifacts --jq .artifacts[].name git /sa1 ithub-script/gitgh git run-script/lib/n/repos/actions/github-script/git/ref/tags/v9 /sa1 --no�� --noprofile git 1/x64/bin/node npx @marp-team/mgh infocmp /usr/bin/git 1/x64/bin/node (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12345/artifacts --jq .artifacts[].name GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linu-trimpath env /ref/tags/v9 GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12345/artifacts --jq .artifacts[].name connection sv (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/1234567890
- Triggering command: /usr/bin/gh gh api repos/{owner}/{repo}/actions/runs/1234567890 --jq {databaseId: .id, number: .run_number, url: .html_url, status: .status, conclusion: .conclusion, workflowName: .name, workflowPath: .path, createdAt: .created_at, startedAt: .run_started_at, updatedAt: .updated_at, event: .event, headBranch: .head_branch, -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api repos/{owner}/{repo}/actions/runs/1234567890 --jq {databaseId: .id, number: .run_number, url: .html_url, status: .status, conclusion: .conclusion, workflowName: .name, workflowPath: .path, createdAt: .created_at, startedAt: .run_started_at, updatedAt: .updated_at, event: .event, headBranch: .head_branch, h ../../../.pret.prettierignore infocmp /node xterm-color nly /usr/bin/git git rev-�� pid; cd /home/REDACTED/work/gh-aw/gh-aw/docs/public && python3 -m http.server 8765 git bject.type] | @tsv ithub-script/gitnode gh (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12346/artifacts --jq .artifacts[].name GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/compile env 1606/001/stability-test.md GO111MODULE cfg GOINSECURE b/gh-aw/pkg/actirev-parse GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12346/artifacts --jq .artifacts[].name connection ow.lock.yml ithub-script/gitgit git bject.type] | @t--show-toplevel bash --no�� te 'scripts/**/*.js' --ignore-path .prettierignore --log-level=error git k/node_modules/.bin/node h ../../../.pretgit infocmp bject.type] | @t--show-toplevel /opt/google/chrome/chrome_crashpad_handler (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/2/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linu--json (http block)
- Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linu/tmp/go-build374504578/b115/vet.cfg (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/2/artifacts --jq .artifacts[].name infocmp sv :slides git /usr/bin/git /systemd-executor --no�� --noprofile git ache/go/1.25.8/x64/pkg/tool/linux_amd64/cgo npx @marp-team/mgit gh /usr/bin/infocmp--show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/cgo (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/3/artifacts --jq .artifacts[].name GO111MODULE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x^remote\..*\.gh-resolved$ (http block)
- Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 verutil_test.go ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linuremote.origin.url (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/3/artifacts --jq .artifacts[].name gh 1/x64/bin/node /repos/actions/ggit --jq /node /sadc t-ha�� ithub/workflows/ab-testing-advisor.md git ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet npx @marp-team/minfocmp git (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/4/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linu--json (http block)
- Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/4/artifacts --jq .artifacts[].name infocmp 1/x64/bin/node xterm-color git de erignore -has�� ithub/workflows/ab-testing-advisor.md git ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet npx @marp-team/mgit git /sh ache/go/1.25.8/x64/pkg/tool/linuorigin (http block)
https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/5/artifacts --jq .artifacts[].name GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/5/artifacts --jq .artifacts[].name git tartedAt,updatedAt,event,headBranch,headSha,displayTitle ./cmd/gh-aw git bject.type] | @t--show-toplevel erignore --no�� ting JSON files..." git ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet npx @marp-team/mgit gh (http block)
https://api.github.com/repos/github/gh-aw/actions/workflows
- Triggering command: /usr/bin/gh gh workflow list --json name,state,path -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD GOMODCACHE ache/go/1.25.8/xremote.origin.url (http block)
https://api.github.com/repos/github/gh-aw/contents/.github/workflows/shared/reporting.md
- Triggering command: /tmp/go-build374504578/b404/cli.test /tmp/go-build374504578/b404/cli.test -test.testlogfile=/tmp/go-build374504578/b404/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
- Triggering command: /tmp/go-build4098453316/b404/cli.test /tmp/go-build4098453316/b404/cli.test -test.testlogfile=/tmp/go-build4098453316/b404/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true --show-toplevel gh /usr/bin/gh /slides && cp slides/*.pdf publ rev-�� '**/*.ts' '**/*.json' --ignore-path ../../../.pr**/*.json gh 1/x64/bin/sh ithub-script/gitsh --jq bject.type] | @tnpx prettier --write '../../../**/*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore git (http block)
https://api.github.com/repos/github/gh-aw/git/ref/tags/v0.47.4
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git runs/20260506-05du remote.origin.ur-k /usr/bin/git git rev-�� --show-toplevel git /usr/bin/infocmp --show-toplevel x_amd64/vet /usr/bin/git infocmp (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq [.object.sha, .object.type] | @tsv --show-toplevel /opt/hostedtoolcache/node/24.14.1/x64/bin/node /usr/bin/git github.actor git /usr/bin/infocmp/tmp/gh-aw/aw-feature-branch.patch git rev-�� --show-toplevel infocmp /usr/bin/infocmp xterm-color l /usr/bin/git infocmp (http block)
https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE er.test GOINSECURE GOMOD GOMODCACHE er.test 7450�� architecture-gua--workflow t_test.go 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE /opt/hostedtoolc--jq (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv phen2529799504/001 phen2529799504/002/work ad_handler --show-toplevel gh /usr/bin/git ad_handler k/gh�� repo3207674199/001 chrome/Crash Reports x86_64/node /../../.prettiergit 24.04.4 LTS /usr/bin/git es/index.html && mkdir -p publicgit-upload-pack '/tmp/TestParseDefaultBranchFromLsRemoteWithRealGitmain_branch787903461/001' (http block)
https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env b/workflows GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv d-robin' --jq es/.bin/node /ref/tags/v9 git ode-gyp-bin/sh gh m/_n�� md --jq node /ref/tags/v9 r sv git (http block)
https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env b/workflows GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv iant-2911220064 gh n-dir/bash /repos/actions/ggit --jq /usr/bin/git git m/_n�� th .prettierignore --log-level=error git ad_handler --show-toplevel r /usr/bin/git ad_handler (http block)
https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env b/workflows GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq [.object.sha, .object.type] | @tsv ned-imports-enabled-with-body-content.md git ache/uv/0.11.10/x86_64/node yBeforeUnloadChegit git /usr/bin/git infocmp m/_n�� th .prettierignore --log-level=error git ad_handler --show-toplevel r /usr/bin/git ad_handler (http block)
https://api.github.com/repos/google-github-actions/auth/git/ref/tags/v2
- Triggering command: /usr/bin/gh gh api /repos/google-github-actions/auth/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git /tmp/TestGuardPogit remote ache/node/24.14.--show-toplevel git rev-�� --show-toplevel ache/node/24.14.1/x64/bin/node /usr/bin/git ere -buildtags /usr/bin/git git (http block)
- Triggering command: /usr/bin/gh gh api /repos/google-github-actions/auth/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel gh /usr/bin/git /repos/actions/ggit --jq /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git /usr/bin/git git /usr/bin/git git (http block)
https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999
- Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq [.object.sha, .object.type] | @tsv b3NCkL0nF 504578/b087/vet.cfg ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet rtcf�� 4538-33316/test-3554400396/.github/workflows tmain.go 64/pkg/tool/linux_amd64/link GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/link (http block)
- Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq [.object.sha, .object.type] | @tsv th .prettierignoremote.origin.url git ad_handler --show-toplevel r /usr/bin/git ad_handler ode_�� ay_c498693019/001 chrome/Crash Reports m/cr/report ns/configure-awsgh 24.04.4 LTS bject.type] | @t/repos/github/gh-aw/git/ref/tags/v2.0.0 npx (http block)
https://api.github.com/repos/nonexistent/repo/actions/runs/12345
- Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE tants GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion ithub-script/gitgit git ode-gyp-bin/node--show-toplevel /opt/google/chrome/chrome_crashpstatus --no�� te '../../../**/*.json' '!../../.github/workflows/test.md --database=/root/.config/google-chrome/Crash Reports 1/x64/bin/node --annotation=cha/usr/bin/git --annotation=lsbremote --annotation=pla-v 1/x64/bin/node (http block)
https://api.github.com/repos/owner/repo/actions/workflows
- Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh workflow list --repo owner/repo --json name,path,state cfg GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linuremote.origin.url ranc�� se 504578/b240/vet.cfg cfg GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x^remote\..*\.gh-resolved$ (http block)
- Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 1/x64/bin/node /repos/actions/gnode --jq (http block)
https://api.github.com/repos/test-owner/test-repo/actions/secrets
- Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name command -v debian-sa1 > /dev/nulGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go go env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
- Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name h ../../../.prettierignore gh ode /repos/actions/gnode 7db574e3..HEAD /usr/bin/git ttpsUpgrades,Len../../../**/*.json 1/x6�� w/js/**/*.json' --ignore-path git 1/x64/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin/sh son git sv git (http block)
https://api.github.com/repos/test/repo
- Triggering command: /usr/bin/gh gh api /repos/test/repo --jq .default_branch oyDE10HDn 504578/b169/vet.cfg 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet env g/constants/constants.go g/constants/engine_constants.go 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/link (http block)
- Triggering command: /usr/bin/gh gh api /repos/test/repo --jq .default_branch prettier --write ache/node/24.14.1/x64/bin/node !../../../pkg/woinfocmp --ignore-path ../../../.prettixterm-color ache/node/24.14.1/x64/bin/node 3922�� /ref/tags/v9 -buildtags sv -errorsas -ifaceassert -nilfunc git-receive-pack (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/39a8ecee-aebc-43fb-92b0-f8469ae6aa6e Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Initial plan

7aba3d1

Copilot AI assigned Copilot and pelikhan May 6, 2026

Copilot started work on behalf of pelikhan May 6, 2026 05:39 View session

Copilot AI linked an issue May 6, 2026 that may be closed by this pull request

[plan] Add robots.txt and llms.txt to docs site for AI crawler discoverability #30482

Open

4 tasks

docs: add crawler-focused robots and llms text files

5ba3e1b

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/39a8ecee-aebc-43fb-92b0-f8469ae6aa6e Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add robots.txt and llms.txt for AI crawler discoverability~~ Add docs-site robots.txt AI crawler rules and a tracked llms.txt knowledge index May 6, 2026

Copilot finished work on behalf of pelikhan May 6, 2026 05:52

Copilot AI requested a review from pelikhan May 6, 2026 05:52

pelikhan closed this May 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docs-site robots.txt AI crawler rules and a tracked llms.txt knowledge index#30524

Add docs-site robots.txt AI crawler rules and a tracked llms.txt knowledge index#30524
Copilot wants to merge 2 commits intomainfrom
copilot/add-robots-txt-llms-txt

Copilot AI commented May 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Copilot AI commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented May 6, 2026 •

edited

Loading