Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions client/composables/shiki.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { MaybeRef } from '@vueuse/core'
import type { HighlighterCore } from 'shiki'
import type { Ref } from 'vue'
import { createHighlighterCore } from 'shiki/core'
import { createJavaScriptRegexEngine } from 'shiki/engine/javascript'
import { computed, ref, toValue } from 'vue'
Expand Down Expand Up @@ -76,7 +76,7 @@ export async function loadShiki() {
return shiki.value
}

export function renderCodeHighlight(code: MaybeRef<string>, lang: 'json' | 'html' | 'bash' | 'robots-txt') {
export function renderCodeHighlight(code: string | Ref<string>, lang: 'json' | 'html' | 'bash' | 'robots-txt') {
return computed(() => {
const colorMode = devtools.value?.colorMode || 'light'
return shiki.value!.codeToHtml(toValue(code), {
Expand Down
30 changes: 11 additions & 19 deletions docs/content/2.guides/1.robots-txt.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,37 +60,29 @@ The following rules are parsed from your `robots.txt` file:
- `Disallow` - An array of paths to disallow for the user-agent.
- `Allow` - An array of paths to allow for the user-agent.
- `Sitemap` - An array of sitemap URLs to include in the generated sitemap.
- `Content-Usage` / `Content-Signal` - Directives for expressing AI usage preferences (see [Content Signals](#content-signals) below).
- `Content-Usage` / `Content-Signal` - Directives for expressing AI usage preferences (see [AI Directives](#ai-directives) below).

This parsed data will be shown for environments that are `indexable`.

## Content Signals
## AI Directives

Content Signals allow you to express preferences about how AI systems should interact with your content. Both `Content-Usage` and `Content-Signal` directives are supported:
AI Directives allow you to control how AI systems, search engines, and automated tools interact with your content. Two standards are supported:

### Content-Usage (IETF Standard)
- **Content-Usage** - IETF standard (`bots`, `train-ai`, `ai-output`, `search`) with `y`/`n` values
- **Content-Signal** - Cloudflare implementation (`search`, `ai-input`, `ai-train`) with `yes`/`no` values

The `Content-Usage` directive follows the [IETF AI Preferences specification](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/):
### Quick Example

```txt [robots.txt]
User-agent: *
Allow: /
Content-Usage: ai=n
Content-Usage: /public/ train-ai=y
Content-Usage: /restricted/ ai=n train-ai=n
Content-Usage: bots=y, train-ai=n
Content-Signal: ai-train=no, search=yes
```

### Content-Signal (Cloudflare Implementation)

The `Content-Signal` directive is [Cloudflare's implementation](https://blog.cloudflare.com/content-signals-policy/), widely deployed across millions of domains:

```txt [robots.txt]
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
```

Both directives are parsed identically and output as `Content-Usage` in the generated robots.txt. Use whichever format matches your preferences or existing tooling.
::alert{type="info"}
See the [**AI Directives Guide**](/docs/robots/guides/ai-directives) for complete documentation, examples, validation rules, and best practices.
::

## Conflicting `public/robots.txt`

Expand Down
Loading