diff --git a/docs/concepts/prompts_instructions.md b/docs-graveyard/concepts/prompts_instructions.md
similarity index 100%
rename from docs/concepts/prompts_instructions.md
rename to docs-graveyard/concepts/prompts_instructions.md
diff --git a/docs/concepts/reasking.md b/docs-graveyard/concepts/reasking.md
similarity index 100%
rename from docs/concepts/reasking.md
rename to docs-graveyard/concepts/reasking.md
diff --git a/docs/defining_guards/pydantic.ipynb b/docs-graveyard/defining_guards/pydantic.ipynb
similarity index 100%
rename from docs/defining_guards/pydantic.ipynb
rename to docs-graveyard/defining_guards/pydantic.ipynb
diff --git a/docs/defining_guards/strings.ipynb b/docs-graveyard/defining_guards/strings.ipynb
similarity index 100%
rename from docs/defining_guards/strings.ipynb
rename to docs-graveyard/defining_guards/strings.ipynb
diff --git a/docs/how_to_guides/envars.md b/docs/how_to_guides/envvars.md
similarity index 95%
rename from docs/how_to_guides/envars.md
rename to docs/how_to_guides/envvars.md
index 784456179..c451c9b29 100644
--- a/docs/how_to_guides/envars.md
+++ b/docs/how_to_guides/envvars.md
@@ -1,4 +1,4 @@
-# Environment Variables
+# Set API Keys and other Environment Variables
Guardrails recognize a handful of environment variables that can be used at runtime. Most of these correlate to envinronment variables used or expected by the various LLM clients. Below you can find a list of these and their uses.
diff --git a/docs/how_to_guides/llm_api_wrappers.md b/docs/how_to_guides/llm_api_wrappers.md
index 14e81dc66..a91d7a9d2 100644
--- a/docs/how_to_guides/llm_api_wrappers.md
+++ b/docs/how_to_guides/llm_api_wrappers.md
@@ -1,4 +1,4 @@
-# Custom LLMs
+# Guard Custom LLMs
Guardrails' `Guard` wrappers provide a simple way to add Guardrails to your LLM API calls. The wrappers are designed to be used with any LLM API.
diff --git a/docs/how_to_guides/logs.md b/docs/how_to_guides/logs.md
index 93ce2dc25..87eb647bc 100644
--- a/docs/how_to_guides/logs.md
+++ b/docs/how_to_guides/logs.md
@@ -1,8 +1,8 @@
-# Logs
+# Inspect Guard Run History and Logs
All `Guard` calls are logged internally, and can be accessed via the guard history.
-## 🇻🇦 Accessing logs via `Guard.history`
+## Accessing logs via `Guard.history`
`history` is an attribute of the `Guard` class. It implements a standard `Stack` interface with a few extra helper methods and properties. For more information on our `Stack` implementation see the [Helper Classes](/docs/api_reference_markdown/helper_classes) page.
diff --git a/docs/how_to_guides/rail.md b/docs/how_to_guides/rail.md
index c6932a0e7..604cffe4f 100644
--- a/docs/how_to_guides/rail.md
+++ b/docs/how_to_guides/rail.md
@@ -1,6 +1,6 @@
-# RailSpec
+# Use Guardrails with Markup
-## 🤖 What is `RAIL`?
+## What is `RAIL`?
`.RAIL` is a dialect of XML. It stands for "**R**eliable **AI** markup **L**anguage", and it can be used to define:
@@ -46,7 +46,7 @@ ${gr.json_suffix_prompt}
-## 🤔 Why `RAIL`?
+## Why `RAIL`?
1. **Language agnostic:** `RAIL` Specifications can be enforced in any language.
2. **Simple and familiar:** `RAIL` should be familiar to anyone familiar with HTML, and should be easy to learn.
@@ -59,7 +59,7 @@ ${gr.json_suffix_prompt}
- HTML, CSS and Javascript: `RAIL` spec is a dialect of XML, and so is similar to HTML. Specifying quality criteria is done via the `format` attribute, which is similar to CSS `style` tags. Corrective actions are specified via the `on-fail-*` attributes, which is similar to Javascript event handlers.
- OpenAPI as an open standard for creating machine-readable RESTful APIs.
-## 📚 Components of an `RAIL` Specification
+## Components of an `RAIL` Specification
The `RAIL` specification contains 2 main components:
@@ -109,3 +109,509 @@ _, validated_output, *rest = guard(
1. A `Guard` object is created from a `RAIL` specification. This object manages the validation and correction of the output of the LLM, as well as the prompt that is sent to the LLM.
2. Wrap the LLM API call (`openai.Completion.create`) with the `Guard` object, and add any additional arguments that you want to pass to the LLM API call. Instead of returning the raw text object, the `Guard` object will return a JSON object that is validated and corrected according to the `RAIL` specification.
+
+# `Instructions` Element
+
+The `` element is passed to the LLM as secondary input. Different model may use these differently. For example, chat models may receive instructions in the system-prompt.
+
+## Components of an Instructions Element
+
+In addition to any static text describing the context of the task, instructions can also contain any of the following:
+
+| Component | Syntax | Description |
+|-------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Variables | `${variable_name}` | These are provided by the user at runtime, and substituted in the instructions. |
+| Output Schema | `${output_schema}` | This is the schema of the expected output, and is compiled based on the `output` element. For more information on how the output schema is compiled for the instructions, check out [`output` element compilation](/docs/concepts/output/#adding-compiled-output-element-to-prompt) |
+| Prompt Primitives | `${gr.prompt_primitive_name}` | These are pre-constructed blocks of text that are useful for common tasks. E.g., some primitives may contain information that helps the LLM understand the output schema better. To see the full list of prompt primitives, check out [`guardrails/constants.xml`](https://github.com/guardrails-ai/guardrails/blob/main/guardrails/constants.xml). |
+
+
+Here's an example of how you could compose instructions using RAIL xml:
+```xml
+
+
+
+You are a helpful assistant only capable of communicating with valid JSON, and no other text.
+
+${gr.json_suffix_prompt_examples}
+
+
+```
+
+1. The instructions element contains high level background information for the LLM containing textual context and constraints.
+2. `${gr.json_suffix_prompt_examples}` is a prompt primitive provided by guardrails. It is equivalent to typing the following lines in the instructions:
+````
+ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter `null`.
+
+Here are examples of simple (XML, JSON) pairs that show the expected behavior:
+- ``]]> => `{'foo': 'example one'}`
+- `]]>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}`
+- `]]>` => `{'baz': {'foo': 'Some String', 'index': 1}}`
+````
+
+Or if you prefer Pydantic:
+```py
+#
+instructions = """You are a helpful assistant only capable of communicating with valid JSON, and no other text.
+
+ ${gr.json_suffix_prompt_examples}""" #
+```
+
+
+1. The instructions element contains high level background information for the LLM containing textual context and constraints.
+2. `${gr.json_suffix_prompt_examples}` is a prompt primitive provided by guardrails. It is equivalent to typing the following lines in the instructions:
+````
+ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter `null`.
+
+Here are examples of simple (XML, JSON) pairs that show the expected behavior:
+- ``]]> => `{'foo': 'example one'}`
+- `]]>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}`
+- `]]>` => `{'baz': {'foo': 'Some String', 'index': 1}}`
+````
+
+When either of the above are compiled, it would looks like this:
+```
+You are a helpful assistant only capable of communicating with valid JSON, and no other text.
+
+ONLY return a valid JSON object (no other text is necessary).
+The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types.
+Be correct and concise. If you are unsure anywhere, enter `null`.
+
+Here are examples of simple (XML, JSON) pairs that show the expected behavior:
+- `` => `{'foo': 'example one'}`
+- `` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}`
+- `` => `{'baz': {'foo': 'Some String', 'index': 1}}`
+```
+
+
+For an example of using instructions alongside a prompt see [this example for using chat models](../examples/guardrails_with_chat_models.ipynb).
+
+# `Prompt` Element
+
+The `` element contains the query that describes the high level task.
+
+## Components of a Prompt Element
+
+In addition to the high level task description, the prompt also contains the following:
+
+| Component | Syntax | Description |
+|-------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Variables | `${variable_name}` | These are provided by the user at runtime, and substituted in the prompt. |
+| Output Schema | `${output_schema}` | This is the schema of the expected output, and is compiled based on the `output` element. For more information on how the output schema is compiled for the prompt, check out [`output` element compilation](/docs/concepts/output/#adding-compiled-output-element-to-prompt). |
+| Prompt Primitives | `${gr.prompt_primitive_name}` | These are pre-constructed prompts that are useful for common tasks. E.g., some primitives may contain information that helps the LLM understand the output schema better. To see the full list of prompt primitives, check out [`guardrails/constants.xml`](https://github.com/guardrails-ai/guardrails/blob/main/guardrails/constants.xml). |
+
+```xml
+
+
+
+Given the following document, answer the following questions. If the answer doesn't exist in the document, enter 'None'.
+
+${document}
+
+
+${gr.xml_prefix_prompt}
+
+
+${output_schema}
+
+
+${gr.json_suffix_prompt}
+
+
+
+```
+
+1. The prompt contains high level task information.
+2. The variable `${document}` is provided by the user at runtime.
+3. `${gr.xml_prefix_prompt}` is a prompt primitive provided by guardrails. It is equivalent to typing the following lines in the prompt: `Given below is XML that describes the information to extract from this document and the tags to extract it into.`
+4. `${output_schema}` is the output schema and contains information about , which is compiled based on the `output` element.
+5. `${gr.json_suffix_prompt}` is a prompt primitive provided by guardrails. It is equivalent to typing the following lines in the prompt:
+```
+ONLY return a valid JSON object (no other text is necessary). The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter `null`.
+```
+
+# `Output` Element
+
+The `` element of a `RAIL` spec is used to give precise specification of the expected output of the LLM. It specifies
+
+1. the structure of the expected output (e.g. JSON),
+2. the type of each field,
+3. the quality criteria for each field to be considered valid (e.g. generated text should be bias-free, generated code should be bug-free), and
+4. the corrective action to take in case the quality criteria is not met (e.g. reask the question to the LLM, filter offending values, progrmatically fix, etc.)
+
+Example:
+
+
+=== "JSON RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Output JSON"
+
+ ```json
+ {
+ "text": "string output",
+ "score": 0.0,
+ "metadata": {
+ "key_1": "string",
+ ...
+ }
+ }
+ ```
+
+=== "String RAIL Spec"
+
+
+ ```xml
+
+
+
+ ```
+
+=== "Output String"
+
+ ```
+ string output
+ ```
+
+## ⚡ Specifying output structure
+
+You can combine `RAIL` elements to create an arbitrarily complex output structure.
+
+### Flat JSON output
+
+=== "RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Output JSON"
+ ```json
+ {
+ "some_key": "string",
+ "some_other_key": 0
+ }
+ ```
+
+
+### JSON output with objects
+
+`object` elements can be used to specify a JSON object, which is a collection of key-value pairs.
+
+- A child of an `object` element represents a key in the JSON object. The child element can be any RAIL element, including another `list` or `object` elements. The value of the key is generated by the LLM based on the info provided by the child element.
+- An object element can have multiple children, each of which can be any RAIL element, including another `list` or `object` elements.
+- Formatters can be applied to the child elements of an object element. For example, if the child element is a `string` element, the `format` attribute can be used to specify the quality criteria for the strings in the list.
+
+
+=== "RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Output JSON"
+ ```json
+ {
+ "some_object": {
+ "some_str_key": "SOME STRING",
+ "some_other_key": 0
+ }
+ }
+ ```
+
+In the above example, `"SOME STRING"` is the value for the `some_str_key` key, and is generated based on the name, description and quality criteria provided by the `` element.
+
+
+!!! note
+ The `object` element doesn't *need* to have children. If child elements are not provided, the LLM will automatically generate keys and values for the object based on the `name`, `description` and `format` attributes of the `object` element.
+
+ Providing child elements is useful when you want to specify the keys and values that the LLM should generate.
+
+### JSON output with lists
+
+`list` elements can be used to specify a list of values.
+
+- Currently, a list element can only contain a single child element. This means that a list can only contain a single type of data. For example, a list can only contain strings, or a list can only contain integers, but a list cannot contain both strings and integers.
+- This child element can be any RAIL element, including another `list` or `object` elements.
+- The child of a list element doesn't need to have a `name` attribute, since items in a list don't have names.
+- Formatters can be applied to the child element of a list element. For example, if the child element is a `string` element, the `format` attribute can be used to specify the quality criteria for the strings in the list.
+
+=== "RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Output JSON"
+ ```json
+ {
+ "some_list": [
+ "STRING 1", "STRING 2"
+ ]
+ }
+ ```
+
+
+!!! note
+ The `list` element doesn't *need* to have a child element. If a child element is not provided, the LLM will automatically generate values for the list based on the `name`, `description` and `format` attributes of the `list` element.
+
+ Providing a child element is useful when you want to have more control over the values that the LLM should generate.
+
+### String output
+
+Generate simple strings by specifying `type="string"` in the `` element.
+All the formatters supported by the `string` element can be used to specify the quality criteria for the generated string.
+
+=== "RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Output"
+ ```
+ string output
+ ```
+
+## `RAIL` Elements
+
+At the heart of the `RAIL` specification is the use of elements. Each element's tag represents a type of data. For example, in the element ``, the tag represents a string, the `` elements represents an integer, the `` element represents an object, etc.
+
+
+!!! note
+ The tag of RAIL element is the same as the "type" of the data it represents.
+
+ E.g. `` element will generate a string, `` element will generate an integer, etc.
+
+### Supported types
+
+Guardrails supports many data types, including:, `string`, `integer`, `float`, `bool`, `list`, `object`, `url`, `email` and many more.
+
+Check out the [RAIL Data Types](/docs/api_reference_markdown/datatypes) page for a list of supported data types.
+
+
+#### Scalar vs Non-scalar types
+
+Guardrails supports two types of data types: scalar and non-scalar.
+
+
+| Scalar | Non Scalar |
+|-------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| Scalar types are void elements, and can't have any child elements. | Non-scalar types can be non-void, and can have closing tags and child elements. |
+| Syntax: ``` ``` | Syntax: ``````|
+| Examples: `string`, `integer`, `float`, `bool`, `url`, `email`, etc. | Examples: `list` and `object` are the only non-scalar types supported by Guardrails. |
+
+
+### Supported attributes
+
+Each element can have attributes that specify additional information about the data, such as:
+
+1. `name` attribute that specifies the name of the field. This will be the key in the output JSON. E.g.
+
+=== "RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Output JSON"
+
+ ```json
+ {
+ "some_key": "..."
+ }
+ ```
+
+2. `description` attribute that specifies the description of the field. This is similar to a prompt that will be provided to the LLM. It can contain more context to help the LLM generate the correct output.
+3. (Coming soon!) `required` attribute that specifies whether the field is required or not. If the field is required, the LLM will be asked to generate the field until it is generated correctly. If the field is not required, the LLM will not be asked to generate the field if it is not generated correctly.
+4. `format` attribute that specifies the quality criteria that the field should respect. The `format` attribute can contain multiple quality criteria separated by a colon (`;`). For example, `two-words; upper-case`.
+5. `on-fail-{quality-criteria}` attribute that specifies the corrective action to take in case the quality criteria is not met. For example, `on-fail-two-words="reask"` specifies that if the field does not have two words, the LLM should be asked to re-generate the field.
+
+
+E.g.,
+
+=== "RAIL Spec"
+ ```xml
+
+
+
+ ```
+
+=== "Output JSON"
+
+ ```json
+ {
+ "some_key": "SOME STRING"
+ }
+ ```
+
+## Specifying quality criteria
+
+The `format` attribute allows specifying the quality criteria for each field in the expected output. The `format` attribute can contain multiple quality criteria separated by a colon (`;`). For example,
+
+```xml
+
+
+
+
+```
+
+The above example specifies that the `text` field should be a string with two words and the text should be returned in upper case.
+
+
+### Quality criteria under the hood
+
+Under the hood, the `format` attribute is parsed into a list of quality criteria.
+
+Each quality criteria is backed by a `Validator` class that checks if the generated output meets the quality criteria. For example, the `two-words` quality criteria is backed by the `TwoWords` class, which checks if the generated output has two words.
+
+Each quality criteria is then checked against the generated output. If the quality criteria is not met, the corrective action specified by the `on-fail-{quality-criteria}` attribute is taken.
+
+
+### Supported criteria
+
+- Each quality critera is relevant to a specific data type. For example, the `two-words` quality criteria is only relevant to strings, and the `positive` quality criteria is only relevant to integers and floats.
+- To see the full list of supported quality criteria, check out the [Validation](/docs/api_reference_markdown/validators) page.
+
+
+## 🛠️ Specifying corrective actions
+
+The `on-fail-{quality-criteria}` attribute allows specifying the corrective action that should be taken if the quality criteria is not met. The corrective action can be one of the following:
+
+| Action | Behavior |
+|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `reask` | Reask the LLM to generate an output that meets the quality criteria. The prompt used for reasking contains information about which quality criteria failed, which is auto-generated by the validator. |
+| `fix` | Programmatically fix the generated output to meet the quality criteria. E.g. for the formatter `two-words`, the programatic `fix` simply takes the first 2 words of the generated string. |
+| `filter` | Filter the incorrect value. This only filters the field that fails, and will return the rest of the generated output. |
+| `refrain` | Refrain from returning an output. If a formatter has the corrective action refrain, then on failure there will be a `None` output returned instead of the JSON. |
+| `noop` | Do nothing. The failure will still be recorded in the logs, but no corrective action will be taken. |
+| `exception` | Raise an exception when validation fails. |
+| `fix_reask` | First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking. |
+
+
+## Adding compiled `output` element to prompt
+
+In order to generate the correct LLM output, the `output` schema needs to be compiled and added to the prompt. This is handled automatically by the `Guardrails` library.
+
+The `output` element can be compiled into different formats to be used in the prompt. Currently, only a passthrough compilation into `XML` is supported, but in the future we will support additional compilation formats like `TypeScript`.
+
+
+### Passthrough (`XML`) compilation
+
+By default, the `output` element will be compiled into `XML` and added to the prompt. Compilation into `XML` involves removing any `on-fail-{quality-criteria}` attributes, and adding the `output` element to the prompt.
+
+An example of the compiled `output` element:
+
+=== "RAIL Spec"
+
+ ```xml
+
+
+
+ ```
+
+=== "Compiled XML added to prompt"
+
+ ```xml
+
+ ```
+
+
+### `TypeScript` Compilation
+
+Coming soon!
+
+
+
+## Unsupported tags and attributes
+
+- By default, Guardrails will not throw an error if you add an unsupported type, attribute or quality criteria. Instead, it will treat the unsupported type as a string, and will not perform any quality checks on the field. Often, LLMs will generate a string for an unsupported type, so this behavior is useful.
+- Unsupported tags and attributes will still be included in the output schema definition that is appended to the prompt.
+- This behavior can be changed by setting the `strict` attribute of the `