Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 60 additions & 46 deletions docs/concepts/logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,21 @@

All `Guard` calls are logged internally, and can be accessed via the guard history.

## Accessing logs via `Guard.history`
Whenever `Guard.__call__` or `Guard.parse` is called, a new `Call` entry is added to a stack in sequence of execution. This `Call` stack can be accessed through `Guard.history`.

Each entry in the history stack is a `Call` log which will contain information specific to a particular `Guard.__call__` or `Guard.parse` call in the order that they were executed within the current session.

For example, if you have a guard:
Calls can be further decomposed into a stack of `Iteration` objects. These are stateless and represent the interactions within a `Call` between llms, validators, inputs and outputs. The `Iteration` stack can be accessed through `call.iterations`.

## General Access
Given:
```py
my_guard = Guard.from_rail(...)
```
my_guard = Guard.from_pydantic(...)

and you call it multiple times:

```py
response_1 = my_guard(...)

response_2 = my_guard.parse(...)
```

Then `guard.history` will have two call logs with the first representing the first call `response_1 = my_guard(...)` and the second representing the following `parse` call `response_2 = my_guard.parse(...)`.
`my_guard.history`'s first `Call` entry will represent the guard execution corresponding to response_1 and the second will correspond to response_2's execution.

To pretty print logs for the latest call, run:

Expand All @@ -35,14 +31,12 @@ docs/html/single-step-history.html

--8<--

The `Call` log will contain initial and final information about a particular guard call.
## Calls
### Initial Input
Inital inputs like prompt and instructions from a call are available on each call.

```py
first_call = my_guard.history.first
```

For example, it tracks the initial inputs as provided:
```py
print("prompt\n-----")
print(first_call.prompt)
print("prompt params\n------------- ")
Expand Down Expand Up @@ -73,7 +67,20 @@ prompt params
{'opp_type': 'grizzly'}
```

as well as the final outputs:
Note: Input messages and msg_history currently can be accessed through iterations
```py
print(guard.history.last.iterations.last.inputs.msg_history)
```
```log
[
{"role":"system","content":"You are a helpful assistant."},
{"role":"user","content":"Tell me a joke"}
]
```


### Final Output
Final output of call is accessible on a call.
```py
print("status: ", first_call.status) # The final status of this guard call
print("validated response:", first_call.validated_output) # The final valid output of this guard call
Expand All @@ -83,23 +90,14 @@ status: pass
validated response: {'action': {'chosen_action': 'freeze', 'duration': 3}}
```


The `Call` log also tracks cumulative values from any iterations that happen within the call.

For example, if the first response from the LLM fails validation and a reask occurs, the `Call` log can provide total tokens consumed (*currently only for OpenAI models), as well as access to all of the raw outputs from the LLM:
### Cumulative Raw LLM outputs
`Call` log also the raw returns of llms before validation
```py
print("prompt token usage: ", first_call.prompt_tokens_consumed) # Total number of prompt tokens consumed across iterations within this call
print("completion token usage: ", first_call.completion_tokens_consumed) # Total number of completion tokens consumed across iterations within this call
print("total token usage: ",first_call.tokens_consumed) # Total number of tokens consumed; equal to the sum of the two values above
print("llm responses\n-------------") # An Stack of the LLM responses in order that they were received
for r in first_call.raw_outputs:
print(r)
```
```log
prompt token usage: 909
completion token usage: 57
total token usage: 966

llm responses
-------------
{"action": {"chosen_action": "freeze"}}
Expand All @@ -117,34 +115,31 @@ llm responses
}
```

For more information on `Call`, see the [History & Logs](/docs/api_reference_markdown/history_and_logs) page.

## 🇻🇦 Accessing logs from individual steps
In addition to the cumulative values available directly on the `Call` log, it also contains a `Stack` of `Iteration`'s. Each `Iteration` represent the logs from within a step in the guardrails process. This includes the call to the LLM, as well as parsing and validating the LLM's response.

Each `Iteration` is treated as a stateless entity so it will only contain information about the inputs and outputs of the particular step it represents.

For example, in order to see the raw LLM response as well as the logs for the specific validations that failed during the first step of a call, we can access this information via that steps `Iteration`:
### Cumulative Token usage
`Call` log also tracks llm token usage (*currently only for OpenAI models)
```py
print("prompt token usage: ", first_call.prompt_tokens_consumed) # Total number of prompt tokens consumed across iterations within this call
print("completion token usage: ", first_call.completion_tokens_consumed) # Total number of completion tokens consumed across iterations within this call
print("total token usage: ",first_call.tokens_consumed) # Total number of tokens consumed; equal to the sum of the two values above
```
```log
prompt token usage: 909
completion token usage: 57
total token usage: 966
```

## Iterations
### Validator logs
Detailed validator logs including outcomes and error spans can be accessed on interations.
```py
first_step = first_call.iterations.first

first_llm_output = first_step.raw_output
print("First LLM response\n------------------")
print(first_llm_output)
print(" ")

validation_logs = first_step.validator_logs
print("\nValidator Logs\n--------------")
for log in validation_logs:
print(log.json(indent=2))
```
```log
First LLM response
------------------
{"action": {"chosen_action": "fight", "weapon": "spoon"}}


Validator Logs
--------------
{
Expand Down Expand Up @@ -174,7 +169,25 @@ Validator Logs
}
```

Similar to the `Call` log, we can also see the token usage for just this step:
Failed validations can be conveniently accessed via `iteration.failed_validations`

### Raw LLM output
If multiple llm calls are made like in the case of the reask. Iterations contain the return of each call to an llm.
```py
first_step = first_call.iterations.first

first_llm_output = first_step.raw_output
print("First LLM response\n------------------")
print(first_llm_output)
```
```log
First LLM response
------------------
{"action": {"chosen_action": "fight", "weapon": "spoon"}}
```

### Token Usage
Token usage on a per step basis can be accessed on an Iteration.
```py
print("prompt token usage: ", first_step.prompt_tokens_consumed)
print("completion token usage: ", first_step.completion_tokens_consumed)
Expand All @@ -186,4 +199,5 @@ completion token usage: 16
token usage for this step: 633
```

For more information on `Call`, see the [History & Logs](/docs/api_reference_markdown/history_and_logs) page.
For more information on the properties available on `Iteration`, see the [History & Logs](/docs/api_reference_markdown/history_and_logs/#guardrails.classes.history.Iteration) page.
189 changes: 0 additions & 189 deletions docs/how_to_guides/logs.md

This file was deleted.