Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ja/guides/weave.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ W&B Weave を使用すると、次のことが可能です:
* 実験から評価、プロダクションまでの LLM ワークフローで生成されたすべての情報を整理

<Note>
Weave のドキュメントをお探しですか?[W&B Weave Docs](https://weave-docs.wandb.ai/) をご覧ください。
Weave のドキュメントをお探しですか?[W&B Weave Docs](/weave) をご覧ください。
</Note>

## 開始方法
Expand Down
2 changes: 1 addition & 1 deletion ja/models/ref.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,5 @@ Node サーバーからメトリクスをトラッキングするベータ版の
</CardGroup>

<Note>
Weave API をお探しですか? [W&B Weave Docs](https://weave-docs.wandb.ai/)を参照してください。
Weave API をお探しですか? [W&B Weave Docs](/weave)を参照してください。
</Note>
2 changes: 1 addition & 1 deletion ko/guides/weave.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ W&B Weave를 사용하면 다음을 수행할 수 있습니다.
* 실험에서 평가, production에 이르기까지 LLM 워크플로우에서 생성된 모든 정보 구성

<Note>
Weave 문서를 찾고 계십니까? [W&B Weave Docs](https://weave-docs.wandb.ai/)를 참조하세요.
Weave 문서를 찾고 계십니까? [W&B Weave Docs](/weave)를 참조하세요.
</Note>

## 시작 방법
Expand Down
2 changes: 1 addition & 1 deletion ko/models/ref.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,5 @@ Node 서버에서 메트릭을 추적하는 베타 JavaScript/TypeScript 클라
</CardGroup>

<Note>
Weave API를 찾고 계십니까? [W&B Weave Docs](https://weave-docs.wandb.ai/)를 참조하십시오.
Weave API를 찾고 계십니까? [W&B Weave Docs](/weave)를 참조하십시오.
</Note>
4 changes: 2 additions & 2 deletions models/evaluate-models.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ results = await evaluation.evaluate(model)

### Integrate Weave evaluations with W&B Models

The [Models and Weave Integration Demo](https://weave-docs.wandb.ai/reference/gen_notebooks/Models_and_Weave_Integration_Demo) shows the complete workflow for:
The [Models and Weave Integration Demo](/weave/reference/gen_notebooks/Models_and_Weave_Integration_Demo) shows the complete workflow for:

1. **Load models from Registry**: Download fine-tuned models stored in W&B Models Registry
2. **Create evaluation pipelines**: Build comprehensive evaluations with custom scorers
Expand Down Expand Up @@ -120,7 +120,7 @@ for model in models:
### Next steps

* [Complete Weave evaluation tutorial](/weave/tutorial-eval/)
* [Models and Weave integration example](https://weave-docs.wandb.ai/reference/gen_notebooks/Models_and_Weave_Integration_Demo)
* [Models and Weave integration example](/reference/gen_notebooks/Models_and_Weave_Integration_Demo)



Expand Down
56 changes: 28 additions & 28 deletions release-notes/server-releases.mdx

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions weave/cookbooks/Intro_to_Weave_Hello_Eval.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -122,5 +122,5 @@ await evaluation.evaluate(model)

## 🚀 Looking for more examples?

- Learn how to build an [evlauation pipeline end-to-end](https://weave-docs.wandb.ai/tutorial-eval).
- Learn how to evaluate a [RAG application by building](https://weave-docs.wandb.ai/tutorial-rag).
- Learn how to build an [evlauation pipeline end-to-end](/weave/tutorial-eval).
- Learn how to evaluate a [RAG application by building](/weave/tutorial-rag).
6 changes: 3 additions & 3 deletions weave/cookbooks/Intro_to_Weave_Hello_Trace.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ extract_fruit(sentence)
```

## 🚀 Looking for more examples?
- Check out the [Quickstart guide](https://weave-docs.wandb.ai/quickstart).
- Learn more about [advanced tracing topics](https://weave-docs.wandb.ai/tutorial-tracing_2).
- Learn more about [tracing in Weave](https://weave-docs.wandb.ai/guides/tracking/tracing)
- Check out the [Quickstart guide](/weave/quickstart).
- Learn more about [advanced tracing topics](/weave/tutorial-tracing_2).
- Learn more about [tracing in Weave](/weave/guides/tracking/tracing)

2 changes: 1 addition & 1 deletion weave/cookbooks/import_from_csv.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -278,4 +278,4 @@ weave.publish(dset)
![image.png](/images/screenshots/csv-4.png)
</Frame>

To learn more about evaluations, check out our [Quickstart](https://weave-docs.wandb.ai/tutorial-rag) on using your newly created dataset to evaluate your RAG application!
To learn more about evaluations, check out our [Quickstart](/weave/tutorial-rag) on using your newly created dataset to evaluate your RAG application!
2 changes: 1 addition & 1 deletion weave/cookbooks/multi-agent-structured-output.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ By using the new parameter `strict: true`, we are able to guarantee the response

The use of structured outputs in a multi-agent system enhances communication by ensuring consistent, easily processed data between agents. It also improves safety by allowing explicit refusals and boosts performance by eliminating the need for retries or validations. This simplifies interactions and increases overall system efficiency.

This tutorial demonstrates how we can utilize structured outputs in multi-agent system and trace them with [Weave](https://weave-docs.wandb.ai/).
This tutorial demonstrates how we can utilize structured outputs in multi-agent system and trace them with [Weave](/weave).

<Tip>
**Source**: This cookbook is based on [sample code from OpenAI's structured outputs](https://cookbook.openai.com/examples/structured_outputs_multi_agent), with some modifications added for improved visualization using Weave.
Expand Down
6 changes: 3 additions & 3 deletions weave/cookbooks/ocr-pipeline.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ Now, create a function called `named_entity_recognation` that:
- Passes the image data to the NER pipeline
- Returns correctly formatted JSON with the results

Use the [`@weave.op()` decorator](https://weave-docs.wandb.ai/reference/python-sdk/weave/trace/weave.trace.op) decorator to automatically track and trace function execution in the W&B UI.
Use the [`@weave.op()` decorator](/weave/reference/python-sdk/weave/trace/op) decorator to automatically track and trace function execution in the W&B UI.

Every `named_entity_recognation` is run, the full trace results are visible in the Weave UI. To view the traces, navigate to the **Traces** tab of your Weave project.

Expand Down Expand Up @@ -198,9 +198,9 @@ You will see something similar to the following in the **Traces** table in the W

## 4. Evaluate the pipeline using Weave

Now that you have created a pipeline to perform NER using a VLM, you can use Weave to systematically evaluate it and find out how well it performs. You can learn more about Evaluations in Weave in [Evaluations Overview](https://weave-docs.wandb.ai/guides/core-types/evaluations).
Now that you have created a pipeline to perform NER using a VLM, you can use Weave to systematically evaluate it and find out how well it performs. You can learn more about Evaluations in Weave in [Evaluations Overview](/weave/guides/core-types/evaluations).

A fundamental part of a Weave Evaluation are [Scorers](https://weave-docs.wandb.ai/guides/evaluation/scorers). Scorers are used to evaluate AI outputs and return evaluation metrics. They take the AI's output, analyze it, and return a dictionary of results. Scorers can use your input data as reference if needed and can also output extra information, such as explanations or reasonings from the evaluation.
A fundamental part of a Weave Evaluation are [Scorers](/weave/guides/evaluation/scorers). Scorers are used to evaluate AI outputs and return evaluation metrics. They take the AI's output, analyze it, and return a dictionary of results. Scorers can use your input data as reference if needed and can also output extra information, such as explanations or reasonings from the evaluation.

In this section, you will create two Scorers to evaluate the pipeline:
1. Programatic Scorer
Expand Down
12 changes: 6 additions & 6 deletions weave/cookbooks/online_monitoring.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ To follow along this tutorial you'll only need to install the following packages
First, we'll set up a function to initialize the Weave client and add costs for each model.

- We have included the standard costs for many standard models but we also make it easy to add your own custom costs and custom models. In the following we'll show how to add custom costs for a few models and use the standard costs for the rest.
- The costs are calculate based on the tracked tokens for each call in Weave. For many LLM vendor libraries, we will automatically track the token usage, but it is also possible to return custom token counts for any call. See this cookbook on how to define the token count and cost calculation for a custom model - [custom cost cookbook](https://weave-docs.wandb.ai/cookbooks/custom_model_cost#setting-up-a-model-with-weave).
- The costs are calculate based on the tracked tokens for each call in Weave. For many LLM vendor libraries, we will automatically track the token usage, but it is also possible to return custom token counts for any call. See this cookbook on how to define the token count and cost calculation for a custom model - [custom cost cookbook](/weave/cookbooks/custom_model_cost#setting-up-a-model-with-weave).

```python lines
PROJECT_NAME = "wandb-smle/weave-cookboook-demo"
Expand Down Expand Up @@ -86,7 +86,7 @@ In order to fetch call data from Weave, we have two options:
The first option to access data from Weave is to retrieve a list of filtered calls and extract the wanted data call-by-call. For that we can use the `calls_query_stream` API to fetch the calls data from Weave:

- `calls_query_stream` API: This API allows us to fetch the calls data from Weave.
- `filter` dictionary: This dictionary contains the filter parameters to fetch the calls data - see [here](https://weave-docs.wandb.ai/reference/python-sdk/weave/trace_server/weave.trace_server.trace_server_interface/#class-callschema) for more details.
- `filter` dictionary: This dictionary contains the filter parameters to fetch the calls data - see [here](/weave/reference/python-sdk/weave/trace_server/trace_server_interface#class-callschema) for more details.
- `expand_columns` list: This list contains the columns to expand in the calls data.
- `sort_by` list: This list contains the sorting parameters for the calls data.
- `include_costs` boolean: This boolean indicates whether to include the costs in the calls data.
Expand Down Expand Up @@ -231,11 +231,11 @@ plot_model_cost_distribution(df_costs)
In this cookbook, we demonstrated how to create a custom production monitoring dashboard using Weave's APIs and functions. Weave currently focuses on fast integrations for easy input of data as well as extraction of the data for custom processes.

- **Data Input:**
- Framework-agnostic tracing with [@weave-op()](https://weave-docs.wandb.ai/quickstart#2-log-a-trace-to-a-new-project) decorator and the possibility to import calls from CSV (see related [import cookbook](https://weave-docs.wandb.ai/cookbooks/import_from_csv))
- Service API endpoints to log to Weave from for various programming frameworks and languages, see [here](https://weave-docs.wandb.ai/reference/service-api/call-start-call-start-post) for more details.
- Framework-agnostic tracing with [@weave-op()](/weave/quickstart#2-log-a-trace-to-a-new-project) decorator and the possibility to import calls from CSV (see related [import cookbook](/weave/cookbooks/import_from_csv))
- Service API endpoints to log to Weave from for various programming frameworks and languages, see [here](/weave/reference/service-api/call-start-call-start-post) for more details.
- **Data Output:**
- Easy download of the data in CSV, TSV, JSONL, JSON formats - see [here](https://weave-docs.wandb.ai/guides/tracking/tracing#querying--exporting-calls) for more details.
- Easy export using programmatic access to the data - see "Use Python" section in the export panel as described in this cookbook. See [here](https://weave-docs.wandb.ai/guides/tracking/tracing#querying--exporting-calls) for more details.
- Easy download of the data in CSV, TSV, JSONL, JSON formats - see [here](/api-reference/calls/call-start) for more details.
- Easy export using programmatic access to the data - see "Use Python" section in the export panel as described in this cookbook. See [here](/weave/guides/tracking/tracing#querying-and-exporting-calls) for more details.

This custom dashboard extends Weave's native Traces view, allowing for tailored monitoring of LLM applications in production. If you're interested in viewing a more complex dashboard, check out a Streamlit example where you can add your own Weave project URL [in this repo](https://github.com/NiWaRe/agent-dev-collection).

2 changes: 1 addition & 1 deletion weave/cookbooks/pii.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ In this guide, you'll learn how to use W&B Weave while ensuring your Personally
2. **Microsoft's [Presidio](https://microsoft.github.io/presidio/)**, a python-based data protection SDK. This tool provides redaction and replacement functionalities.
3. **[Faker](https://faker.readthedocs.io/en/master/)**, a Python library to generate fake data, combined with Presidio to anonymize PII data.

Additionally, you'll learn how to use _`weave.op` input/output logging customization_ and _`autopatch_settings`_ to integrate PII redaction and anonymization into the workflow. For more information, see [Customize logged inputs and outputs](https://weave-docs.wandb.ai/guides/tracking/ops/#customize-logged-inputs-and-outputs).
Additionally, you'll learn how to use _`weave.op` input/output logging customization_ and _`autopatch_settings`_ to integrate PII redaction and anonymization into the workflow. For more information, see [Customize logged inputs and outputs](/weave/guides/tracking/ops/#customize-logged-inputs-and-outputs).

To get started, do the following:

Expand Down
6 changes: 3 additions & 3 deletions weave/cookbooks/weave_via_service_api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ Before beginning, complete the [prerequisites](#prerequisites-set-variables-and-

The following code sets the URL endpoints that will be used to access the Service API:

- [`https://trace.wandb.ai/call/start`](https://weave-docs.wandb.ai/reference/service-api/call-start-call-start-post)
- [`https://trace.wandb.ai/call/end`](https://weave-docs.wandb.ai/reference/service-api/call-end-call-end-post)
- [`https://trace.wandb.ai/calls/stream_query`](https://weave-docs.wandb.ai/reference/service-api/calls-query-stream-calls-stream-query-post)
- [`https://trace.wandb.ai/call/start`](/api-reference/calls/call-start)
- [`https://trace.wandb.ai/call/end`](/api-reference/calls/call-end)
- [`https://trace.wandb.ai/calls/stream_query`](/api-reference/calls/calls-query-stream)

Additionally, you must set the following variables:

Expand Down
2 changes: 1 addition & 1 deletion weave/guides/core-types/media.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -838,7 +838,7 @@ content = Content.from_bytes(video_bytes, mimetype='video/mp4')

### Content properties

For a comprehensive list of class attributes and methods, view the [Content reference docs](https://weave-docs.wandb.ai/reference/python-sdk/weave/#class-content)
For a comprehensive list of class attributes and methods, view the [Content reference docs](/weave/reference/python-sdk/weave#class-content)

#### Attributes

Expand Down
2 changes: 1 addition & 1 deletion weave/guides/integrations/langchain.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ print(output)

## Tracking Call Metadata

To track metadata from your LangChain calls, you can use the [`weave.attributes`](https://weave-docs.wandb.ai/reference/python-sdk/weave/#function-attributes) context manager. This context manager allows you to set custom metadata for a specific block of code, such as a chain or a single request.
To track metadata from your LangChain calls, you can use the [`weave.attributes`](/weave/reference/python-sdk/weave#function-attributes) context manager. This context manager allows you to set custom metadata for a specific block of code, such as a chain or a single request.

```python lines
import weave
Expand Down
2 changes: 1 addition & 1 deletion weave/guides/integrations/verdict.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ print(output)

## Tracking Call Metadata

To track metadata from your Verdict pipeline calls, you can use the [`weave.attributes`](https://weave-docs.wandb.ai/reference/python-sdk/weave/#function-attributes) context manager. This context manager allows you to set custom metadata for a specific block of code, such as a pipeline run or evaluation batch.
To track metadata from your Verdict pipeline calls, you can use the [`weave.attributes`](/weave/reference/python-sdk/weave#function-attributes) context manager. This context manager allows you to set custom metadata for a specific block of code, such as a pipeline run or evaluation batch.

```python lines
import weave
Expand Down
2 changes: 1 addition & 1 deletion weave/guides/tools/evaluation_playground.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ It is also important to appropriately name the columns in your dataset `user_inp

### Add a model

[Models](https://weave-docs.wandb.ai/guides/core-types/models), in the context of Weave, are a combination of an AI model (such as GPT) and the environment (in this case the system prompt) that defines how the model operates during the evaluation. You can select existing models in your project or create new ones to evaluate, and you can add multiple models at once to evaluate them simultaneously with the same dataset and scorer. **You can only use models created using the playground feature.**
[Models](/weave/guides/core-types/models), in the context of Weave, are a combination of an AI model (such as GPT) and the environment (in this case the system prompt) that defines how the model operates during the evaluation. You can select existing models in your project or create new ones to evaluate, and you can add multiple models at once to evaluate them simultaneously with the same dataset and scorer. **You can only use models created using the playground feature.**

To add a model in the **Models** section of the Evaluation Playground:

Expand Down
6 changes: 3 additions & 3 deletions weave/guides/tracking/costs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ description: "Track and manage costs for LLM operations in Weave"

<Tabs>
<Tab title="Python">
You can add a custom cost by using the [`add_cost`](/weave/reference/python-sdk/weave/trace/weave_client.mdx#add_cost) method.
You can add a custom cost by using the [`add_cost`](/weave/reference/python-sdk/weave/trace/weave_client#method-add-cost) method.
The three required fields are `llm_id`, `prompt_token_cost`, and `completion_token_cost`.
`llm_id` is the name of the LLM (e.g. `gpt-4o`). `prompt_token_cost` and `completion_token_cost` are cost per token for the LLM (if the LLM prices were specified inper million tokens, make sure to convert the value).
You can also set `effective_date` to a datetime, to make the cost effective at a specific date, this defaults to the current date.
Expand Down Expand Up @@ -47,7 +47,7 @@ description: "Track and manage costs for LLM operations in Weave"

<Tabs>
<Tab title="Python">
You can query for costs by using the [`query_costs`](/weave/reference/python-sdk/weave/trace/weave_client.mdx#query_costs) method.
You can query for costs by using the [`query_costs`](/weave/reference/python-sdk/weave/trace/weave_client#method-query-costs) method.
There are a few ways to query for costs, you can pass in a singular cost id, or a list of LLM model names.

```python lines
Expand All @@ -74,7 +74,7 @@ description: "Track and manage costs for LLM operations in Weave"

<Tabs>
<Tab title="Python">
You can purge a custom cost by using the [`purge_costs`](/weave/reference/python-sdk/weave/trace/weave_client.mdx#purge_costs) method. You pass in a list of cost ids, and the costs with those ids are purged.
You can purge a custom cost by using the [`purge_costs`](/weave/reference/python-sdk/weave/trace/weave_client#method-purge-costs) method. You pass in a list of cost ids, and the costs with those ids are purged.

```python lines
import weave
Expand Down
2 changes: 1 addition & 1 deletion weave/guides/tracking/tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,7 @@ You can perform all of these mutations from the UI by navigating to the call det

<Tabs>
<Tab title="Python">
In order to set the display name of a call, you can use the [`Call.set_display_name`](/weave/reference/python-sdk/weave/trace/weave_client#method-set_display_name) method.
In order to set the display name of a call, you can use the [`Call.set_display_name`](/weave/reference/python-sdk/weave/trace/weave_client#method-set-display-name) method.

```python lines lines
import weave
Expand Down