diff --git a/docs/blog/.authors.yml b/docs/blog/.authors.yml index 2441ac862..34e8d7f07 100644 --- a/docs/blog/.authors.yml +++ b/docs/blog/.authors.yml @@ -2,7 +2,7 @@ authors: jxnl: name: Jason Liu description: Creator - avatar: https://pbs.twimg.com/profile_images/1682267111555571714/VDORoUy__400x400.jpg + avatar: https://pbs.twimg.com/profile_images/1724672723748638720/qOBwmkOI_400x400.jpg ivanleomk: name: Ivan Leo description: Contributor diff --git a/docs/blog/index.md b/docs/blog/index.md index 5058819f0..3c26d98a7 100644 --- a/docs/blog/index.md +++ b/docs/blog/index.md @@ -4,9 +4,10 @@ The goal of the blog is to capture some content that does not neatly fit within ## Advanced Topics -- [Query Understanding and Expansion for RAG](posts/rag-and-beyond.md) -- [GPT-4 Level summarization with GPT3.5 Finetuning](posts/chain-of-density.md) -- [Deepdive on LLM Guardrails / Validation](posts/validation-part1.md) +- [Query Understanding for RAG: Beyond Embeddings](posts/rag-and-beyond.md) +- [Finetuning: GPT-4 level summaries with GPT-3.5-turbo](posts/chain-of-density.md) +- [Introduction to Guardrails and Validation](posts/validation-part1.md) +- [Validating Citations](posts/citations.md) - [A Guide to Fine-Tuning and Distillation](posts/distilation-part1.md) ## Learning Python diff --git a/docs/blog/posts/citations.md b/docs/blog/posts/citations.md new file mode 100644 index 000000000..240c37a44 --- /dev/null +++ b/docs/blog/posts/citations.md @@ -0,0 +1,268 @@ +--- +draft: False +date: 2023-11-18 +slug: validate-citations +tags: + - pydantic + - validation + - finetuneing + - citations + - hallucination +authors: + - jxnl +--- + +# Verifying LLM Citations with Pydantic + +Ensuring the accuracy of information is crucial. This blog post explores how Pydantic's powerful and flexible validators can enhance data accuracy through citation verification. + +We'll start with using a simple substring check to verify citations. Then we'll use `instructor` itself to power an LLM to verify citations and align answers with the given citations. Finally, we'll explore how we can use these techniques to generate a dataset of accurate responses. + +## Example 1: Simple Substring Check + +In this example, we use the `Statements` class to verify if a given substring quote exists within a text chunk. If the substring is not found, an error is raised. + +### Code Example: + +```python +from typing import List, Optional +from openai import OpenAI +from pydantic import BaseModel, Field, ValidationError, ValidationInfo, field_validator, model_validator +import instructor + +client = instructor.patch(OpenAI()) + +class Statements(BaseModel): + body: str + substring_quote: str + + @field_validator("substring_quote") + @classmethod + def substring_quote_exists(cls, v: str, info: ValidationInfo): + context = info.context.get("text_chunks", None) + + for text_chunk in context.values(): + if v in text_chunk: # (1) + return v + raise ValueError("Could not find substring_quote `{v}` in contexts") + + +class AnswerWithCitaton(BaseModel): + question: str + answer: List[Statements] +``` + +1. While we use a simple substring check in this example, we can use more complex techniques like regex or Levenshtein distance. + +Once the class is defined, we can use it to validate the context and raise an error if the substring is not found. + +```python +try: + AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Paris", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is not the capital of France", + 3: "Irrelevant data", + } + }, + ) +except ValidationError as e: + print(e) +``` + +### Error Message Example: + +``` +answer.0.substring_quote + Value error, Could not find substring_quote `Paris is the capital of France` in contexts [type=value_error, input_value='Paris is the capital of France', input_type=str] + For further information visit [https://errors.pydantic.dev/2.4/v/value_error](https://errors.pydantic.dev/2.4/v/value_error) +``` + +Pydantic raises a validation error when the `substring_quote` attribute does not exist in the context. This approach can be used to validate more complex data using techniques like regex or Levenshtein distance. + +## Example 2: Using LLM for Verification + +This approach leverages OpenAI's LLM to validate citations. If the citation does not exist in the context, the LLM returns an error message. + +### Code Example: + +```python +class Validation(BaseModel): + is_valid: bool + error_messages: Optional[str] = Field(None, description="Error messages if any") + + +class Statements(BaseModel): + body: str + substring_quote: str + + @model_validator(mode="after") + def substring_quote_exists(self, info: ValidationInfo): + context = info.context.get("text_chunks", None) + + resp: Validation = client.chat.completions.create( + response_model=Validation, + messages=[ + { + "role": "user", + "content": f"Does the following citation exist in the following context?\n\nCitation: {self.substring_quote}\n\nContext: {context}", + } + ], + model="gpt-3.5-turbo", + ) + + if resp.is_valid: + return self + + raise ValueError(resp.error_messages) + + +class AnswerWithCitaton(BaseModel): + question: str + answer: List[Statements] +``` + +Now when we use a correct citation, the LLM returns a valid response. + +```python +resp = AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Paris", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is the capital of France", + 3: "Irrelevant data", + } + }, +) +print(resp.model_dump_json(indent=2)) +``` + +### Result: + +```json +{ + "question": "What is the capital of France?", + "answer": [ + { + "body": "Paris", + "substring_quote": "Paris is the capital of France" + } + ] +} +``` + +When we have citations that don't exist in the context, the LLM returns an error message. + +```python +try: + AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Paris", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is not the capital of France", + 3: "Irrelevant data", + } + }, + ) +except ValidationError as e: + print(e) +``` + +### Error Message Example: + +``` +1 validation error for AnswerWithCitaton +answer.0 + Value error, Citation not found in context [type=value_error, input_value={'body': 'Paris', 'substr... the capital of France'}, input_type=dict] + For further information visit [https://errors.pydantic.dev/2.4/v/value_error](https://errors.pydantic.dev/2.4/v/value_error) +``` + +## Example 3: Aligning Citations and Answers + +In this example, we ensure that the provided answers are aligned with the given citations and context. The LLM is used to verify the alignment. + +We use the same `Statements` model as above, but we add a new model for the answer that also verifies the alignment of citations. + +### Code Example: + +```python +class AnswerWithCitaton(BaseModel): + question: str + answer: List[Statements] + + @model_validator(mode="after") + def validate_answer(self, info: ValidationInfo): + context = info.context.get("text_chunks", None) + + resp: Validation = client.chat.completions.create( + response_model=Validation, + messages=[ + { + "role": "user", + "content": f"Does the following answers match the question and the context?\n\nQuestion: {self.question}\n\nAnswer: {self.answer}\n\nContext: {context}", + } + ], + model="gpt-3.5-turbo", + ) + + if resp.is_valid: + return self + + raise ValueError(resp.error_messages) +``` + +When we have a mismatch between the answer and the citation, the LLM returns an error message. + +```python +try: + AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Texas", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is the capital of France", + 3: "Irrelevant data", + } + }, + ) +except ValidationError as e: + print(e) +``` + +### Error Message Example: + +``` +1 validation error for AnswerWithCitaton + Value error, The answer does not match the question and context [type=value_error, input_value={'question': 'What is the...he capital of France'}]}, input_type=dict] + For further information visit [https://errors.pydantic.dev/2.4/v/value_error](https://errors.pydantic.dev/2.4/v/value_error) +``` + +## Conclusion + +These examples demonstrate the potential of using Pydantic and OpenAI to enhance data accuracy through citation verification. While the LLM-based approach may not be efficient for runtime operations, it has exciting implications for generating a dataset of accurate responses. By leveraging this method during data generation, we can fine-tune a model that excels in citation accuracy. Similar to our last post on [finetuning a better summarizer](https://jxnl.github.io/instructor/blog/2023/11/05/chain-of-density/). + +If you like the content check out our [GitHub](https://github.com/jxnl/instructor) as give us a start and checkout the library. diff --git a/examples/citations/run.py b/examples/citations/run.py new file mode 100644 index 000000000..c8af6994e --- /dev/null +++ b/examples/citations/run.py @@ -0,0 +1,225 @@ +from typing import List, Optional +from openai import OpenAI +from pydantic import ( + BaseModel, + Field, + ValidationError, + ValidationInfo, + field_validator, + model_validator, +) + +import instructor + +client = instructor.patch(OpenAI()) + +""" +Example 1) Simple Substring check that compares a citation to a text chunk +""" + + +class Statements(BaseModel): + body: str + substring_quote: str + + @field_validator("substring_quote") + @classmethod + def substring_quote_exists(cls, v: str, info: ValidationInfo): + context = info.context.get("text_chunks", None) + + # Check if the substring_quote is in the text_chunk + # if not, raise an error + for text_chunk in context.values(): + if v in text_chunk: + return v + raise ValueError( + f"Could not find substring_quote `{v}` in contexts", + ) + + +class AnswerWithCitaton(BaseModel): + question: str + answer: List[Statements] + + +try: + AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Paris", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is not the capital of France", + 3: "Irrelevant data", + } + }, + ) +except ValidationError as e: + print(e) +""" +answer.0.substring_quote + Value error, Could not find substring_quote `Paris is the capital of France` in contexts [type=value_error, input_value='Paris is the capital of France', input_type=str] + For further information visit https://errors.pydantic.dev/2.4/v/value_error +""" + + +""" +Example 2) Using an LLM to verify if a +""" + + +class Validation(BaseModel): + """ + Verfication response from the LLM, + the error message should be detailed if the is_valid is False + but keep it to less than 100 characters, reference specific + attributes that you are comparing, use `...` is the string is too long + """ + + is_valid: bool + error_messages: Optional[str] = Field(None, description="Error messages if any") + + +class Statements(BaseModel): + body: str + substring_quote: str + + @model_validator(mode="after") + def substring_quote_exists(self, info: ValidationInfo): + context = info.context.get("text_chunks", None) + + resp: Validation = client.chat.completions.create( + response_model=Validation, + messages=[ + { + "role": "user", + "content": f"Does the following citation exist in the following context?\n\nCitation: {self.substring_quote}\n\nContext: {context}", + } + ], + model="gpt-3.5-turbo", + ) + + if resp.is_valid: + return self + + raise ValueError(resp.error_messages) + + +class AnswerWithCitaton(BaseModel): + question: str + answer: List[Statements] + + +resp = AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Paris", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is the capital of France", + 3: "Irrelevant data", + } + }, +) +# output: notice that there are no errors +print(resp.model_dump_json(indent=2)) +{ + "question": "What is the capital of France?", + "answer": [{"body": "Paris", "substring_quote": "Paris is the capital of France"}], +} + +# Now we change the text chunk to something else, and we get an error +try: + AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Paris", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is not the capital of France", + 3: "Irrelevant data", + } + }, + ) +except ValidationError as e: + print(e) +""" +1 validation error for AnswerWithCitaton +answer.0 + Value error, Citation not found in context [type=value_error, input_value={'body': 'Paris', 'substr... the capital of France'}, input_type=dict] + For further information visit https://errors.pydantic.dev/2.4/v/value_error +""" + +# Example 3) Using an LLM to verify if the citations and the answers are all aligned + + +# we keep the same model as above for Statements, but we add a new model for the answer +# that also verifies that the citations are aligned with the answers +class AnswerWithCitaton(BaseModel): + question: str + answer: List[Statements] + + @model_validator(mode="after") + def validate_answer(self, info: ValidationInfo): + context = info.context.get("text_chunks", None) + + resp: Validation = client.chat.completions.create( + response_model=Validation, + messages=[ + { + "role": "user", + "content": f"Does the following answers match the question and the context?\n\nQuestion: {self.question}\n\nAnswer: {self.answer}\n\nContext: {context}", + } + ], + model="gpt-3.5-turbo", + ) + + if resp.is_valid: + return self + + raise ValueError(resp.error_messages) + + +""" +Using LLMs for citation verification is inefficient during runtime. +However, we can utilize them to create a dataset consisting only of accurate responses +where citations must be valid (as determined by LLM, fuzzy text search, etc.). + +This approach would require an initial investment during data generation to obtain +a finely-tuned model for improved citation. +""" +try: + AnswerWithCitaton.model_validate( + { + "question": "What is the capital of France?", + "answer": [ + {"body": "Texas", "substring_quote": "Paris is the capital of France"}, + ], + }, + context={ + "text_chunks": { + 1: "Jason is a pirate", + 2: "Paris is the capital of France", + 3: "Irrelevant data", + } + }, + ) +except ValidationError as e: + print(e) +""" +1 validation error for AnswerWithCitaton + Value error, The answer does not match the question and context [type=value_error, input_value={'question': 'What is the...he capital of France'}]}, input_type=dict] + For further information visit https://errors.pydantic.dev/2.4/v/value_error +""" diff --git a/instructor/__init__.py b/instructor/__init__.py index b28ce7903..4bd40d7a1 100644 --- a/instructor/__init__.py +++ b/instructor/__init__.py @@ -1,7 +1,6 @@ from .distil import FinetuneFormat, Instructions from .dsl import CitationMixin, Maybe, MultiTask, llm_validator from .function_calls import OpenAISchema, openai_function, openai_schema -from .dsl import MultiTask, Maybe, llm_validator, CitationMixin from .patch import patch, apatch __all__ = [ diff --git a/tutorials/1.introduction.ipynb b/tutorials/1.introduction.ipynb index d178beadd..6053d3f32 100644 --- a/tutorials/1.introduction.ipynb +++ b/tutorials/1.introduction.ipynb @@ -24,7 +24,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 22, "metadata": {}, "outputs": [], "source": [ @@ -43,7 +43,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 23, "metadata": {}, "outputs": [ { @@ -62,7 +62,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/home/m/cookbook/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", "\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str" ] } @@ -93,7 +93,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 24, "metadata": {}, "outputs": [ { @@ -102,7 +102,7 @@ "Person(name='Sam', age=30)" ] }, - "execution_count": 2, + "execution_count": 24, "metadata": {}, "output_type": "execute_result" } @@ -121,7 +121,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 25, "metadata": {}, "outputs": [ { @@ -130,7 +130,7 @@ "Person(name='Sam', age=30)" ] }, - "execution_count": 4, + "execution_count": 25, "metadata": {}, "output_type": "execute_result" } @@ -143,7 +143,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 26, "metadata": {}, "outputs": [ { @@ -153,7 +153,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/home/m/cookbook/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", "\u001b[0;31mAssertionError\u001b[0m: " ] } @@ -165,7 +165,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 27, "metadata": {}, "outputs": [ { @@ -175,7 +175,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39m# Data is validated to get better error messages\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m person \u001b[39m=\u001b[39m Person\u001b[39m.\u001b[39;49mmodel_validate({\u001b[39m\"\u001b[39;49m\u001b[39mname\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mage\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39m30.2\u001b[39;49m\u001b[39m\"\u001b[39;49m})\n\u001b[1;32m 3\u001b[0m person\n", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39m# Data is validated to get better error messages\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m person \u001b[39m=\u001b[39m Person\u001b[39m.\u001b[39;49mmodel_validate({\u001b[39m\"\u001b[39;49m\u001b[39mname\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mage\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39m30.2\u001b[39;49m\u001b[39m\"\u001b[39;49m})\n\u001b[1;32m 3\u001b[0m person\n", "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:503\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m 501\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 502\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 503\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(\n\u001b[1;32m 504\u001b[0m obj, strict\u001b[39m=\u001b[39;49mstrict, from_attributes\u001b[39m=\u001b[39;49mfrom_attributes, context\u001b[39m=\u001b[39;49mcontext\n\u001b[1;32m 505\u001b[0m )\n", "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Person\nage\n Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/int_parsing" ] @@ -191,32 +191,144 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "By introducing pydantic into any python codebase you can get a lot of benefits. You can get type checking, you can get validation, and you can get autocomplete. This is a huge win, because it means you can catch errors before they happen. This is even more useful when we rely on language models to generate data for us." + "By introducing pydantic into any python codebase you can get a lot of benefits. You can get type checking, you can get validation, and you can get autocomplete. This is a huge win, because it means you can catch errors before they happen. This is even more useful when we rely on language models to generate data for us.\n", + "\n", + "You can also define validators that are run on the data. This is useful because it means you can catch errors before they happen. For example, you can define a validator that checks if the age is greater than 0. This is useful because it means you can catch errors before they happen." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Person(name='Sam', age=-10)" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "Person(name=\"Sam\", age=-10)" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "ename": "ValidationError", + "evalue": "1 validation error for Person\nage\n Input should be greater than 0 [type=greater_than, input_value=-10, input_type=int]\n For further information visit https://errors.pydantic.dev/2.4/v/greater_than", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 14\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 2\u001b[0m name: \u001b[39mstr\u001b[39m\n\u001b[1;32m 3\u001b[0m age: \u001b[39mint\u001b[39m \u001b[39m=\u001b[39m Field(\u001b[39m.\u001b[39m\u001b[39m.\u001b[39m\u001b[39m.\u001b[39m, gt\u001b[39m=\u001b[39m\u001b[39m0\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m Person(name\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, age\u001b[39m=\u001b[39;49m\u001b[39m-\u001b[39;49m\u001b[39m10\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Person\nage\n Input should be greater than 0 [type=greater_than, input_value=-10, input_type=int]\n For further information visit https://errors.pydantic.dev/2.4/v/greater_than" + ] + } + ], + "source": [ + "class Person(BaseModel):\n", + " name: str\n", + " age: int = Field(..., gt=0)\n", + "\n", + "Person(name=\"Sam\", age=-10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Asking for JSON from OpenAI" + "Lastly you can also define functions that run on the data." ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "ename": "ValidationError", + "evalue": "1 validation error for Person\nname\n Value error, must contain a space [type=value_error, input_value='Sam', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 16\u001b[0m line \u001b[0;36m1\n\u001b[1;32m 11\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\u001b[39m\"\u001b[39m\u001b[39mmust contain a space\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 12\u001b[0m \u001b[39mreturn\u001b[39;00m v\n\u001b[0;32m---> 14\u001b[0m Person(name\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, age\u001b[39m=\u001b[39;49m\u001b[39m10\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Person\nname\n Value error, must contain a space [type=value_error, input_value='Sam', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" + ] + } + ], + "source": [ + "from pydantic import field_validator\n", + "\n", + "\n", + "class Person(BaseModel):\n", + " name: str\n", + " age: int = Field(..., gt=0)\n", + "\n", + " @field_validator(\"name\")\n", + " def name_must_contain_space(cls, v):\n", + " if \" \" not in v:\n", + " raise ValueError(\"must contain a space\")\n", + " return v\n", + " \n", + "Person(name=\"Sam\", age=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "Person(name='Jason', age=25)" + "Person(name='Sam Liu', age=10)" ] }, - "execution_count": 3, + "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], + "source": [ + "Person(name=\"Sam Liu\", age=10)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Asking for JSON from OpenAI" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [ + { + "ename": "ValidationError", + "evalue": "1 validation error for Person\nname\n Value error, must contain a space [type=value_error, input_value='Jason', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 19\u001b[0m line \u001b[0;36m1\n\u001b[1;32m 3\u001b[0m client \u001b[39m=\u001b[39m OpenAI()\n\u001b[1;32m 5\u001b[0m resp \u001b[39m=\u001b[39m client\u001b[39m.\u001b[39mchat\u001b[39m.\u001b[39mcompletions\u001b[39m.\u001b[39mcreate(\n\u001b[1;32m 6\u001b[0m model\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mgpt-3.5-turbo\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m 7\u001b[0m messages\u001b[39m=\u001b[39m[\n\u001b[1;32m 8\u001b[0m {\u001b[39m\"\u001b[39m\u001b[39mrole\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39m\"\u001b[39m\u001b[39muser\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m\"\u001b[39m\u001b[39mcontent\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39m\"\u001b[39m\u001b[39mExtract `Jason is 25 years old` into json\u001b[39m\u001b[39m\"\u001b[39m},\n\u001b[1;32m 9\u001b[0m ]\n\u001b[1;32m 10\u001b[0m )\n\u001b[0;32m---> 12\u001b[0m Person\u001b[39m.\u001b[39;49mmodel_validate_json(resp\u001b[39m.\u001b[39;49mchoices[\u001b[39m0\u001b[39;49m]\u001b[39m.\u001b[39;49mmessage\u001b[39m.\u001b[39;49mcontent)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:530\u001b[0m, in \u001b[0;36mBaseModel.model_validate_json\u001b[0;34m(cls, json_data, strict, context)\u001b[0m\n\u001b[1;32m 528\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 529\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 530\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_json(json_data, strict\u001b[39m=\u001b[39;49mstrict, context\u001b[39m=\u001b[39;49mcontext)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Person\nname\n Value error, must contain a space [type=value_error, input_value='Jason', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" + ] + } + ], "source": [ "from openai import OpenAI\n", "\n", @@ -236,18 +348,7 @@ "cell_type": "code", "execution_count": null, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "Person(name='Jason Liu', age=30)" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", @@ -268,32 +369,9 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"name\": \"Jason Liu\",\n", - " \"age\": 30,\n", - " \"birthday\": \"2023-11-17\"\n", - "}\n", - "name='Jason Liu' age=30\n" - ] - }, - { - "data": { - "text/plain": [ - "PersonBirthday(name='Jason Liu', age=30, birthday=datetime.date(2023, 11, 17))" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "import datetime\n", "\n", @@ -329,19 +407,18 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 33, "metadata": {}, "outputs": [ { - "ename": "ValidationError", - "evalue": "1 validation error for PersonBirthday\nbirthday\n Input should be a valid date or datetime, input is too short [type=date_from_datetime_parsing, input_value='yesterday', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/date_from_datetime_parsing", + "ename": "NameError", + "evalue": "name 'datetime' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 19\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m schema \u001b[39m=\u001b[39m {\n\u001b[1;32m 2\u001b[0m \u001b[39m'\u001b[39m\u001b[39mproperties\u001b[39m\u001b[39m'\u001b[39m: \n\u001b[1;32m 3\u001b[0m {\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[39m'\u001b[39m\u001b[39mtype\u001b[39m\u001b[39m'\u001b[39m: \u001b[39m'\u001b[39m\u001b[39mobject\u001b[39m\u001b[39m'\u001b[39m\n\u001b[1;32m 10\u001b[0m }\n\u001b[1;32m 12\u001b[0m resp \u001b[39m=\u001b[39m client\u001b[39m.\u001b[39mchat\u001b[39m.\u001b[39mcompletions\u001b[39m.\u001b[39mcreate(\n\u001b[1;32m 13\u001b[0m model\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mgpt-3.5-turbo\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m 14\u001b[0m messages\u001b[39m=\u001b[39m[\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 18\u001b[0m function_call\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mauto\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 19\u001b[0m )\n\u001b[0;32m---> 22\u001b[0m PersonBirthday\u001b[39m.\u001b[39;49mmodel_validate_json(resp\u001b[39m.\u001b[39;49mchoices[\u001b[39m0\u001b[39;49m]\u001b[39m.\u001b[39;49mmessage\u001b[39m.\u001b[39;49mfunction_call\u001b[39m.\u001b[39;49marguments)\n", - "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:530\u001b[0m, in \u001b[0;36mBaseModel.model_validate_json\u001b[0;34m(cls, json_data, strict, context)\u001b[0m\n\u001b[1;32m 528\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 529\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 530\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_json(json_data, strict\u001b[39m=\u001b[39;49mstrict, context\u001b[39m=\u001b[39;49mcontext)\n", - "\u001b[0;31mValidationError\u001b[0m: 1 validation error for PersonBirthday\nbirthday\n Input should be a valid date or datetime, input is too short [type=date_from_datetime_parsing, input_value='yesterday', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/date_from_datetime_parsing" + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 24\u001b[0m line \u001b[0;36m1\n\u001b[1;32m 1\u001b[0m schema \u001b[39m=\u001b[39m {\n\u001b[1;32m 2\u001b[0m \u001b[39m'\u001b[39m\u001b[39mproperties\u001b[39m\u001b[39m'\u001b[39m: \n\u001b[1;32m 3\u001b[0m {\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[39m'\u001b[39m\u001b[39mtype\u001b[39m\u001b[39m'\u001b[39m: \u001b[39m'\u001b[39m\u001b[39mobject\u001b[39m\u001b[39m'\u001b[39m\n\u001b[1;32m 10\u001b[0m }\n\u001b[1;32m 12\u001b[0m resp \u001b[39m=\u001b[39m client\u001b[39m.\u001b[39mchat\u001b[39m.\u001b[39mcompletions\u001b[39m.\u001b[39mcreate(\n\u001b[1;32m 13\u001b[0m model\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mgpt-3.5-turbo\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m 14\u001b[0m messages\u001b[39m=\u001b[39m[\n\u001b[0;32m---> 15\u001b[0m {\u001b[39m\"\u001b[39m\u001b[39mrole\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39m\"\u001b[39m\u001b[39muser\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m\"\u001b[39m\u001b[39mcontent\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mExtract `Jason Liu is thirty years old his birthday is yesturday` into json today is \u001b[39m\u001b[39m{\u001b[39;00mdatetime\u001b[39m.\u001b[39mdate\u001b[39m.\u001b[39mtoday()\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m},\n\u001b[1;32m 16\u001b[0m ],\n\u001b[1;32m 17\u001b[0m functions\u001b[39m=\u001b[39m[{\u001b[39m\"\u001b[39m\u001b[39mname\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39m\"\u001b[39m\u001b[39mPerson\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m\"\u001b[39m\u001b[39mparameters\u001b[39m\u001b[39m\"\u001b[39m: schema}],\n\u001b[1;32m 18\u001b[0m function_call\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mauto\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 19\u001b[0m )\n\u001b[1;32m 22\u001b[0m PersonBirthday\u001b[39m.\u001b[39mmodel_validate_json(resp\u001b[39m.\u001b[39mchoices[\u001b[39m0\u001b[39m]\u001b[39m.\u001b[39mmessage\u001b[39m.\u001b[39mfunction_call\u001b[39m.\u001b[39marguments)\n", + "\u001b[0;31mNameError\u001b[0m: name 'datetime' is not defined" ] } ], @@ -379,23 +456,19 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 34, "metadata": {}, "outputs": [ { - "data": { - "text/plain": [ - "{'properties': {'name': {'title': 'Name', 'type': 'string'},\n", - " 'age': {'title': 'Age', 'type': 'integer'},\n", - " 'birthday': {'format': 'date', 'title': 'Birthday', 'type': 'string'}},\n", - " 'required': ['name', 'age', 'birthday'],\n", - " 'title': 'PersonBirthday',\n", - " 'type': 'object'}" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" + "ename": "NameError", + "evalue": "name 'PersonBirthday' is not defined", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 26\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> 1\u001b[0m PersonBirthday\u001b[39m.\u001b[39mmodel_json_schema()\n", + "\u001b[0;31mNameError\u001b[0m: name 'PersonBirthday' is not defined" + ] } ], "source": [ @@ -411,7 +484,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 35, "metadata": {}, "outputs": [ { @@ -427,14 +500,14 @@ " 'type': 'object'}},\n", " 'description': 'A Person with an address',\n", " 'properties': {'name': {'title': 'Name', 'type': 'string'},\n", - " 'age': {'title': 'Age', 'type': 'integer'},\n", + " 'age': {'exclusiveMinimum': 0, 'title': 'Age', 'type': 'integer'},\n", " 'address': {'$ref': '#/$defs/Address'}},\n", " 'required': ['name', 'age', 'address'],\n", " 'title': 'PersonAddress',\n", " 'type': 'object'}" ] }, - "execution_count": 13, + "execution_count": 35, "metadata": {}, "output_type": "execute_result" } @@ -464,7 +537,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 37, "metadata": {}, "outputs": [ { @@ -473,13 +546,14 @@ "PersonAddress(name='Jason Liu', age=30, address=Address(address='123 Main St', city='San Francisco', state='CA'))" ] }, - "execution_count": 14, + "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import instructor\n", + "import datetime\n", "\n", "client = instructor.patch(client)\n", "\n", @@ -516,7 +590,14 @@ "\n", "More importantly, we've also added straight forward validation and reasking to the mix.\n", "\n", - "The goal of instructor is to show you how to think about structured prompting and provide examples and documentation that you can take with you to any framework." + "The goal of instructor is to show you how to think about structured prompting and provide examples and documentation that you can take with you to any framework.\n", + "\n", + "\n", + "- [Marvin](https://www.askmarvin.ai/)\n", + "- [Langchain](https://python.langchain.com/docs/modules/model_io/output_parsers/pydantic)\n", + "- [LlamaIndex](https://gpt-index.readthedocs.io/en/latest/examples/output_parsing/openai_pydantic_program.html)\n", + "\n", + "The main difference between these libraries is that they all have different approaches to how they do it. With instructor the goal is to be as light weight as possible, get you as close as possible to the openai api, and then get out of your way." ] } ], @@ -536,7 +617,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.12" + "version": "3.11.6" } }, "nbformat": 4, diff --git a/tutorials/2.tips.ipynb b/tutorials/2.tips.ipynb index 7abb79ead..7fadceb9e 100644 --- a/tutorials/2.tips.ipynb +++ b/tutorials/2.tips.ipynb @@ -32,7 +32,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 5, "id": "fdf5e1d9-31ad-4e8a-a55e-e2e70fff598d", "metadata": {}, "outputs": [ @@ -42,15 +42,16 @@ "{'age': 17, 'name': 'Harry Potter', 'house': }" ] }, - "execution_count": 9, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from enum import Enum\n", - "from pydantic import BaseModel,Field\n", - "from typing import List,Literal\n", + "from pydantic import BaseModel, Field\n", + "from typing_extensions import Literal\n", + "\n", "import instructor\n", "from openai import OpenAI\n", "\n", @@ -83,7 +84,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 6, "id": "03db160c-81e9-4373-bfec-7a107224b6dd", "metadata": {}, "outputs": [ @@ -93,7 +94,7 @@ "{'age': 17, 'name': 'Harry Potter', 'house': 'Gryffindor'}" ] }, - "execution_count": 10, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -129,7 +130,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 7, "id": "0e7938b8-4666-4df4-bd80-f53e8baf7550", "metadata": {}, "outputs": [ @@ -139,13 +140,13 @@ "{'age': 38,\n", " 'name': 'Severus Snape',\n", " 'house': 'Slytherin',\n", - " 'properties': [{'key': 'role', 'value': 'Professor of Potions'},\n", - " {'key': 'loyalty', 'value': 'Dumbledore'},\n", + " 'properties': [{'key': 'position', 'value': 'Professor of Potions'},\n", + " {'key': 'loyalty', 'value': 'Dumbledore, Hogwarts'},\n", " {'key': 'patronus', 'value': 'Doe'},\n", - " {'key': 'played_by', 'value': 'Alan Rickman'}]}" + " {'key': 'skill', 'value': 'Occlumency'}]}" ] }, - "execution_count": 4, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -190,7 +191,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 8, "id": "69a58d01-ab6f-41b6-bc0c-b0e55fdb6fe4", "metadata": {}, "outputs": [ @@ -202,18 +203,18 @@ " 'house': 'Slytherin',\n", " 'properties': [{'index': '1',\n", " 'key': 'Occupation',\n", - " 'value': 'Potions Master, Defense Against the Dark Arts Professor, Headmaster'},\n", + " 'value': 'Professor of Potions and later Defence Against the Dark Arts'},\n", " {'index': '2',\n", - " 'key': 'Loyalty',\n", - " 'value': 'Hogwarts, Order of the Phoenix, Albus Dumbledore, Lily Potter'},\n", - " {'index': '3',\n", + " 'key': 'Allegiance',\n", + " 'value': 'Order of the Phoenix, Hogwarts'},\n", + " {'index': '3', 'key': 'Patronus', 'value': 'Doe'},\n", + " {'index': '4',\n", " 'key': 'Skills',\n", - " 'value': 'Potions, Occlumency, Legilimency, Spell creation'},\n", - " {'index': '4', 'key': 'Patronus', 'value': 'Doe'},\n", - " {'index': '5', 'key': 'Actor', 'value': 'Alan Rickman'}]}" + " 'value': 'Potions master, Occlumens, Legilimens'},\n", + " {'index': '5', 'key': 'Portrayed by', 'value': 'Alan Rickman'}]}" ] }, - "execution_count": 5, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -255,7 +256,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 9, "id": "1f2a2b14-a956-4f96-90c9-e11ca04ab7d1", "metadata": {}, "outputs": [ @@ -266,33 +267,33 @@ " 'name': 'Severus Snape',\n", " 'house': 'Slytherin',\n", " 'properties': [{'index': '1',\n", - " 'key': 'Occupation',\n", - " 'value': 'Potions Master, Defense Against the Dark Arts teacher, Headmaster'},\n", - " {'index': '2',\n", - " 'key': 'Loyalty',\n", - " 'value': 'Hogwarts School, Order of the Phoenix, Albus Dumbledore'},\n", - " {'index': '3',\n", - " 'key': 'Skills',\n", - " 'value': 'Potions, Occlumency, Legilimency'},\n", - " {'index': '4', 'key': 'Patronus', 'value': 'Doe'},\n", + " 'key': 'Role',\n", + " 'value': 'Professor of Potions, later Defence Against the Dark Arts, and head of Slytherin House'},\n", + " {'index': '2', 'key': 'Patronus', 'value': 'Doe'},\n", + " {'index': '3', 'key': 'Loyalty', 'value': 'Dumbledore, Harry, Hogwarts'},\n", + " {'index': '4',\n", + " 'key': 'Special Skill',\n", + " 'value': 'Occlumens, Potions Master'},\n", " {'index': '5', 'key': 'Played by', 'value': 'Alan Rickman'}]},\n", " {'age': 115,\n", " 'name': 'Albus Dumbledore',\n", " 'house': 'Gryffindor',\n", " 'properties': [{'index': '1',\n", - " 'key': 'Occupation',\n", - " 'value': 'Headmaster, Founder of the Order of the Phoenix'},\n", - " {'index': '2',\n", + " 'key': 'Role',\n", + " 'value': 'Headmaster of Hogwarts'},\n", + " {'index': '2', 'key': 'Patronus', 'value': 'Phoenix'},\n", + " {'index': '3',\n", " 'key': 'Loyalty',\n", - " 'value': 'Hogwarts School, Order of the Phoenix'},\n", - " {'index': '3', 'key': 'Skills', 'value': 'Transfiguration, Alchemy'},\n", - " {'index': '4', 'key': 'Patronus', 'value': 'Phoenix'},\n", + " 'value': 'Order of the Phoenix, Hogwarts'},\n", + " {'index': '4',\n", + " 'key': 'Special Skill',\n", + " 'value': 'Considered to be the most powerful wizard of his time'},\n", " {'index': '5',\n", " 'key': 'Played by',\n", - " 'value': 'Richard Harris (films 1-2), Michael Gambon (films 3-8)'}]}]}" + " 'value': 'Richard Harris (films 1-2), Michael Gambon (films 3-6)'}]}]}" ] }, - "execution_count": 6, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -326,25 +327,21 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 12, "id": "6de8768e-b36a-4a51-9cf9-940d178552f6", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "{'users': [{'id': 1, 'name': 'Harry Potter', 'friends': [2, 3, 4, 5, 6, 7]},\n", - " {'id': 2, 'name': 'Ron Weasley', 'friends': [1, 3, 5, 6, 7]},\n", - " {'id': 3, 'name': 'Hermione Granger', 'friends': [1, 2, 5, 6, 7]},\n", - " {'id': 4, 'name': 'Draco Malfoy', 'friends': [1, 7]},\n", - " {'id': 5, 'name': 'Neville Longbottom', 'friends': [1, 2, 3]},\n", - " {'id': 6, 'name': 'Luna Lovegood', 'friends': [1, 2, 3]},\n", - " {'id': 7, 'name': 'Ginny Weasley', 'friends': [1, 2, 3]},\n", - " {'id': 8, 'name': 'Fred Weasley', 'friends': [2, 7]},\n", - " {'id': 9, 'name': 'George Weasley', 'friends': [2, 7]}]}" + "{'users': [{'id': 1, 'name': 'Harry Potter', 'friends': [2, 3, 4, 5]},\n", + " {'id': 2, 'name': 'Hermione Granger', 'friends': [1, 3, 4, 5]},\n", + " {'id': 3, 'name': 'Ron Weasley', 'friends': [1, 2, 4, 5]},\n", + " {'id': 4, 'name': 'Ginny Weasley', 'friends': [1, 2, 3, 5]},\n", + " {'id': 5, 'name': 'Neville Longbottom', 'friends': [1, 2, 3, 4]}]}" ] }, - "execution_count": 7, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -363,7 +360,7 @@ " messages=[\n", " {\n", " \"role\": \"user\", \n", - " \"content\": \"The kids from from Harry Potter\"\n", + " \"content\": \"The 5 kids from from Harry Potter\"\n", " }\n", " ],\n", " response_model=Characters\n", @@ -373,7 +370,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "id": "b31e10d7-ebd2-49b4-b2c4-20dd67ca135d", "metadata": {}, "outputs": [ @@ -386,153 +383,105 @@ "\n", "\n", - "\n", - "\n", - "\n", + "\n", + "\n", + "\n", "\n", "\n", "1\n", - "\n", - "Harry Potter\n", + "\n", + "Harry Potter\n", "\n", "\n", "\n", "2\n", - "\n", - "Ron Weasley\n", + "\n", + "Hermione Granger\n", "\n", "\n", "\n", "1->2\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "3\n", - "\n", - "Hermione Granger\n", + "\n", + "Ron Weasley\n", "\n", "\n", "\n", "1->3\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "4\n", - "\n", - "Draco Malfoy\n", + "\n", + "Ginny Weasley\n", "\n", "\n", "\n", "1->4\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "5\n", - "\n", - "Neville Longbottom\n", + "\n", + "Neville Longbottom\n", "\n", "\n", "\n", "1->5\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "6\n", - "\n", - "Luna Lovegood\n", + "\n", + "\n", "\n", - "\n", + "\n", "\n", - "1->6\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "7\n", - "\n", - "Ginny Weasley\n", + "2->3\n", + "\n", + "\n", "\n", - "\n", + "\n", "\n", - "1->7\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "2->3\n", - "\n", - "\n", + "2->4\n", + "\n", + "\n", "\n", "\n", - "\n", + "\n", "2->5\n", - "\n", - "\n", + "\n", + "\n", "\n", - "\n", - "\n", - "2->6\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "2->7\n", - "\n", - "\n", + "\n", + "\n", + "3->4\n", + "\n", + "\n", "\n", "\n", - "\n", + "\n", "3->5\n", - "\n", - "\n", + "\n", + "\n", "\n", - "\n", - "\n", - "3->6\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "3->7\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "4->7\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "8\n", - "\n", - "Fred Weasley\n", - "\n", - "\n", - "\n", - "9\n", - "\n", - "George Weasley\n", + "\n", + "\n", + "4->5\n", + "\n", + "\n", "\n", "\n", "\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -573,9 +522,9 @@ ], "metadata": { "kernelspec": { - "display_name": "instructor-env", + "display_name": "Python 3 (ipykernel)", "language": "python", - "name": "instructor-env" + "name": "python3" }, "language_info": { "codemirror_mode": { diff --git a/tutorials/3.applications-rag.ipynb b/tutorials/3.applications-rag.ipynb index 771a3b084..92c188680 100644 --- a/tutorials/3.applications-rag.ipynb +++ b/tutorials/3.applications-rag.ipynb @@ -88,7 +88,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 16, "metadata": {}, "outputs": [], "source": [ @@ -121,7 +121,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 17, "metadata": {}, "outputs": [], "source": [ @@ -133,7 +133,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 15, "metadata": {}, "outputs": [ { @@ -141,25 +141,24 @@ "output_type": "stream", "text": [ "{\n", - " \"summary\": \"The simplest implementation of RAG or Retriever-Augmented Generation includes embedding a user query and conducting a single embedding search in a vector database such as a store of Wikipedia articles. This approach faces limitations such as query and document mismatch, monolithic search backend, text search limitations and limited planning ability.\",\n", + " \"summary\": \"Simple RAG, an approach to query embedding and document retrieval, often falls short due to misalignment between query and document embeddings, reliance on a single search method and backend, text search limitations, and inadequate consideration of contextual information.\",\n", " \"hypothetical_questions\": [\n", - " \"What is the simplest implementation of RAG?\",\n", - " \"What are the limitations of the simplest RAG implementation?\",\n", - " \"How does a Query-Document mismatch occur in RAG?\",\n", - " \"Why is the monolithic search backend a limitation for RAG?\",\n", - " \"What is the issue with text search limitations in RAG?\",\n", - " \"how does limited planning ability affect RAG performance?\"\n", + " \"How can RAG be improved to better handle complex queries?\",\n", + " \"What other methods exist to enhance the alignment of queries and documents in vector spaces?\",\n", + " \"How can multiple data sources be incorporated into the search mechanism of RAG?\",\n", + " \"In what ways can advanced search features be integrated into RAG?\",\n", + " \"What approaches enable RAG to use contextual information for improved search results?\"\n", " ],\n", " \"keywords\": [\n", " \"RAG\",\n", - " \"Simple implementation\",\n", - " \"Vector database\",\n", - " \"Wikipedia articles\",\n", - " \"Embedding\",\n", - " \"Query-Document mismatch\",\n", - " \"Monolithic search backend\",\n", - " \"Text search limitations\",\n", - " \"Limited planning ability\"\n", + " \"query embedding\",\n", + " \"document retrieval\",\n", + " \"vector database\",\n", + " \"query-document mismatch\",\n", + " \"monolithic search backend\",\n", + " \"text search limitations\",\n", + " \"planning ability\",\n", + " \"contextual information\"\n", " ]\n", "}\n" ] @@ -190,7 +189,7 @@ "\"\"\"\n", "\n", "extraction = client.chat.completions.create(\n", - " model=\"gpt-4\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=Extraction,\n", " messages=[{\n", " \"role\": \"system\", \n", @@ -222,7 +221,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 25, "metadata": {}, "outputs": [], "source": [ @@ -255,7 +254,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 26, "metadata": {}, "outputs": [ { @@ -263,11 +262,10 @@ "output_type": "stream", "text": [ "{\n", - " \"rewritten_query\": \"Find information on recent developments in Artificial Intelligence\",\n", + " \"rewritten_query\": \"recent developments in Artificial Intelligence\",\n", " \"published_daterange\": {\n", - " \"chain_of_thought\": \"The user is asking for recent information. This term can be subjective, so to narrow it down, let's target information from the past two years.\",\n", - " \"start\": \"2021-11-10\",\n", - " \"end\": \"2023-11-10\"\n", + " \"start\": \"2023-04-01\",\n", + " \"end\": \"2023-11-18\"\n", " }\n", "}\n" ] @@ -275,7 +273,7 @@ ], "source": [ "query = client.chat.completions.create(\n", - " model=\"gpt-4\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=Query,\n", " messages=[\n", " {\n", @@ -292,9 +290,14 @@ "print(query.model_dump_json(indent=4)) # Printing the Json dump of the model" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 23, "metadata": {}, "outputs": [ { @@ -302,11 +305,11 @@ "output_type": "stream", "text": [ "{\n", - " \"rewritten_query\": \"recent advancements in artificial intelligence\",\n", + " \"rewritten_query\": \"recent developments in artificial intelligence\",\n", " \"published_daterange\": {\n", - " \"chain_of_thought\": \"The user is looking for recent developments, so we look for information in the last year.\",\n", - " \"start\": \"2022-11-10\",\n", - " \"end\": \"2023-11-10\"\n", + " \"chain_of_thought\": \"Given that it's currently late 2023, a recent timeframe would ideally be within the last year to ensure the developments are current. Therefore, a suitable date range for recent AI developments could start from late 2022 to the present date in 2023.\",\n", + " \"start\": \"2022-11-18\",\n", + " \"end\": \"2023-11-18\"\n", " }\n", "}\n" ] @@ -314,8 +317,7 @@ ], "source": [ "class DateRange(BaseModel):\n", - " chain_of_thought: str = Field( # Chain of thought prompting \n", - " None,\n", + " chain_of_thought: str = Field(\n", " description=\"Think step by step to plan what is the best time range to search in\"\n", " )\n", " start: date\n", @@ -327,7 +329,7 @@ "\n", "\n", "query = client.chat.completions.create(\n", - " model=\"gpt-4\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=Query,\n", " messages=[\n", " {\n", @@ -359,7 +361,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 27, "metadata": {}, "outputs": [], "source": [ @@ -389,7 +391,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 28, "metadata": {}, "outputs": [ { @@ -399,17 +401,18 @@ "{\n", " \"queries\": [\n", " {\n", - " \"query\": \"Jason's events for 2023-11-10\",\n", + " \"query\": \"schedule\",\n", " \"keywords\": [\n", - " \"event\",\n", - " \"meeting\"\n", + " \"appointments\",\n", + " \"meetings\",\n", + " \"schedule\",\n", + " \"events\"\n", " ],\n", - " \"email\": \"jason@gmail.com\",\n", + " \"email\": \"jason.assistant@busybot.com\",\n", " \"source\": \"calendar\",\n", " \"date_range\": {\n", - " \"chain_of_thought\": \"events for 2023-11-10\",\n", - " \"start\": \"2023-11-10\",\n", - " \"end\": \"2023-11-10\"\n", + " \"start\": \"2023-11-18\",\n", + " \"end\": \"2023-11-18\"\n", " }\n", " }\n", " ]\n", @@ -419,7 +422,7 @@ ], "source": [ "retrival = client.chat.completions.create(\n", - " model=\"gpt-4\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=Retrival,\n", " messages=[\n", " {\"role\": \"system\", \"content\":f\"You are Jason's personal assistant. Today is {date.today()}\"},\n", @@ -438,7 +441,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 29, "metadata": {}, "outputs": [ { @@ -450,30 +453,31 @@ " {\n", " \"query\": \"meetings\",\n", " \"keywords\": [\n", - " \"meeting\"\n", + " \"meetings\",\n", + " \"appointments\",\n", + " \"schedule\",\n", + " \"calendar\"\n", " ],\n", - " \"email\": \"jason@example.com\",\n", + " \"email\": \"user@email.com\",\n", " \"source\": \"calendar\",\n", " \"date_range\": {\n", - " \"chain_of_thought\": \"today\",\n", - " \"start\": \"2023-11-10\",\n", - " \"end\": \"2023-11-10\"\n", + " \"start\": \"2023-11-18\",\n", + " \"end\": \"2023-11-18\"\n", " }\n", " },\n", " {\n", " \"query\": \"important emails\",\n", " \"keywords\": [\n", - " \"urgent\",\n", " \"important\",\n", - " \"asap\",\n", - " \"action required\"\n", + " \"priority\",\n", + " \"urgent\",\n", + " \"follow-up\"\n", " ],\n", - " \"email\": \"jason@example.com\",\n", + " \"email\": \"user@email.com\",\n", " \"source\": \"gmail\",\n", " \"date_range\": {\n", - " \"chain_of_thought\": \"today\",\n", - " \"start\": \"2023-11-10\",\n", - " \"end\": \"2023-11-10\"\n", + " \"start\": \"2023-11-18\",\n", + " \"end\": \"2023-11-18\"\n", " }\n", " }\n", " ]\n", @@ -483,7 +487,7 @@ ], "source": [ "retrival = client.chat.completions.create(\n", - " model=\"gpt-4\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=Retrival,\n", " messages=[\n", " {\"role\": \"system\", \"content\": f\"You are Jason's personal assistant. Today is {date.today()}\"},\n", @@ -513,9 +517,14 @@ "This would not be done correctly as a single query, nor would it be done in parallel, however there are some opportunities try to be parallel since not all of the sub-questions are dependent on each other." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 31, "metadata": {}, "outputs": [ { @@ -523,31 +532,31 @@ "output_type": "stream", "text": [ "{\n", - " \"root_question\": \"What is the difference between the population of Jason's home country and Canada?\",\n", + " \"root_question\": \"What is the difference between the population of jason's home country and canada?\",\n", " \"plan\": [\n", " {\n", - " \"id\": 0,\n", + " \"id\": 1,\n", " \"query\": \"What is Jason's home country?\",\n", " \"subquestions\": []\n", " },\n", " {\n", - " \"id\": 1,\n", - " \"query\": \"What is the population of Jason's home country?\",\n", - " \"subquestions\": [\n", - " 0\n", - " ]\n", - " },\n", - " {\n", " \"id\": 2,\n", " \"query\": \"What is the population of Canada?\",\n", " \"subquestions\": []\n", " },\n", " {\n", " \"id\": 3,\n", - " \"query\": \"What is the difference between two numbers?\",\n", + " \"query\": \"What is the population of {Jason's home country}?\",\n", + " \"subquestions\": [\n", + " 1\n", + " ]\n", + " },\n", + " {\n", + " \"id\": 4,\n", + " \"query\": \"What is the difference between the population of {Jason's home country} and the population of Canada?\",\n", " \"subquestions\": [\n", - " 1,\n", - " 2\n", + " 2,\n", + " 3\n", " ]\n", " }\n", " ]\n", @@ -568,7 +577,7 @@ "\n", "\n", "retrival = client.chat.completions.create(\n", - " model=\"gpt-4\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=QueryPlan,\n", " messages=[\n", " {\"role\": \"system\", \"content\":\"You are a query understanding system capable of decomposing a question into subquestions.\"},\n", diff --git a/tutorials/4.knowledge-graphs.ipynb b/tutorials/4.knowledge-graphs.ipynb index e4f3882c1..91a4fc17e 100644 --- a/tutorials/4.knowledge-graphs.ipynb +++ b/tutorials/4.knowledge-graphs.ipynb @@ -51,16 +51,7 @@ }, { "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "!pip install instructor graphviz --quiet" - ] - }, - { - "cell_type": "code", - "execution_count": 7, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -74,7 +65,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Install Graphviz based on your operation system https://graphviz.org/download/" + "Install the Graphviz based on your operation system https://graphviz.org/download/" ] }, { @@ -90,7 +81,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ @@ -129,7 +120,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 4, "metadata": {}, "outputs": [], "source": [ @@ -167,17 +158,17 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "def generate_graph(input) -> KnowledgeGraph:\n", " return client.chat.completions.create(\n", - " model=\"gpt-3.5-turbo\",\n", + " model=\"gpt-4-1106-preview\",\n", " messages=[\n", " {\n", " \"role\": \"user\",\n", - " \"content\": f\"Help me understand the following by describing it as a detailed knowledge graph: {input}\",\n", + " \"content\": f\"Help me understand the following by describing it as small knowledge graph: {input}\",\n", " }\n", " ],\n", " response_model=KnowledgeGraph,\n", @@ -186,7 +177,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 9, "metadata": {}, "outputs": [ { @@ -198,125 +189,112 @@ "\n", "\n", - "\n", - "\n", - "\n", + "\n", + "\n", + "\n", "\n", "\n", "1\n", - "\n", - "Artificial Intelligence\n", + "\n", + "Quantum Mechanics\n", "\n", "\n", "\n", "2\n", - "\n", - "Machine Learning\n", + "\n", + "Quantum Particles\n", "\n", "\n", "\n", "1->2\n", - "\n", - "\n", - "is a subset of\n", + "\n", + "\n", + "studies\n", "\n", "\n", "\n", "3\n", - "\n", - "Deep Learning\n", + "\n", + "Wave-Particle Duality\n", "\n", - "\n", + "\n", "\n", - "2->3\n", - "\n", - "\n", - "is a subset of\n", + "1->3\n", + "\n", + "\n", + "describes\n", "\n", "\n", "\n", "4\n", - "\n", - "Neural Network\n", + "\n", + "Quantum States\n", "\n", - "\n", + "\n", "\n", - "3->4\n", - "\n", - "\n", - "is a subset of\n", - "\n", - "\n", - "\n", - "8\n", - "\n", - "Weights\n", - "\n", - "\n", - "\n", - "4->8\n", - "\n", - "\n", - "has\n", - "\n", - "\n", - "\n", - "9\n", - "\n", - "Activation Function\n", - "\n", - "\n", - "\n", - "4->9\n", - "\n", - "\n", - "uses\n", + "1->4\n", + "\n", + "\n", + "involves\n", "\n", "\n", "\n", "5\n", - "\n", - "Input Layer\n", + "\n", + "Uncertainty Principle\n", "\n", - "\n", + "\n", "\n", - "5->4\n", - "\n", - "\n", - "has\n", + "1->5\n", + "\n", + "\n", + "introduces\n", "\n", "\n", "\n", "6\n", - "\n", - "Hidden Layer\n", + "\n", + "Schrodinger's Equation\n", "\n", - "\n", + "\n", "\n", - "6->4\n", - "\n", - "\n", - "has\n", + "1->6\n", + "\n", + "\n", + "defined by\n", "\n", "\n", "\n", "7\n", - "\n", - "Output Layer\n", + "\n", + "Quantum Entanglement\n", "\n", - "\n", + "\n", "\n", - "7->4\n", - "\n", - "\n", - "has\n", + "1->7\n", + "\n", + "\n", + "predicts\n", + "\n", + "\n", + "\n", + "8\n", + "\n", + "Superposition\n", + "\n", + "\n", + "\n", + "1->8\n", + "\n", + "\n", + "introduces\n", "\n", "\n", "\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -324,7 +302,7 @@ } ], "source": [ - "generate_graph(\"Explain the concept of a neural network.\").visualize_knowledge_graph()" + "generate_graph(\"Explain quantum mechanics\").visualize_knowledge_graph()" ] }, { @@ -363,7 +341,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 10, "metadata": {}, "outputs": [], "source": [ @@ -387,7 +365,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -415,6 +393,11 @@ " return display(dot)\n" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, { "cell_type": "markdown", "metadata": {}, @@ -437,7 +420,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ @@ -448,7 +431,7 @@ " # Iterate over the input list\n", " for i, inp in enumerate(input):\n", " new_updates = client.chat.completions.create(\n", - " model=\"gpt-3.5-turbo-16k\",\n", + " model=\"gpt-4-1106-preview\",\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", @@ -502,9 +485,413 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 13, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "Physicist\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "Professor\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "Quantum Mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows about\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "Physicist\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Smart\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "Professor\n", + "\n", + "\n", + "\n", + "4->5\n", + "\n", + "\n", + "are\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "Quantum Mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows about\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "Physicist\n", + "\n", + "\n", + "\n", + "6\n", + "\n", + "Sarah\n", + "\n", + "\n", + "\n", + "7\n", + "\n", + "Student\n", + "\n", + "\n", + "\n", + "6->7\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "Professor\n", + "\n", + "\n", + "\n", + "6->4\n", + "\n", + "\n", + "student of\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "6->1\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Smart\n", + "\n", + "\n", + "\n", + "4->5\n", + "\n", + "\n", + "are\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "Quantum Mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows about\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "Physicist\n", + "\n", + "\n", + "\n", + "9\n", + "\n", + "Canada\n", + "\n", + "\n", + "\n", + "8\n", + "\n", + "University of Toronto\n", + "\n", + "\n", + "\n", + "8->9\n", + "\n", + "\n", + "is in\n", + "\n", + "\n", + "\n", + "6\n", + "\n", + "Sarah\n", + "\n", + "\n", + "\n", + "6->8\n", + "\n", + "\n", + "student at\n", + "\n", + "\n", + "\n", + "7\n", + "\n", + "Student\n", + "\n", + "\n", + "\n", + "6->7\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "Professor\n", + "\n", + "\n", + "\n", + "6->4\n", + "\n", + "\n", + "student of\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "6->1\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Smart\n", + "\n", + "\n", + "\n", + "4->5\n", + "\n", + "\n", + "are\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "Quantum Mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows about\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "text_chunks = [\n", " \"Jason knows a lot about quantum mechanics. He is a physicist. He is a professor\",\n", @@ -588,7 +975,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -606,5 +993,5 @@ } }, "nbformat": 4, - "nbformat_minor": 4 + "nbformat_minor": 2 } diff --git a/tutorials/5.validation.ipynb b/tutorials/5.validation.ipynb index dad31164e..fcde9fe28 100644 --- a/tutorials/5.validation.ipynb +++ b/tutorials/5.validation.ipynb @@ -5,7 +5,7 @@ "id": "5a01f3ac-5306-4a1b-9e47-a5d254bce93a", "metadata": {}, "source": [ - "# Validation" + "# Validators" ] }, { @@ -13,13 +13,11 @@ "id": "9dcc78ac-ed6d-49e3-b71b-fb2fb25f16a8", "metadata": {}, "source": [ - "## Introduction\n", + "Instead of framing \"self-critique\" or \"self-reflection\" in AI as new concepts, we can view them as validation errors with clear error messages that the systen can use to self correct.\n", "\n", - "**What is a validation**\n", + "Pydantic offers an customizable and expressive validation framework for Python. Instructor leverages Pydantic's validation framework to provide a uniform developer experience for both code-based and LLM-based validation, as well as a reasking mechanism for correcting LLM outputs based on validation errors. To learn more check out the Pydantic docs on validators.\n", "\n", - "Validation is the backbone of reliable software. Essentially, it involves checking whether the output of a function matches our expectations. Traditionally, validation was deterministic and rule-based, focusing on specific criteria (for example, 'output must not be larger than a certain value'). However, with the advent of Large Language Models (LLMs), we've seen the integration of probabilistic validation into our processes. This type of validation, often labeled as 'guardrail', might appear novel, but it's essentially another form of validation that leverages LLMs instead of strict rules to flag responses.\n", - "\n", - "In our approach at instructor, we treat all forms of validation equally. Whether it's rule-based, probabilistic, or a combination of both, everything is managed within a singular, cohesive framework. We achieve this through extensive use of Pydantic's powerful [validators](https://docs.pydantic.dev/latest/concepts/validators/#field-validators) feature." + "Note: For the majourity of this notebook we won't be calling openai, just using validators to see how we can control the validation of the objects." ] }, { @@ -27,34 +25,53 @@ "id": "064c286b", "metadata": {}, "source": [ - "Validators will enable us to control outputs by defining a function like so:" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "d4bb6258-b03a-4621-8a73-29056a20ec0f", - "metadata": {}, - "outputs": [], - "source": [ + "Validators will enable us to control outputs by defining a function like so:\n", + "\n", + "\n", + "```python\n", "def validation_function(value):\n", " if condition(value):\n", " raise ValueError(\"Value is not valid\")\n", - " return mutation(value)" + " return mutation(value)\n", + "```\n", + "\n", + "Before we get started lets go over the general shape of a validator:" ] }, { - "cell_type": "markdown", - "id": "11d41e88-4629-4a44-a701-2bcdb297f090", + "cell_type": "code", + "execution_count": 61, + "id": "d4bb6258-b03a-4621-8a73-29056a20ec0f", "metadata": {}, + "outputs": [ + { + "ename": "ValidationError", + "evalue": "1 validation error for UserDetail\nname\n Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 4\u001b[0m line \u001b[0;36m1\n\u001b[1;32m 11\u001b[0m age: \u001b[39mint\u001b[39m\n\u001b[1;32m 12\u001b[0m name: Annotated[\u001b[39mstr\u001b[39m, AfterValidator(name_must_contain_space)]\n\u001b[0;32m---> 14\u001b[0m person \u001b[39m=\u001b[39m UserDetail(age\u001b[39m=\u001b[39;49m\u001b[39m29\u001b[39;49m, name\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mJason\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for UserDetail\nname\n Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" + ] + } + ], "source": [ - "The validation process in this framework unfolds in three key steps:\n", + "from pydantic import BaseModel, ValidationError\n", + "from typing_extensions import Annotated\n", + "from pydantic import AfterValidator\n", "\n", - "* Condition Verification: The first step involves the validator checking whether a value meets a set condition. This is where the core validation logic is applied.\n", + "def name_must_contain_space(v: str) -> str:\n", + " if \" \" not in v:\n", + " raise ValueError(\"Name must contain a space.\")\n", + " return v.lower()\n", "\n", - "* Error Handling with Optional Retry: If the value doesn't meet the condition, the system raises an error. There's an option to retry, offering a chance for correcting and reevaluating the value.\n", + "class UserDetail(BaseModel):\n", + " age: int\n", + " name: Annotated[str, AfterValidator(name_must_contain_space)]\n", "\n", - "* Value Processing: When the value meets the condition, the validator returns either the original or a modified version of the value. This ensures the output is valid and meets specific requirements or preferences." + "person = UserDetail(age=29, name=\"Jason\")" ] }, { @@ -104,7 +121,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 62, "id": "1aa2c503-82f8-4735-aae3-373b55fb1064", "metadata": {}, "outputs": [], @@ -157,7 +174,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 63, "id": "59330d7d-082a-4240-98c4-eaee18f02728", "metadata": {}, "outputs": [], @@ -181,18 +198,20 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 64, "id": "9bb87f47-db98-4f1d-80cb-ad5f39df8793", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for UserMessage\n", - "message\n", - " Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for Response\nmessage\n Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 17\u001b[0m line \u001b[0;36m1\n\u001b[1;32m 11\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m`\u001b[39m\u001b[39m{\u001b[39;00mword\u001b[39m}\u001b[39;00m\u001b[39m` was found in the message `\u001b[39m\u001b[39m{\u001b[39;00mv\u001b[39m}\u001b[39;00m\u001b[39m`\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 12\u001b[0m \u001b[39mreturn\u001b[39;00m v\n\u001b[0;32m---> 14\u001b[0m Response(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI will hurt him\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Response\nmessage\n Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" ] } ], @@ -200,7 +219,7 @@ "from pydantic import BaseModel, ValidationError, field_validator\n", "from pydantic.fields import Field\n", "\n", - "class UserMessage(BaseModel):\n", + "class Response(BaseModel):\n", " message: str\n", "\n", " @field_validator('message')\n", @@ -210,10 +229,7 @@ " raise ValueError(f\"`{word}` was found in the message `{v}`\")\n", " return v\n", "\n", - "try:\n", - " UserMessage(message=\"I will hurt him\")\n", - "except ValidationError as e:\n", - " print(e)" + "Response(message=\"I will hurt him\")" ] }, { @@ -242,12 +258,12 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 65, "id": "82521112-5301-4442-acce-82b495bd838f", "metadata": {}, "outputs": [], "source": [ - "class UserMessage(BaseModel):\n", + "class Response(BaseModel):\n", " message: str\n", "\n", " @field_validator('message')\n", @@ -271,26 +287,25 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 66, "id": "54a9de1b-c6e7-4a5f-854c-506083a06a9d", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for UserMessage\n", - "message\n", - " Value error, `I want to make them suffer the consequences` was flagged for ['harassment', 'harassment_threatening', 'violence', 'harassment/threatening'] [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for Response\nmessage\n Value error, `I want to make them suffer the consequences` was flagged for ['harassment', 'harassment_threatening', 'violence', 'harassment/threatening'] [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 23\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> 1\u001b[0m Response(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI want to make them suffer the consequences\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Response\nmessage\n Value error, `I want to make them suffer the consequences` was flagged for ['harassment', 'harassment_threatening', 'violence', 'harassment/threatening'] [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" ] } ], "source": [ - "try:\n", - " UserMessage(message=\"I want to make them suffer the consequences\")\n", - "except ValidationError as e:\n", - " print(e)" + "Response(message=\"I want to make them suffer the consequences\")" ] }, { @@ -303,26 +318,25 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 67, "id": "feb77670-afd7-4947-89f8-a9446f6fb12c", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for UserMessage\n", - "message\n", - " Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for Response\nmessage\n Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 25\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> 1\u001b[0m Response(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI will mock their religion\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Response\nmessage\n Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" ] } ], "source": [ - "try:\n", - " UserMessage(message=\"I will mock their religion\")\n", - "except ValidationError as e:\n", - " print(e)" + "Response(message=\"I will mock their religion\")" ] }, { @@ -340,55 +354,41 @@ "source": [ "In addition to content-based flags, we can also set criteria based on other aspects of the input text. For instance, to maintain user engagement, we might want to prevent the assistant from returning excessively long texts. \n", "\n", - "We can implement this using `instructor` to set a maximum word or character limit on the assistant's responses:" + "Here, noticed that `Field` has built-in validators for `min_length` and `max_length`. to learn more checkout [Field Contraints](https://docs.pydantic.dev/latest/concepts/fields)" ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 68, "id": "45ffdbd4-deae-4a46-9637-1b5339904f53", "metadata": {}, "outputs": [], "source": [ "class AssistantMessage(BaseModel):\n", - " message: str\n", - "\n", - " @field_validator('message')\n", - " def message_must_be_short(cls, v: str) -> str:\n", - " if len(v.split())>=100:\n", - " raise ValueError(f\"Text was flagged for being longer than 100 words.\")\n", - " \n", - " return v " + " message: str = Field(..., max_length=100)" ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 69, "id": "66430dc5-b78c-45e2-a53b-ddc392b20583", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for AssistantMessage\n", - "message\n", - " Value error, Text was flagged for being longer than 100 words. [type=value_error, input_value=\"\\n Certainly! Lorem i... on the actual content.\", input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for AssistantMessage\nmessage\n String should have at most 100 characters [type=string_too_long, input_value=\"Certainly! Lorem ipsum i... on the actual content.\", input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/string_too_long", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 29\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> 1\u001b[0m AssistantMessage(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mCertainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39ms a sample of Lorem ipsum text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39ms often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for AssistantMessage\nmessage\n String should have at most 100 characters [type=string_too_long, input_value=\"Certainly! Lorem ipsum i... on the actual content.\", input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/string_too_long" ] } ], "source": [ - "try:\n", - " AssistantMessage(message=\"\"\"\n", - " Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text:\n", - "\n", - "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus.\n", - "\n", - "Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\"\"\"\n", - " )\n", - "except ValidationError as e:\n", - " print(e)" + "AssistantMessage(message=\"Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\")" ] }, { @@ -396,7 +396,7 @@ "id": "050e72fe-4b13-4002-a1d0-94f7b88b784b", "metadata": {}, "source": [ - "### Avoiding hallucination" + "### Avoiding hallucination with citations" ] }, { @@ -404,20 +404,12 @@ "id": "e3f2869e-c8a3-4b93-82e7-55eb70930900", "metadata": {}, "source": [ - "When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. " - ] - }, - { - "cell_type": "markdown", - "id": "f67f7f92", - "metadata": {}, - "source": [ - "We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:" + "When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 70, "id": "638fc368-5cf7-4ae7-9d3f-efea1b84eec0", "metadata": {}, "outputs": [], @@ -435,7 +427,7 @@ " if context:\n", " context = context.get('text_chunk')\n", " if v not in context:\n", - " raise ValueError(f\"Citation `{v}` not found in text chunks\")\n", + " raise ValueError(f\"Citation `{v}` not found in text\")\n", " return v" ] }, @@ -444,34 +436,36 @@ "id": "3064b06b-7f85-40ec-8fe2-4fa2cce36585", "metadata": {}, "source": [ - "When the model responds with information not present in the provided context, the validation process comes into play:" + "Here we assume that there is a \"text_chunk\" field that contains the text that the model is supposed to use as context. We then use the `field_validator` decorator to define a validator that checks if the citation is included in the text chunk. If it's not, we raise a `ValueError` with a message that will be returned to the user." ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 71, "id": "0f3030b6-e6cf-45bf-a366-12de996fea40", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for AnswerWithCitation\n", - "citation\n", - " Value error, Citation `Blueberries contain high levels of protein` not found in text chunks [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for AnswerWithCitation\ncitation\n Value error, Citation `Blueberries contain high levels of protein` not found in text [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 34\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> 1\u001b[0m AnswerWithCitation\u001b[39m.\u001b[39;49mmodel_validate(\n\u001b[1;32m 2\u001b[0m {\n\u001b[1;32m 3\u001b[0m \u001b[39m\"\u001b[39;49m\u001b[39manswer\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mBlueberries are packed with protein\u001b[39;49m\u001b[39m\"\u001b[39;49m, \n\u001b[1;32m 4\u001b[0m \u001b[39m\"\u001b[39;49m\u001b[39mcitation\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mBlueberries contain high levels of protein\u001b[39;49m\u001b[39m\"\u001b[39;49m\n\u001b[1;32m 5\u001b[0m },\n\u001b[1;32m 6\u001b[0m context\u001b[39m=\u001b[39;49m{\u001b[39m\"\u001b[39;49m\u001b[39mtext_chunk\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mBlueberries are very rich in antioxidants\u001b[39;49m\u001b[39m\"\u001b[39;49m}, \n\u001b[1;32m 7\u001b[0m )\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:503\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m 501\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 502\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 503\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(\n\u001b[1;32m 504\u001b[0m obj, strict\u001b[39m=\u001b[39;49mstrict, from_attributes\u001b[39m=\u001b[39;49mfrom_attributes, context\u001b[39m=\u001b[39;49mcontext\n\u001b[1;32m 505\u001b[0m )\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for AnswerWithCitation\ncitation\n Value error, Citation `Blueberries contain high levels of protein` not found in text [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" ] } ], "source": [ - "try:\n", - " AnswerWithCitation.model_validate(\n", - " {\"answer\": \"Blueberries are packed with protein\", \"citation\": \"Blueberries contain high levels of protein\"},\n", - " context={\"text_chunk\": \"Blueberries are very rich in antioxidants\"}, \n", - " )\n", - "except ValidationError as e:\n", - " print(e)" + "AnswerWithCitation.model_validate(\n", + " {\n", + " \"answer\": \"Blueberries are packed with protein\", \n", + " \"citation\": \"Blueberries contain high levels of protein\"\n", + " },\n", + " context={\"text_chunk\": \"Blueberries are very rich in antioxidants\"}, \n", + ")" ] }, { @@ -487,27 +481,15 @@ "id": "1907df5b-472f-45ac-9181-45235e3cd0c3", "metadata": {}, "source": [ - "For scenarios requiring more nuanced validation than what rule-based methods offer, we turn to probabilistic validation. This approach utilizes LLMs as a core component of the validation workflow, enabling a more sophisticated assessment of outputs.\n", + "For scenarios requiring more nuanced validation than rule-based methods, we use probabilistic validation. This approach incorporates LLMs into the validation workflow for a sophisticated assessment of outputs.\n", "\n", - "The `instructor` library facilitates this with its `llm_validator` utility. By specifying the desired directive, we can employ LLMs for complex validation tasks. Let's explore some intriguing use cases that are made possible through LLMs." - ] - }, - { - "cell_type": "markdown", - "id": "bd43d4c3-930a-4b3a-aded-3ad7308454ba", - "metadata": {}, - "source": [ - "### Keeping an agent on topic" - ] - }, - { - "cell_type": "markdown", - "id": "21a9d80c-755f-434e-be52-2f5b58142b8c", - "metadata": {}, - "source": [ - "In the case of creating an agent focused on health improvement, providing answers and daily practice suggestions, it's vital to ensure that the agent strictly adheres to health-related topics. This is important because the knowledge base is limited to health topics, and venturing beyond this scope could lead to fabricated ('hallucinated') responses.\n", + "The `instructor` library offers the `llm_validator` utility for this purpose. By specifying the desired directive, we can use LLMs for complex validation tasks. Let's explore some intriguing use cases enabled by LLMs.\n", "\n", - "To achieve this focus, we'll employ a similar process to the one previously discussed, but with an important addition: integrating an LLM into our validator. " + "### Keeping an agent on topic\n", + "\n", + "When creating an agent focused on health improvement, providing answers and daily practice suggestions, it's crucial to ensure strict adherence to health-related topics. This is important because the knowledge base is limited to health topics, and veering off-topic could result in fabricated responses.\n", + "\n", + "To achieve this focus, we'll follow a similar process as before, but with an important addition: integrating an LLM into our validator." ] }, { @@ -520,18 +502,20 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 73, "id": "8cf00cad-c4c0-49dd-9be5-fb02338a5a7f", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for AssistantMessage\n", - "message\n", - " Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/assertion_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for AssistantMessage\nmessage\n Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/assertion_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 38\u001b[0m line \u001b[0;36m1\n\u001b[1;32m 5\u001b[0m \u001b[39mclass\u001b[39;00m \u001b[39mAssistantMessage\u001b[39;00m(BaseModel):\n\u001b[1;32m 6\u001b[0m message: Annotated[\u001b[39mstr\u001b[39m, \n\u001b[1;32m 7\u001b[0m AfterValidator(\n\u001b[1;32m 8\u001b[0m llm_validator(\u001b[39m\"\u001b[39m\u001b[39mdon\u001b[39m\u001b[39m'\u001b[39m\u001b[39mt talk about any other topic except health best practices and topics\u001b[39m\u001b[39m\"\u001b[39m, \n\u001b[1;32m 9\u001b[0m openai_client\u001b[39m=\u001b[39mclient))]\n\u001b[0;32m---> 11\u001b[0m AssistantMessage(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI would suggest you to visit Sicily as they say it is very nice in winter.\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for AssistantMessage\nmessage\n Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/assertion_error" ] } ], @@ -541,12 +525,12 @@ "from instructor import llm_validator\n", "\n", "class AssistantMessage(BaseModel):\n", - " message: Annotated[str, AfterValidator(llm_validator(\"don't talk about any other topic except health best practices and topics\"))]\n", + " message: Annotated[str, \n", + " AfterValidator(\n", + " llm_validator(\"don't talk about any other topic except health best practices and topics\", \n", + " openai_client=client))]\n", "\n", - "try:\n", - " AssistantMessage(message=\"I would suggest you to visit Sicily as they say it is very nice in winter.\")\n", - "except ValidationError as e:\n", - " print(e)" + "AssistantMessage(message=\"I would suggest you to visit Sicily as they say it is very nice in winter.\")" ] }, { @@ -554,7 +538,7 @@ "id": "1dce5a7a-024e-4742-a124-fe51973df5f2", "metadata": {}, "source": [ - "Great! With these validations, the model will reliably stick to its knowledge base, ensuring accurate and topic-specific responses." + "Important that for these examples we're not waiting for the messages, to get this message we would need to call the openai with `response_model=AssistantMessage`." ] }, { @@ -570,30 +554,16 @@ "id": "424d915b-f332-48f3-a75e-6e1cd6d12075", "metadata": {}, "source": [ - "Using probabilistic validation, we can also assess the agent's reasoning process to ensure it's logical before providing a response. With [chain of thought](https://learnprompting.org/docs/intermediate/chain_of_thought) prompting, the model is expected to think in steps and arrive at an answer following its logical progression. If there are errors in this logic, the final response may be incorrect." - ] - }, - { - "cell_type": "markdown", - "id": "79c95242-6517-4ce2-aa99-4437db658057", - "metadata": {}, - "source": [ + "Using probabilistic validation, we can also assess the agent's reasoning process to ensure it's logical before providing a response. With [chain of thought](https://learnprompting.org/docs/intermediate/chain_of_thought) prompting, the model is expected to think in steps and arrive at an answer following its logical progression. If there are errors in this logic, the final response may be incorrect.\n", + "\n", "Here we will use Pydantic's [model_validator](https://docs.pydantic.dev/latest/concepts/validators/#model-validators) which allows us to apply validation over all the properties of the `AIResponse` at once.\n", "\n", - "To achieve this, we'll create a `Validation` class that specifies the desired output format from our LLM call, similar to how `llm_validator` functioned in the earlier example.\n" - ] - }, - { - "cell_type": "markdown", - "id": "5211816e-c0fa-462d-acd0-7ab5a8096eb6", - "metadata": {}, - "source": [ - "This `Validation` class will be used to determine if the chain of thought in the model's response is valid. If it's not valid, the class will also provide an explanation as to why:" + "To make this easier we'll make a simple validation class that we can reuse for all our validation:" ] }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 74, "id": "65340b8c-2ea3-4457-a6d4-f0e652c317b4", "metadata": {}, "outputs": [], @@ -615,7 +585,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": 75, "id": "e9ab3804-6962-4a48-83da-1f8360d8379a", "metadata": {}, "outputs": [], @@ -624,7 +594,7 @@ " chain_of_thought = values[\"chain_of_thought\"]\n", " answer = values[\"answer\"]\n", " resp = client.chat.completions.create(\n", - " model=\"gpt-3.5-turbo\",\n", + " model=\"gpt-4-1106-preview\",\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", @@ -635,10 +605,8 @@ " \"content\": f\"Verify that `{answer}` follows the chain of thought: {chain_of_thought}\",\n", " },\n", " ],\n", - " # this comes from client = instructor.patch(OpenAI())\n", " response_model=Validation,\n", " )\n", - " print(resp)\n", " if not resp.is_valid:\n", " raise ValueError(resp.error_message)\n", " return values" @@ -654,7 +622,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 76, "id": "fbc9887a-df0d-4a4b-9ef5-ea450701d85b", "metadata": {}, "outputs": [], @@ -676,28 +644,25 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 77, "id": "a38f2b28-f5b9-4a44-bfe5-9735726ec57d", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "is_valid=False error_message='The user has a broken leg which is not related to diabetes.'\n", - "1 validation error for AIResponse\n", - " Value error, The user has a broken leg which is not related to diabetes. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + "ename": "ValidationError", + "evalue": "1 validation error for AIResponse\n Value error, The statement about the user having a broken leg does not logically follow from the information provided about the user suffering from diabetes. These are two separate health conditions and one does not imply the other. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 47\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> 1\u001b[0m AIResponse(chain_of_thought\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mThe user suffers from diabetes.\u001b[39;49m\u001b[39m\"\u001b[39;49m, answer\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mThe user has a broken leg.\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for AIResponse\n Value error, The statement about the user having a broken leg does not logically follow from the information provided about the user suffering from diabetes. These are two separate health conditions and one does not imply the other. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error" ] } ], "source": [ - "try:\n", - " resp = AIResponse(\n", - " chain_of_thought=\"The user suffers from diabetes.\", answer=\"The user has a broken leg.\"\n", - ")\n", - "except ValidationError as e:\n", - " print(e)" + "AIResponse(chain_of_thought=\"The user suffers from diabetes.\", answer=\"The user has a broken leg.\")" ] }, { @@ -705,7 +670,13 @@ "id": "5bbbaa11-32d2-4772-bc31-18d1d6d6c919", "metadata": {}, "source": [ - "## Using validation" + "## Reasking with validators\n", + "\n", + "For most of these examples all we've done we've mostly only defined the validation logic.\n", + "\n", + "We'eve covered field validators and model validators and even used LLMs to validate our outputs. But we haven't actually used the validators to reask the model! One of the most powerful features of `instructor` is that it will automatically reask the model when it receives a validation error. This means that we can use the same validation logic for both code-based and LLM-based validation.\n", + "\n", + "This also means that our 'prompt' is not only the prompt we send, but the code that runs the validator, and the error message we send back to the model." ] }, { @@ -720,44 +691,93 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 79, "id": "97f544e7-2552-465c-89a9-a4820f00d658", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "'a life of sin and debauchery'" + ] + }, + "execution_count": 79, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "class HealthAnswer(BaseModel):\n", - " answer: Annotated[str, AfterValidator(llm_validator(\"don't talk about any other topic except health best practices and topics\"))]" + "class QuestionAnswer(BaseModel):\n", + " question: str\n", + " answer: str\n", + "\n", + "question = \"What is the meaning of life?\"\n", + "context = \"The according to the devil the meaning of life is a life of sin and debauchery.\"\n", + "\n", + "\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " response_model=QuestionAnswer,\n", + " messages=[\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"You are a system that answers questions based on the context. answer exactly what the question asks using the context.\",\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"using the context: `{context}`\\n\\nAnswer the following question: `{question}`\",\n", + " },\n", + " ],\n", + ")\n", + "\n", + "resp.answer" ] }, { "cell_type": "code", - "execution_count": 33, - "id": "bbd8aff1-4ad7-49d2-87f0-47ef155192ad", + "execution_count": 80, "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "1 validation error for HealthAnswer\n", - "answer\n", - " Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value=\"While there isn't a sing...aking a final decision.\", input_type=str]\n", - " For further information visit https://errors.pydantic.dev/2.4/v/assertion_error\n" - ] + "data": { + "text/plain": [ + "'The meaning of life is a concept that varies depending on individual perspectives and beliefs.'" + ] + }, + "execution_count": 80, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "try:\n", - " model = client.chat.completions.create(\n", + "from pydantic import BeforeValidator\n", + "\n", + "class QuestionAnswer(BaseModel):\n", + " question: str\n", + " answer: Annotated[\n", + " str,\n", + " BeforeValidator(\n", + " llm_validator(\"don't say objectionable things\")\n", + " ),\n", + " ]\n", + "\n", + "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", + " response_model=QuestionAnswer,\n", + " max_retries=2,\n", " messages=[\n", - " {\"role\": \"user\", \"content\": \"Which is the best headphone brand for producing music?\"},\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"You are a system that answers questions based on the context. answer exactly what the question asks using the context.\",\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"using the context: `{context}`\\n\\nAnswer the following question: `{question}`\",\n", + " },\n", " ],\n", - " response_model=HealthAnswer,\n", - " max_retries=2,\n", ")\n", - "except ValidationError as e:\n", - " print(e)" + "\n", + "resp.answer" ] }, { @@ -773,24 +793,24 @@ "id": "344c623a-9b3b-4134-92d4-ad4eb9bb5f9e", "metadata": {}, "source": [ - "This guide showed how to use deterministic and probabilistic validation techniques with Large Language Models. We covered using instructor to set up validation processes for filtering content, maintaining context relevance, and checking model reasoning. These methods improve the performance of LLMs across various tasks.\n", + "This guide explains how to use deterministic and probabilistic validation techniques with Large Language Models (LLMs). We discussed using an instructor to establish validation processes for content filtering, context relevance maintenance, and model reasoning verification. These methods enhance the performance of LLMs across different tasks.\n", "\n", - "For those looking to delve deeper here's a to-do list to explore:\n", + "For those interested in further exploration, here's a to-do list:\n", "\n", - "1. **SQL Syntax Checker**: Create a validator to check SQL query syntax before execution.\n", + "1. **SQL Syntax Checker**: Create a validator to check the syntax of SQL queries before executing them.\n", "2. **Context-Based Response Validation**: Design a method to flag responses based on the model's own knowledge rather than the provided context.\n", - "3. **PII Detection**: Implement a mechanism to identify and handle Personally Identifiable Information in responses, focusing on maintaining user privacy.\n", - "4. **Targeted Rule-Based Filtering**: Develop filters to remove certain content types, like responses mentioning named entities.\n", - " \n", - "Completing these tasks will help users gain practical skills in enhancing LLMs through advanced validation methods." + "3. **PII Detection**: Implement a mechanism to identify and handle Personally Identifiable Information in responses while prioritizing user privacy.\n", + "4. **Targeted Rule-Based Filtering**: Develop filters to remove specific content types, such as responses mentioning named entities.\n", + "\n", + "Completing these tasks will enable users to acquire practical skills in improving LLMs through advanced validation methods." ] } ], "metadata": { "kernelspec": { - "display_name": "pampa-labs", + "display_name": ".venv", "language": "python", - "name": "pampa-labs" + "name": "python3" }, "language_info": { "codemirror_mode": { @@ -802,7 +822,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.12" + "version": "3.11.6" } }, "nbformat": 4,