Skip to content

Commit

Permalink
Merge a755af6 into 7033488
Browse files Browse the repository at this point in the history
  • Loading branch information
@fpingham
fpingham committed Feb 17, 2024
2 parents 7033488 + a755af6 commit b2d108c
Show file tree
Hide file tree
Showing 3 changed files with 364 additions and 0 deletions.
113 changes: 113 additions & 0 deletions docs/examples/extract_slides.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Data extraction from slides

In this guide, we demonstrate how to extract data from slides.

!!! tips "Motivation"

When we want to translate key information from slides into structured data, simply isolating the text and running extraction might not be enough. Sometimes the important data is in the images on the slides, so we should consider including them in our extraction pipeline.

## Defining the necessary Data Structures

Let's say we want to extract the competitors from various presentations and categorize them according to their respective industries.

Our data model will have `Industry` which will be a list of `Competitor`'s for a specific industry, and `Competition` which will aggregate the competitors for all the industries.

```python
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Optional, List

class Competitor(BaseModel):
name: str
features: Optional[List[str]]


# Define models
class Industry(BaseModel):
"""
Represents competitors from a specific industry extracted from an image using AI.
"""

name: str = Field(
description="The name of the industry"
)
competitor_list: List[Competitor] = Field(
description="A list of competitors for this industry"
)

class Competition(BaseModel):
"""
This class serves as a structured representation of
competitors and their qualities.
"""

industry_list: List[IndustryCompetition] = Field(
description="A list of industries and their competitors"
)
```

## Competitors extraction

To extract competitors from slides we will define a function which will read images from urls and extract the relevant information from them.

```python
import instructor
from openai import OpenAI

# Apply the patch to the OpenAI client
# enables response_model keyword
client = instructor.patch(
OpenAI(), mode=instructor.Mode.MD_JSON
)

# Define functions
def read_images(image_urls: List[str]) -> Competition:
"""
Given a list of image URLs, identify the competitors in the images.
"""
return client.chat.completions.create(
model="gpt-4-vision-preview",
response_model=Competition,
max_tokens=2048,
temperature=0,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Identify competitors and generate key features for each competitor.",
},
*[
{"type": "image_url", "image_url": {"url": url}}
for url in image_urls
],
],
}
],
)
```

## Execution

Finally, we will run the previous function with a few sample slides to see the data extractor in action.

As we can see, our model extracted the relevant information for each competitor regardless of how this information was formatted in the original presentations.

```python
url = [
'https://miro.medium.com/v2/resize:fit:1276/0*h1Rsv-fZWzQUyOkt',
'https://earlygame.vc/wp-content/uploads/2020/06/startup-pitch-deck-5.jpg'
]
model = read_images(url)
print(model.model_json_dump(indent=2))
```
industry_list=[

Industry(name='Accommodation and Hospitality', competitor_list=[Competitor(name='CouchSurfing', features=['Affordable', 'Online Transaction']), Competitor(name='Craigslist', features=['Affordable', 'Offline Transaction']), Competitor(name='BedandBreakfast.com', features=['Affordable', 'Offline Transaction']), Competitor(name='AirBed&Breakfast', features=['Affordable', 'Online Transaction']), Competitor(name='Hostels.com', features=['Affordable', 'Online Transaction']), Competitor(name='VRBO', features=['Expensive', 'Offline Transaction']), Competitor(name='Rentahome', features=['Expensive', 'Online Transaction']), Competitor(name='Orbitz', features=['Expensive', 'Online Transaction']), Competitor(name='Hotels.com', features=['Expensive', 'Online Transaction'])]),

Industry(name='Wine E-commerce', competitor_list=[Competitor(name='WineSimple', features=['Ecommerce Retailers', 'True Personalized Selections', 'Brand Name Wine', 'No Inventory Cost', 'Target Mass Market']), Competitor(name='NakedWines', features=['Ecommerce Retailers', 'Target Mass Market']), Competitor(name='Club W', features=['Ecommerce Retailers', 'Brand Name Wine', 'Target Mass Market']), Competitor(name='Tasting Room', features=['Ecommerce Retailers', 'True Personalized Selections', 'Brand Name Wine']), Competitor(name='Drync', features=['Ecommerce Retailers', 'True Personalized Selections', 'No Inventory Cost']), Competitor(name='Hello Vino', features=['Ecommerce Retailers', 'Brand Name Wine', 'Target Mass Market'])])

]
```
```
250 changes: 250 additions & 0 deletions examples/vision/slides.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
import json
import logging
import sys
from typing import List, Optional

from dotenv import find_dotenv, load_dotenv
from openai import OpenAI
from pydantic import BaseModel, Field
from rich import print as rprint

import instructor

load_dotenv(find_dotenv())

IMAGE_FILE = "image-file.txt" # file with all the images to be processed

# Add logger
logging.basicConfig()
logger = logging.getLogger("app")
logger.setLevel("INFO")

class Competitor(BaseModel):
name: str
features: Optional[List[str]]


# Define models
class Industry(BaseModel):
"""
Represents competitors from a specific industry extracted from an image using AI.
"""

name: str = Field(
description="The name of the industry"
)
competitor_list: List[Competitor] = Field(
description="A list of competitors for this industry"
)

class Competition(BaseModel):
"""
Represents competitors extracted from an image using AI.
This class serves as a structured representation of
competitors and their qualities.
"""

industry_list: List[Industry] = Field(
description="A list of industries and their competitors"
)

# Define clients
client_image = instructor.patch(
OpenAI(), mode=instructor.Mode.MD_JSON
)

# Define functions
def read_images(image_urls: List[str]) -> Competition:
"""
Given a list of image URLs, identify the competitors in the images.
"""

logger.info(f"Identifying competitors in images... {len(image_urls)} images")

return client_image.chat.completions.create(
model="gpt-4-vision-preview",
response_model=Competition,
max_tokens=2048,
temperature=0,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Identify competitors and generate key features for each competitor.",
},
*[
{"type": "image_url", "image_url": {"url": url}}
for url in image_urls
],
],
}
],
)



def process_and_identify_competitors():
"""
Main function to process the image list file and identify competitors.
"""

logger.info("Starting app...")

try:
with open(IMAGE_FILE, "r") as file:
logger.info(f"Reading images from file: {IMAGE_FILE}")
image_list = file.read().splitlines()
logger.info(f"{len(image_list)} images read from file: {IMAGE_FILE}")
except Exception as e:
logger.error(f"Error reading images from file: {IMAGE_FILE}")
logger.error(e)
sys.exit(1)

competitors = read_images(image_list)

rprint(f"[green]{len(competitors.industry_list)} industries identified:[/green]")
for industry in competitors.industry_list:
rprint(f"[green]{industry.name}[/green]")
rprint(f"[blue]Features: {industry.competitor_list}[/blue]")

logger.info("Writing results to file...")

with open("results.json", "w") as f:
json.dump(
{
"competitors": competitors.model_dump(),
},
f,
indent=4,
)

if __name__ == "__main__":
process_and_identify_competitors()

"""
Example output:
{
"competitors": {
"industry_list": [
{
"name": "Accommodation and Hospitality",
"competitor_list": [
{
"name": "craigslist",
"features": [
"Transactions Offline",
"Inexpensive"
]
},
{
"name": "couchsurfing",
"features": [
"Transactions Offline",
"Inexpensive"
]
},
{
"name": "BedandBreakfast.com",
"features": [
"Transactions Offline",
"Inexpensive"
]
},
{
"name": "airbnb",
"features": [
"Transactions Online",
"Inexpensive"
]
},
{
"name": "HOSTELS.com",
"features": [
"Transactions Online",
"Inexpensive"
]
},
{
"name": "VRBO",
"features": [
"Transactions Offline",
"Costly"
]
},
{
"name": "Rentahome",
"features": [
"Transactions Online",
"Costly"
]
},
{
"name": "Orbitz",
"features": [
"Transactions Online",
"Costly"
]
},
{
"name": "Hotels.com",
"features": [
"Transactions Online",
"Costly"
]
}
]
},
{
"name": "E-commerce Wine Retailers",
"competitor_list": [
{
"name": "winesimple",
"features": [
"Ecommerce Retailers",
"True Personalized Selections",
"Brand Name Wine",
"No Inventory Cost",
"Target Mass Market"
]
},
{
"name": "nakedwines.com",
"features": [
"Ecommerce Retailers",
"Target Mass Market"
]
},
{
"name": "Club W",
"features": [
"Ecommerce Retailers",
"Brand Name Wine",
"Target Mass Market"
]
},
{
"name": "Tasting Room",
"features": [
"Ecommerce Retailers",
"True Personalized Selections",
"Brand Name Wine"
]
},
{
"name": "hellovino",
"features": [
"Ecommerce Retailers",
"True Personalized Selections",
"No Inventory Cost",
"Target Mass Market"
]
}
]
}
]
}
}
"""
1 change: 1 addition & 0 deletions mkdocs.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ nav:
- Batch Classification (User Defined): 'examples/batch_classification.md'
- LLM Self Critique: 'examples/self_critique.md'
- Extracting Tables with GPT-V: 'examples/extracting_tables.md'
- Extracting From Slides with GPT-V: 'examples/extract_slides.md'
- Content Moderation: 'examples/moderation.md'
- Citing Sources (RAG): 'examples/exact_citations.md'
- Extracting Knowledge Graphs: 'examples/knowledge_graph.md'
Expand Down

0 comments on commit b2d108c

Please sign in to comment.