Claude 3 Image Query Support #700

emjay07 · 2024-03-19T22:21:49Z

Adding Claude 3 Image Query support via both Anthropic and Bedrock.

image_query_driver=AnthropicImageQueryDriver(
    api_key=anthropic_api_key,
    model="claude-3-sonnet-20240229"
)

image_query_driver=AmazonBedrockImageQueryDriver(
    image_query_model_driver=BedrockClaudeImageQueryModelDriver(),
    model="anthropic.claude-3-sonnet-20240229-v1:0"
)

collindutter · 2024-03-19T23:35:43Z

Once things stabilize on a handful of PRs we should add the vision drivers to #702

pyproject.toml

griptape/drivers/image_query/anthropic_image_query_driver.py

griptape/drivers/image_query/amazon_bedrock_image_query_driver.py

collindutter · 2024-03-20T21:19:40Z

griptape/drivers/image_query/anthropic_image_query_driver.py

+    """
+
+    api_key: str = field(kw_only=True, metadata={"serializable": True})
+    model: str = field(default="claude-3-sonnet-20240229", kw_only=True, metadata={"serializable": True})


Remove default model value.

Can do, but I was curious about this. I saw we set defaults for OpenAI drivers, but not others. Why is that?

Ah I didn't realize we set in openai. That probably shouldn't be set their either. But for sake of consistency we can keep it.

@andrewfrench thoughts?

I had already removed the anthropic default. I removed the openai default as well in the name of consistency.

griptape/drivers/image_query/anthropic_image_query_driver.py

collindutter · 2024-03-20T21:21:50Z

griptape/drivers/image_query/anthropic_image_query_driver.py

+        ),
+        kw_only=True,
+    )
+    max_output_tokens: Optional[int] = field(default=4096, kw_only=True, metadata={"serializable": True})


Do we want to provide a default value here?

Anthropic requires some value to be specified. I tried with None and it gets angry. So I thought a default would be best because then it's more upfront to the customer vs. hiding it in our internal call. Thoughts?

Got it, thanks Anthropic. If we do this, we should probably standardize across the Query Drivers. OpenAi does not have one set, for instance.

Any thoughts @andrewfrench?

OpenAI's default no-value behavior is unhelpful enough as to consider max_tokens a required parameter there as well. Perhaps we should update both the OpenAI and Anthropic drivers to explicitly require this field? In either case, a refactor of the tokenizers + prompt stack to support image and media inputs will go a long way to help here.

I'm good with staying consistent across all image_query drivers and making max_output_tokens a required value for now. That means no default and put the onus on the customer, or should we provide a "middle-of-the-road" default for both OpenAI and Anthropic? I was thinking yes to still providing a default so that customers can use it out-of-the-box without needing to go deeply understand each LLMs tokenization. Thoughts?

Put max_output_tokens on the base class and made the default 256.

Let's keep it named as max_tokens for consistency with the Prompt Drivers. We can do a full rename in the future.

griptape/drivers/image_query/anthropic_image_query_driver.py

griptape/drivers/image_query_model/base_image_query_model_driver.py

griptape/drivers/image_query_model/bedrock_claude_image_query_model_driver.py

andrewfrench

Spent some time this morning playing with Claude 3 image queries with great success, awesome stuff! If we can choose a path on this conversation I think we're good to go.

emjay07 · 2024-03-21T20:03:20Z

Once things stabilize on a handful of PRs we should add the vision drivers to #702

Added these in

griptape/drivers/prompt_model/bedrock_claude_prompt_model_driver.py

griptape/drivers/image_query_model/bedrock_claude_image_query_model_driver.py

collindutter · 2024-03-21T20:10:14Z

griptape/drivers/image_query/base_image_query_driver.py

@@ -16,6 +16,7 @@
 @define
 class BaseImageQueryDriver(SerializableMixin, ExponentialBackoffMixin, ABC):
    structure: Optional[Structure] = field(default=None, kw_only=True)
+    max_output_tokens: int = field(default=256, kw_only=True, metadata={"serializable": True})


Lets keep it named as max_tokens for consistency with other Drivers.

I thought I saw a PR that updated max_tokens to max_input_tokens and max_output_tokens? Was that just when you are providing a tokenizer?

I am fine with updating this to match the rest of the drivers, but I would say max_output_tokens is better because that is explicitly what it is. Is that the same for the other drivers as well?

Agreed that max_output_tokens is probably a better field name, but the other PR was only in the context of the Tokenizers, not the Drivers.

All the other drivers have inputs as text, not images. so to clarify my question: are the other drivers that have max_tokens intended to be for the output tokens or the input tokens?

Updated back to max_tokens

CHANGELOG.md

griptape/drivers/image_query/amazon_bedrock_image_query_driver.py

…river

emjay07 force-pushed the feature/claude-3-image-query branch from b9183de to a16caa7 Compare March 19, 2024 22:54

emjay07 requested a review from a team March 19, 2024 23:32

emjay07 assigned emjay07 and unassigned emjay07 Mar 19, 2024

emjay07 marked this pull request as ready for review March 19, 2024 23:38

collindutter requested a review from andrewfrench March 19, 2024 23:44

collindutter reviewed Mar 19, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

vasinov requested changes Mar 20, 2024

View reviewed changes

griptape/drivers/image_query/anthropic_image_query_driver.py Outdated Show resolved Hide resolved

griptape/drivers/image_query/anthropic_image_query_driver.py Show resolved Hide resolved

collindutter requested changes Mar 20, 2024

View reviewed changes

emjay07 requested review from collindutter and vasinov March 20, 2024 23:19

emjay07 force-pushed the feature/claude-3-image-query branch from ea47d85 to 8277686 Compare March 21, 2024 16:28

andrewfrench reviewed Mar 21, 2024

View reviewed changes

emjay07 added 16 commits March 21, 2024 12:32

Adding anthropic image query driver

5b56352

adding base files for bedrock-claude support

c7496c0

adding bedrock driver and claude model driver

afb28a5

reformatting

0d8107c

adding tests for anthropic driver

da3e5e8

adding models tests

f46c9ca

adding max_tokens defaults back in

e5e1221

resolving merge conflicts

059a4e2

updating lock file for new dependencies

8af40e5

stripping the extra structures around the output

9c4b5f2

adding OpenAIVisionImageQueryDriver testing

685fba3

updating to use ImageArtifact's base64 method

ca814c4

updating whitespace and setting defaults

3565689

addressing pr comments for renames and minor refactors

7c38119

updating toml after merge

41157b4

updating lock file after merge

7bbdbd6

emjay07 added 4 commits March 21, 2024 12:32

updating changelog after merge

294dac1

bringing lock file back up to dev

fbee2d8

updating drivers to make max_output_tokens be required

978c8b9

updating structures and anthropic_version to static

3ee66c1

emjay07 force-pushed the feature/claude-3-image-query branch from 918282d to 3ee66c1 Compare March 21, 2024 19:57

updating changelog

8a2e640

andrewfrench previously approved these changes Mar 21, 2024

View reviewed changes

collindutter reviewed Mar 21, 2024

View reviewed changes

griptape/drivers/prompt_model/bedrock_claude_prompt_model_driver.py Outdated Show resolved Hide resolved

collindutter reviewed Mar 21, 2024

View reviewed changes

griptape/drivers/image_query_model/bedrock_claude_image_query_model_driver.py Outdated Show resolved Hide resolved

collindutter reviewed Mar 21, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

collindutter reviewed Mar 21, 2024

View reviewed changes

griptape/drivers/image_query/amazon_bedrock_image_query_driver.py Outdated Show resolved Hide resolved

nit in changelog and surfacing better errors in bedrock image query d…

5528638

…river

emjay07 dismissed andrewfrench’s stale review via 5528638 March 21, 2024 20:23

emjay07 added 4 commits March 21, 2024 13:28

updating name for static field

ec6e7ea

fixing naming in tests

91e3e0e

renaming to max_tokens

b70e373

removing default model from openai's image query driver

f8fc43a

emjay07 requested review from collindutter and removed request for vasinov March 21, 2024 21:39

vasinov approved these changes Mar 21, 2024

View reviewed changes

andrewfrench approved these changes Mar 21, 2024

View reviewed changes

collindutter approved these changes Mar 21, 2024

View reviewed changes

emjay07 merged commit f96af2e into dev Mar 21, 2024
6 checks passed

emjay07 deleted the feature/claude-3-image-query branch March 21, 2024 22:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Claude 3 Image Query Support #700

Claude 3 Image Query Support #700

emjay07 commented Mar 19, 2024

collindutter commented Mar 19, 2024

collindutter Mar 20, 2024

emjay07 Mar 20, 2024

collindutter Mar 21, 2024

emjay07 Mar 21, 2024 •

edited

Loading

collindutter Mar 20, 2024

emjay07 Mar 20, 2024

collindutter Mar 20, 2024

andrewfrench Mar 21, 2024 •

edited

Loading

emjay07 Mar 21, 2024

emjay07 Mar 21, 2024

collindutter Mar 21, 2024

emjay07 Mar 21, 2024

andrewfrench left a comment

emjay07 commented Mar 21, 2024

collindutter Mar 21, 2024

emjay07 Mar 21, 2024

collindutter Mar 21, 2024

emjay07 Mar 21, 2024

emjay07 Mar 21, 2024

Claude 3 Image Query Support #700

Claude 3 Image Query Support #700

Conversation

emjay07 commented Mar 19, 2024

collindutter commented Mar 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emjay07 Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewfrench Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewfrench left a comment

Choose a reason for hiding this comment

emjay07 commented Mar 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emjay07 Mar 21, 2024 •

edited

Loading

andrewfrench Mar 21, 2024 •

edited

Loading