-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VectorStoreClient sometimes returns "I could not find an answer" #637
Comments
Here was the original thread that sparked his issue: https://discord.com/channels/1096466116672487547/1140674107600551987/1200638078134517800 |
What is your priority on solving this issue? |
This seems to be a combination of the phrasing of I would suggest either updating the Here is a working version for me: from dotenv import load_dotenv
from pathlib import Path
from griptape.structures import Agent
from griptape.tools import VectorStoreClient
from griptape.loaders import TextLoader
from griptape.engines import VectorQueryEngine
from griptape.drivers import (
LocalVectorStoreDriver,
OpenAiChatPromptDriver,
OpenAiEmbeddingDriver,
)
load_dotenv()
namespace = "lunchtime"
# Create an engine
engine = VectorQueryEngine(
prompt_driver=OpenAiChatPromptDriver(model="gpt-4"),
vector_store_driver=LocalVectorStoreDriver(
embedding_driver=OpenAiEmbeddingDriver()
),
)
engine.upsert_text_artifacts(
TextLoader().load(Path("lunchtime.md")), namespace=namespace
)
vector_db = VectorStoreClient(
description="This DB has helpful information",
query_engine=engine,
namespace=namespace,
off_prompt=False,
)
agent = Agent(tools=[vector_db])
agent.run("What's for dinner monday?")
agent.run("What's for dinner?")
|
@collindutter from my testing this is still a problem and the fix only works in simple cases but the proposed solution doesn't work for more complicated examples that spawned the reason why this issue was opened. I think this issue was closed perhaps prematurely. |
using the attached file, and this simple example: from dotenv import load_dotenv
from pathlib import Path
import logging
from griptape.structures import Agent
from griptape.utils import Chat
from griptape.tools import VectorStoreClient
from griptape.drivers import (
LocalVectorStoreDriver,
OpenAiEmbeddingDriver,
OpenAiChatPromptDriver
)
from griptape.engines import VectorQueryEngine
from griptape.loaders import TextLoader
from griptape.rules import Rule, Ruleset
load_dotenv()
# Upsert documentation into the vector database
namespace = "shotgrid_api"
if __name__ == '__main__':
# Create the query engine
engine = VectorQueryEngine(
vector_store_driver=LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver()),
prompt_driver=OpenAiChatPromptDriver(model="gpt-4"))
# Load the API documentation
examples = TextLoader().load(Path(r'shotgun-api-examples.md'))
engine.upsert_text_artifacts(examples, namespace)
# Instantiate the Vector Store Client
vector_store_tool = VectorStoreClient(
description="Contains information about ShotGrid api. Use it to help with ShotGrid client requests.",
query_engine=engine,
namespace=namespace,
off_prompt=False,
)
# Create the ruleset
shotgrid_agent_ruleset = Ruleset(
name="ShotGrid Agent",
rules=[
Rule("Only Use the Vectorstore Client to answer questions about the shotgrid api"),
],
)
# Instantiate the agent
agent = Agent(
logger_level=logging.ERROR,
tools=[vector_store_tool],
rulesets=[shotgrid_agent_ruleset],
stream=True
)
# Start chatting
Chat(agent).start() if you ask the question "How to find active projects that are not archived in ShotGrid?" you will reliably get an answer that tells you you need to use the following filter syntax |
however if you use something more complicated that has other tools, more rules, and more information in the VectorStoreClient then the results become more unreliable from dotenv import load_dotenv
from textwrap import dedent
from pathlib import Path
import os
import logging
from griptape.structures import Agent
from griptape.utils import Chat
from griptape.tools import DateTime, VectorStoreClient, Calculator
from griptape.drivers import (
LocalVectorStoreDriver,
OpenAiEmbeddingDriver,
OpenAiChatPromptDriver
)
from griptape.engines import VectorQueryEngine
from griptape.loaders import WebLoader, TextLoader
from griptape.rules import Rule, Ruleset
from shotgrid_tool import ShotGridTool
load_dotenv()
# Upsert documentation into the vector database
namespace = "shotgrid_api"
if __name__ == '__main__':
# Create the query engine
engine = VectorQueryEngine(
vector_store_driver=LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver()),
prompt_driver=OpenAiChatPromptDriver(model="gpt-4"))
# API Documentation
shotgrid_api_urls = [
"https://developers.shotgridsoftware.com/python-api/reference.html",
"https://developers.shotgridsoftware.com/python-api/reference.html#filter-syntax",
"https://developers.shotgridsoftware.com/python-api/cookbook/usage_tips.html",
"https://developers.shotgridsoftware.com/python-api/cookbook/attachments.html",
"https://developers.shotgridsoftware.com/python-api/cookbook/tasks/updating_tasks.html",
"https://developers.shotgridsoftware.com/python-api/cookbook/tasks/task_dependencies.html",
"https://developers.shotgridsoftware.com/python-api/cookbook/tasks/split_tasks.html",
]
# Load the API documentation
examples = TextLoader().load(Path(r'shotgun-api-examples.md'))
engine.upsert_text_artifacts(examples, namespace)
for url in shotgrid_api_urls:
print(url)
artifacts = WebLoader().load(url)
engine.upsert_text_artifacts(artifacts, namespace)
# Instantiate the Vector Store Client
vector_store_tool = VectorStoreClient(
description="Contains information about ShotGrid api. Use it to help with ShotGrid client requests.",
query_engine=engine,
namespace=namespace,
off_prompt=False,
)
SHOTGRID_URL = os.getenv("SHOTGRID_URL")
SHOTGRID_API_KEY = os.getenv("SHOTGRID_API_KEY")
SHOTGRID_SCRIPT = "GriptapeApi"
SHOTGRID_USER = os.getenv("SHOTGRID_USER")
SHOTGRID_PASSWORD = os.getenv("SHOTGRID_PASSWORD")
# Instantiate the tool
shotgrid_tool = ShotGridTool(
base_url=SHOTGRID_URL,
api_key=SHOTGRID_API_KEY,
script_name=SHOTGRID_SCRIPT,
user_login=SHOTGRID_USER,
user_password=SHOTGRID_PASSWORD,
login_method="user",
off_prompt=False
)
# Create the ruleset
shotgrid_agent_ruleset = Ruleset(
name="ShotGrid Agent",
rules=[
Rule("Act as a studio coordinator who is an expert with Autodesk ShotGrid"),
Rule(
"Your only objective is to find and update information in ShotGrid using the ShotGridTool"
),
Rule(
dedent(
"""
For specific information about how to use ShotGrid API activities, the
VectorStoreClient should be used. Take the necessary time to consult the
VectorStoreClient to ensure the most accurate and context-aware decisions
when interacting with the ShotGridTool and API.
"""
)
),
Rule(
dedent(
"""
Always use the ShotGrid batch() method when performing multiple create, update, or delete operations in
ShotGrid. This should be done in a single request to improve efficiency and speed. Batch method uses
"request_type", "entity_type", and "entity_id". Failure to use the batch() method in these instances will
be considered a violation of the rules."""
)
),
Rule(
dedent(
"""
When asked to find, list, create, update, delete, retrieve details of entities,
update task status, update task data, list task assignments, retrieve the history of changes to
entities, manage versions of assets, manage project timelines, track project
progress, manage notes and reviews, manage user roles and permissions, or
integrate with other tools and workflows, the ShotGrid Client API is always used."""
)
),
],
)
# Instantiate the agent
agent = Agent(
logger_level=logging.ERROR,
tools=[DateTime(off_prompt=False), Calculator(off_prompt=False), shotgrid_tool, vector_store_tool],
rulesets=[shotgrid_agent_ruleset],
stream=True
)
# Start chatting
Chat(agent).start() |
@collindutter & @shhlife to test the results I've been running the application, asking the question, shutting it down, running it again, asking the same question, and repeating this process. |
Thanks for the extra context @mgcollie, and apologies for the premature closing! Will continue to play with this to see if we can reproduce and resolve for good. |
Describe the bug
When using the VectorStoreClient, it sometimes responds with "I could not find an answer" even when the answer should be able to be returned.
To Reproduce
I created a relatively simple agent that takes a data file and upserts it, then asks some questions. It should be able to figure it out, but it doesn't.
Here's the text file:
lunchtime.md
And here's the script
When you run it, when you ask "What's for dinner Monday" it responds with
I could not find an answer
. But if you ask "What's for dinner" it responds with the correctWe don't serve dinner, go home
.Expected behavior
I would expect it to return
We don't serve dinner, go home
with the first question as well as the second.Additional info
I also found if I asked "When are we having food with pork?" it would reply with "We are having food with pork on Tuesday and Thursday", but it would forget about Friday.
The text was updated successfully, but these errors were encountered: