add agent docs (#6866)

Co-authored-by: Simon Suo <simonsdsuo@gmail.com>

docs/core_modules/agent_modules/agents/modules.md

@@ -0,0 +1,27 @@
+# Module Guides
+
+These guide provide an overview of how to use our agent classes.
+
+For more detailed guides on how to use specific tools, check out our [tools module guides]().
+
+## OpenAI Agent
+```{toctree}
+---
+maxdepth: 1
+---
+/examples/agent/openai_agent.ipynb
+/examples/agent/openai_agent_with_query_engine.ipynb
+/examples/agent/openai_agent_retrieval.ipynb
+/examples/agent/openai_agent_query_cookbook.ipynb
+/examples/agent/openai_agent_query_plan.ipynb
+/examples/agent/openai_agent_context_retrieval.ipynb
+```
+
+## ReAct Agent
+```{toctree}
+---
+maxdepth: 1
+---
+/examples/agent/react_agent_with_query_engine.ipynb
+```
+

docs/core_modules/agent_modules/agents/root.md

@@ -0,0 +1,69 @@
+# Data Agents
+
+## Concept
+Data Agents are LLM-powered knowledge workers in LlamaIndex that can intelligently perform various tasks over your data, in both a “read” and “write” function. They are capable of the following:
+
+- Perform automated search and retrieval over different types of data - unstructured, semi-structured, and structured.
+- Calling any external service API in a structured fashion, and processing the response + storing it for later.
+
+In that sense, agents are a step beyond our [query engines](/core_modules/query_modules/query_engine/root.md) in that they can not only "read" from a static source of data, but can dynamically ingest and modify data from a variety of different tools.
+
+Building a data agent requires the following core components:
+
+- A reasoning loop
+- Tool abstractions
+
+A data agent is initialized with set of APIs, or Tools, to interact with; these APIs can be called by the agent to return information or modify state. Given an input task, the data agent uses a reasoning loop to decide which tools to use, in which sequence, and the parameters to call each tool.
+
+### Reasoning Loop
+The reasoning loop depends on the type of agent. We have support for the following agents: 
+- OpenAI Function agent (built on top of the OpenAI Function API)
+- a ReAct agent (which works across any chat/text completion endpoint).
+
+### Tool Abstractions
+
+You can learn more about our Tool abstractions in our [Tools section](/core_modules/agent_modules/tools/root.md).
+
+### Blog Post
+
+For full details, please check out our detailed [blog post]().
+
+
+## Usage Pattern
+
+Data agents can be used in the following manner (the example uses the OpenAI Function API)
+```python
+from llama_index.agent import OpenAIAgent
+from llama_index.llms import OpenAI
+
+# import and define tools
+...
+
+# initialize llm
+llm = OpenAI(model="gpt-3.5-turbo-0613")
+
+# initialize openai agent
+agent = OpenAIAgent.from_tools(tools, llm=llm, verbose=True)
+```
+
+See our usage pattern guide for more details.
+```{toctree}
+---
+maxdepth: 1
+---
+usage_pattern.md
+```
+
+## Modules
+
+Learn more about our different agent types in our module guides below.
+
+Also take a look at our [tools section](/core_modules/agent_modules/tools/root.md)!
+
+```{toctree}
+---
+maxdepth: 2
+---
+modules.md
+```
+

docs/core_modules/agent_modules/agents/usage_pattern.md

@@ -0,0 +1,193 @@
+# Usage Pattern
+
+## Get Started
+
+An agent is initialized from a set of Tools. Here's an example of instantiating a ReAct
+agent from a set of Tools.
+
+```python
+from llama_index.tools import FunctionTool
+from llama_index.llms import OpenAI
+from llama_index.agent import ReActAgent
+
+# define sample Tool
+def multiply(a: int, b: int) -> int:
+    """Multiple two integers and returns the result integer"""
+    return a * b
+
+multiply_tool = FunctionTool.from_defaults(fn=multiply)
+
+# initialize llm
+llm = OpenAI(model="gpt-3.5-turbo-0613")
+
+# initialize ReAct agent
+agent = ReActAgent.from_tools([multiply_tool], llm=llm, verbose=True)
+```
+
+An agent supports both `chat` and `query` endpoints, inheriting from our `ChatEngine` and `QueryEngine` respectively.
+
+Example usage:
+```python
+agent.chat("What is 2123 * 215123")
+```
+
+
+## Query Engine Tools
+
+It is easy to wrap query engines as tools for an agent as well. Simply do the following:
+
+```python
+
+from llama_index.agent import ReActAgent
+from llama_index.tools import QueryEngineTool
+
+# NOTE: lyft_index and uber_index are both SimpleVectorIndex instances
+lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
+uber_engine = uber_index.as_query_engine(similarity_top_k=3)
+
+query_engine_tools = [
+    QueryEngineTool(
+        query_engine=lyft_engine,
+        metadata=ToolMetadata(
+            name="lyft_10k",
+            description="Provides information about Lyft financials for year 2021. "
+            "Use a detailed plain text question as input to the tool.",
+        ),
+    ),
+    QueryEngineTool(
+        query_engine=uber_engine,
+        metadata=ToolMetadata(
+            name="uber_10k",
+            description="Provides information about Uber financials for year 2021. "
+            "Use a detailed plain text question as input to the tool.",
+        ),
+    ),
+]
+
+# initialize ReAct agent
+agent = ReActAgent.from_tools(query_engine_tools, llm=llm, verbose=True)
+
+```
+
+## Use other agents as Tools
+
+A nifty feature of our agents is that since they inherit from `BaseQueryEngine`, you can easily define other agents as tools
+through our `QueryEngineTool`. 
+
+```python
+from llama_index.tools import QueryEngineTool
+
+query_engine_tools = [
+    QueryEngineTool(
+        query_engine=sql_agent,
+        metadata=ToolMetadata(
+            name="sql_agent",
+            description="Agent that can execute SQL queries."
+        ),
+    ),
+    QueryEngineTool(
+        query_engine=gmail_agent,
+        metadata=ToolMetadata(
+            name="gmail_agent",
+            description="Tool that can send emails on Gmail."
+        ),
+    ),
+]
+
+outer_agent = ReActAgent.from_tools(query_engine_tools, llm=llm, verbose=True)
+```
+
+## Advanced Concepts (for `OpenAIAgent`, in beta)
+
+You can also use agents in more advanced settings. For instance, being able to retrieve tools from an index during query-time, and
+being able to perform query planning over an existing set of Tools.
+
+These are largely implemented with our `OpenAIAgent` classes (which depend on the OpenAI Function API). Support
+for our more general `ReActAgent` is something we're actively investigating.
+
+NOTE: these are largely still in beta. The abstractions may change and become more general over time.
+
+### Function Retrieval Agents
+
+If the set of Tools is very large, you can create an `ObjectIndex` to index the tools, and then pass in an `ObjectRetriever` to the agent during query-time, to first dynamically retrieve the relevant tools before having the agent pick from the candidate tools.
+
+We first build an `ObjectIndex` over an existing set of Tools.
+
+```python
+# define an "object" index over these tools
+from llama_index import VectorStoreIndex
+from llama_index.objects import ObjectIndex, SimpleToolNodeMapping
+
+tool_mapping = SimpleToolNodeMapping.from_objects(all_tools)
+obj_index = ObjectIndex.from_objects(
+    all_tools,
+    tool_mapping,
+    VectorStoreIndex,
+)
+```
+
+We then define our `FnRetrieverOpenAIAgent`:
+
+```python
+from llama_index.agent import FnRetrieverOpenAIAgent
+
+agent = FnRetrieverOpenAIAgent.from_retriever(obj_index.as_retriever(), verbose=True)
+```
+
+### Context Retrieval Agents
+
+Our context-augmented OpenAI Agent will always perform retrieval before calling any tools.
+
+This helps to provide additional context that can help the agent better pick Tools, versus
+just trying to make a decision without any context.
+
+```python
+from llama_index.schema import Document
+from llama_index.agent import ContextRetrieverOpenAIAgent
+
+
+# toy index - stores a list of abbreviations
+texts = [
+    "Abbrevation: X = Revenue",
+    "Abbrevation: YZ = Risk Factors",
+    "Abbreviation: Z = Costs",
+]
+docs = [Document(text=t) for t in texts]
+context_index = VectorStoreIndex.from_documents(docs)
+
+# add context agent
+context_agent = ContextRetrieverOpenAIAgent.from_tools_and_retriever(
+    query_engine_tools, context_index.as_retriever(similarity_top_k=1), verbose=True
+)
+response = context_agent.chat("What is the YZ of March 2022?")
+```
+
+### Query Planning
+
+OpenAI Function Agents can be capable of advanced query planning. The trick is to provide the agent
+with a `QueryPlanTool` - if the agent calls the QueryPlanTool, it is forced to infer a full Pydantic schema representing a query
+plan over a set of subtools.
+
+```python
+# define query plan tool
+from llama_index.tools import QueryPlanTool
+from llama_index import get_response_synthesizer
+
+response_synthesizer = get_response_synthesizer(service_context=service_context)
+query_plan_tool = QueryPlanTool.from_defaults(
+    query_engine_tools=[query_tool_sept, query_tool_june, query_tool_march],
+    response_synthesizer=response_synthesizer,
+)
+
+# initialize agent
+agent = OpenAIAgent.from_tools(
+    [query_plan_tool],
+    max_function_calls=10,
+    llm=OpenAI(temperature=0, model="gpt-4-0613"),
+    verbose=True,
+)
+
+# should output a query plan to call march, june, and september tools
+response = agent.query("Analyze Uber revenue growth in March, June, and September")
+
+```

docs/core_modules/agent_modules/tools/llamahub_tools_guide.md

@@ -0,0 +1,65 @@
+# LlamaHub Tools Guide
+
+We offer a rich set of Tool Specs that are offered through [LlamaHub](https://llamahub.ai/) 🦙. 
+![](/_static/data_connectors/llamahub.png)
+
+These tool specs represent an initial curated list of services that an agent can interact with and enrich its capability to perform different actions. 
+
+We also provide a list of **utility tools** that help to abstract away pain points when designing agents to interact with different API services that return large amounts of data.
+
+## Tool Specs
+
+Coming soon! 
+
+## Utility Tools
+
+Oftentimes, directly querying an API can return a massive volume of data, which on its own may overflow the context window of the LLM (or at the very least unnecessarily increase the number of tokens that you are using). 
+
+To tackle this, we’ve provided an initial set of “utility tools” in LlamaHub Tools - utility tools are not conceptually tied to a given service (e.g. Gmail, Notion), but rather can augment the capabilities of existing Tools. In this particular case, utility tools help to abstract away common patterns of needing to cache/index and query data that’s returned from any API request.
+
+Let’s walk through our two main utility tools below.
+
+### OnDemandLoaderTool
+
+This tool turns any existing LlamaIndex data loader ( `BaseReader` class) into a tool that an agent can use. The tool can be called with all the parameters needed to trigger `load_data` from the data loader, along with a natural language query string. During execution, we first load data from the data loader, index it (for instance with a vector store), and then query it “on-demand”. All three of these steps happen in a single tool call.
+
+Oftentimes this can be preferable to figuring out how to load and index API data yourself. While this may allow for data reusability, oftentimes users just need an ad-hoc index to abstract away prompt window limitations for any API call. 
+
+A usage example is given below:
+
+```python
+from llama_hub.wikipedia.base import WikipediaReader
+from llama_hub.tools.on_demand_loader_tool import OnDemandLoaderTool
+
+tool = OnDemandLoaderTool.from_defaults(
+	reader,
+	name="Wikipedia Tool",
+	description="A tool for loading data and querying articles from Wikipedia"
+)
+```
+
+### LoadAndSearchToolSpec
+
+The LoadAndSearchToolSpec takes in any existing Tool as input. As a tool spec, it implements `to_tool_list` , and when that function is called, two tools are returned: a `load` tool and then a `search` tool.
+
+The `load` Tool execution would call the underlying Tool, and the index the output (by default with a vector index). The `search` Tool execution would take in a query string as input and call the underlying index.
+
+This is helpful for any API endpoint that will by default return large volumes of data - for instance our WikipediaToolSpec will by default return entire Wikipedia pages, which will easily overflow most LLM context windows.
+
+Example usage is shown below:
+
+```python
+from llama_hub.tools.tool_spec.wikipedia.base import WikipediaToolSpec
+from llama_hub.tools.tool_spec.load_and_search.base import LoadAndSearchToolSpec
+
+wiki_spec = WikipediaToolSpec()
+# Get the search wikipedia tool
+tool = wiki_spec.to_tool_list()[1]
+
+# Create the Agent with load/search tools
+agent = OpenAIAgent.from_tools(
+ LoadAndSearchToolSpec.from_defaults(
+    tool
+ ).to_tool_list(), verbose=True
+)
+```