GPT Researcher is an autonomous agent designed for comprehensive online research on a variety of tasks.
The agent can produce detailed, factual and unbiased research reports, with customization options for focusing on relevant resources, outlines, and lessons. Inspired by the recent Plan-and-Solve and RAG papers, GPT Researcher addresses issues of speed, determinism and reliability, offering a more stable performance and increased speed through parallelized agent work, as opposed to synchronous operations.
Our mission is to empower individuals and organizations with accurate, unbiased, and factual information by leveraging the power of AI.
- To form objective conclusions for manual research tasks can take time, sometimes weeks to find the right resources and information.
- Current LLMs are trained on past and outdated information, with heavy risks of hallucinations, making them almost irrelevant for research tasks.
- Services that enable web search (such as ChatGPT + Web Plugin), only consider limited sources and content that in some cases result in superficial and biased answers.
- Using only a selection of web sources can create bias in determining the right conclusions for research tasks.
demo.mp4
The main idea is to run "planner" and "execution" agents, whereas the planner generates questions to research, and the execution agents seek the most related information based on each generated research question. Finally, the planner filters and aggregates all related information and creates a research report.
The agents leverage both gpt3.5-turbo and gpt-4-turbo (128K context) to complete a research task. We optimize for costs using each only when necessary. The average research task takes around 3 minutes to complete, and costs ~$0.1.
More specifically:
- Create a domain specific agent based on research query or task.
- Generate a set of research questions that together form an objective opinion on any given task.
- For each research question, trigger a crawler agent that scrapes online resources for information relevant to the given task.
- For each scraped resources, summarize based on relevant information and keep track of its sources.
- Finally, filter and aggregate all summarized sources and generate a final research report.
- π Generate research, outlines, resources and lessons reports
- π Can generate long and detailed research reports (over 2K words)
- π Aggregates over 20 web sources per research to form objective and factual conclusions
- π₯οΈ Includes an easy-to-use web interface (HTML/CSS/JS)
- π Scrapes web sources with javascript support
- π Keeps track and context of visited and used web sources
- π Export research reports to PDF, Word and more...
Please see here for full documentation on:
- Getting started (installation, setting up the environment, simple examples)
- Customization and configuration
- How-To examples (demos, integrations, docker support)
- Reference (full API docs)
Step 0 - Install Python 3.11 or later. See here for a step-by-step guide.
Step 1 - Download the project and navigate to its directory
git clone https://github.com/assafelovic/gpt-researcher.git
cd gpt-researcherStep 3 - Set up API keys using two methods: exporting them directly or storing them in a
.envfile.
For Linux/Windows temporary setup, use the export method:
export OPENAI_API_KEY={Your OpenAI API Key here}
export TAVILY_API_KEY={Your Tavily API Key here}For a more permanent setup, create a .env file in the current gpt-researcher directory and input the keys as follows:
OPENAI_API_KEY={Your OpenAI API Key here}
TAVILY_API_KEY={Your Tavily API Key here}- For LLM, we recommend OpenAI GPT, but you can use any other LLM model (including open sources) supported by Langchain Adapter, simply change the llm model and provider in config/config.py.
- For web search API, we recommend Tavily Search API, but you can also refer to other search APIs of your choice by changing the search provider in config/config.py to
"duckduckgo","googleAPI","bing","googleSerp","searx"and more. Then add the corresponding env API key as seen in the config.py file.
Step 1 - Install dependencies
pip install -r requirements.txtStep 2 - Run the agent with FastAPI
uvicorn main:app --reloadStep 3 - Go to http://localhost:8000 on any browser and enjoy researching!
To learn how to get started with Docker, Poetry or a virtual environment check out the documentation page.
pip install gpt-researcherfrom gpt_researcher import GPTResearcher
query = "why is Nvidia stock going up?"
researcher = GPTResearcher(query=query, report_type="research_report")
# Conduct research on the given query
await researcher.conduct_research()
# Write the report
report = await researcher.write_report()For more examples and configurations, please refer to the PIP documentation page.
We highly welcome contributions! Please check out contributing if you're interested.
Please check out our roadmap page and reach out to us via our Discord community if you're interested in joining our mission.
- Community Discord
- Our email: assafelovic@gmail.com
This project, GPT Researcher, is an experimental application and is provided "as-is" without any warranty, express or implied. We are sharing codes for academic purposes under the MIT license. Nothing herein is academic advice, and NOT a recommendation to use in academic or research papers.
Our view on unbiased research claims:
- The main goal of GPT Researcher is to reduce incorrect and biased facts. How? We assume that the more sites we scrape the less chances of incorrect data. By scraping over 20 sites per research, and choosing the most frequent information, the chances that they are all wrong is extremely low.
- We do not aim to eliminate biases; we aim to reduce it as much as possible. We are here as a community to figure out the most effective human/llm interactions.
- In research, people also tend towards biases as most have already opinions on the topics they research about. This tool scrapes many opinions and will evenly explain diverse views that a biased person would never have read.
Please note that the use of the GPT-4 language model can be expensive due to its token usage. By utilizing this project, you acknowledge that you are responsible for monitoring and managing your own token usage and the associated costs. It is highly recommended to check your OpenAI API usage regularly and set up any necessary limits or alerts to prevent unexpected charges.