Multi-modal Agent Starter

Create a cloud-hosted LLM Agent with custom personality, multi-modal tools, and memory.

This repository is designed to pair with this Agent Building Guidebook

Getting Started

You can be up and running in under a minute. A full setup walk-through is here.

Clone this repository, then set up a Python virtual environment with:

python3.8 -m venv .venv
source .venv/bin/activate
python3.8 -m pip install -r requirements.txt

Running your agent

A full guide to running is here.

With the proper Python environment set up and your STEAMSHIP_API_KEY environment variable set, just run:

PYTHONPATH=src python3.8 src/api.py

Modifying your agent

Modify your agent by editing src/api.py. You can:

• Change its personality (edit the SYSTEM_PROMPT)
• Add tools, allowing your agent's reasoning process to do new things (see tool list below)
• Add mixins, allowing your agent to connect to different channels (see mixin list below)

Other examples are found in the example_agents folder. Copy/paste one of these into api.py to use it.

Deploying your agent

A full guide to deploying is here.

This project can be deployed straight to the cloud. Simply type:

ship deploy

and follow the prompts.

What tools can I use with my agent?

Tools help your agent perform actions or fetch information from the outside world. The Steamship SDK includes a large set of multi-modal & memory-aware tools you can use right away.

Your starter project already has a few tools in src/example_tools.

And you can import or find more open source tools in the Steamship SDK:

• Audio Transcription:
  • Assembly AI - Turns audio into text
  • Whisper - Turns audio into text
  • RSS Download - Returns Audio URLs from an RSS feed
• Classification:
  • Sentiment Analysis - Can report on the sentiment of a piece of text
  • Zero Shot Classification - Can classify a piece of text
• Image Generation:
  • DALL-E - Generate images with DALL-E
  • Stable Diffusion - Generate images with Stable Diffusion
  • Google Image Search - Perform a Google Image Search and return the results
• Speech Generation:
  • Eleven Labs - Turn text into the spoken word
• Search:
  • Google Search - Find answers to questions on the web
• Question Answering:
  • Vector Search QA - Find answers to questions in the Steamship Vector Database
  • Prompt Database QA - Find answers to questions from a pre-loaded prompt database
• Text Generation:
  • Image Prompt Generation - Rewrite a topic into a Stable Diffusion image prompt
  • Personality Tool - Reword a response according to a particular personality
  • Text Summarization - Summarize text
  • Text Rewriter - Utility tool for building tools that use prompts to operate
  • Translation - Translate text using an LLM
• Conversation Starters:
  • Knock-Knock Joke Starter - Initiate a knock knock joke. The world's most useful tool.

What mixins can I use with my agent?

Mixins provide additional API endpoints to your agent, which can connect your agent to a communication channel like Telegram, or provide additional data loading functionality.

Several mixins are provided out of the box in the Steamship SDK:

• Transports (Communication Channels)
  • Steamship Web Widget - For testing your agent in the Web UI
  • Telegram - for communicating with your agent in Telegram
• Data Loading
  • Indexer Pipeline - For loading documents into your agent's question-answer ability