MCP Agent - Web Scraper & Email Management System

A comprehensive AI-powered application built with mcp-agent featuring web scraping, email management, and social media content generation capabilities.

Features

• 🌐 Web Scraping: Extract articles from websites
• 📧 Email Management: Process and draft emails
• 📊 Zapier Integration: Connect with various services and tools
• 📱 Social Media Content: Generate LinkedIn, Twitter, and Instagram posts
• 🖼️ Image Generation: Create visual content for social posts
• 🔧 Streamlit Interface: Easy-to-use web interface for all operations

Prerequisites

• Python 3.10+ (required by mcp-agent)
• Node.js 18+ (required for MCP servers)
• uv package manager: Install with pip install uv

Installation

1. Clone and Setup

# Clone the repository
git clone <repository-url>
cd mcp_basic_agent

# Install dependencies using uv
uv sync

2. Install Node.js Dependencies

The project uses several Node.js-based MCP servers:

# These will be automatically installed when first run, but you can pre-install:
npx -y @modelcontextprotocol/server-filesystem
npx -y @modelcontextprotocol/server-zapier

API Keys & Configuration

Required API Keys

You need the following API keys to run the application:

1. Anthropic API Key (Claude)

   • Get from: https://console.anthropic.com
   • Format: sk-ant-api03-...

2. Zapier Natural Language Actions (NLA) API Key

   • Go to: https://zapier.com/l/natural-language-actions
   • Create account and get your MCP URL by creating new server OTHER OPTION
   • add Gmail tools and login with credentials
   • go to settings and copy the url for zapier connection
   • Format: https://mcp.zapier.com/api/mcp/s/...

Configuration Files

The project uses two configuration files:

1. API Keys Configuration

Copy the example secrets file and add your API keys:

cp mcp_agent_secrets.yaml.example mcp_agent.secrets.yaml

Edit mcp_agent.secrets.yaml and replace the placeholder values:

anthropic:
  api_key: "sk-ant-api03-your_anthropic_api_key_here"

mcp:
  servers:
    zapier:
      url: "https://mcp.zapier.com/api/mcp/s/your_zapier_mcp_url_here"
      transport: "streamable_http"

2. Server Configuration

The mcp_agent.config.yaml file is already configured but you can modify:

• Logging levels and paths
• Model selection (default: Claude Sonnet 4)
• MCP server settings

Running the Application

Option 1: Streamlit Web Interface (Recommended)

Start the web interface:

uv run streamlit run streamlit.py

The web interface provides:

• Web Scraper:

  • Configuration: Set URLs, criteria, and image styles
  • Run: Execute scraping workflow
  • Preview: Edit and save social media posts

• Email Management:

  • Settings: Configure email guidelines and contacts
  • Run: Generate email drafts

Option 2: Direct Script Execution

Run individual components:

# Run web scraper
uv run python webscraper.py

# Run email agent
uv run python mail_agent.py

Project Structure

mcp_basic_agent/
├── README.md                          # This file
├── mcp_agent.config.yaml             # MCP server configuration
├── mcp_agent.secrets.yaml            # API keys (create from .example)
├── mcp_agent_secrets.yaml.example    # Template for secrets
├── streamlit.py                       # Web interface
├── webscraper.py                      # Web scraping workflow
├── mail_agent.py                      # Email management workflow
├── webscraper_inputs/                 # Web scraper configuration
│   ├── sources.txt                   # URLs to scrape
│   ├── selection_criteria.txt        # Article selection criteria
│   └── image_style.txt               # Image generation style
├── webscraper_outputs/               # Generated content by date
│   └── YYYY-MM-DD/
│       ├── linkedin.txt
│       ├── twitter.txt
│       ├── instagram.txt
│       └── image_prompt.txt
├── mail_agent_inputs/                # Email agent configuration
│   ├── contacts.csv                  # Contact database
│   ├── mail_agent_context.txt        # Context and categories
│   └── email_guidelines.txt          # Email templates
├── tests/                            # Test files
└── logs/                            # Application logs

First-Time Setup Walkthrough

1. Using Streamlit Interface (Easiest)

# Start the application
uv run streamlit run streamlit.py

# The web interface will guide you through:
# 1. API key configuration
# 2. Google authentication (first run)
# 3. Setting up your preferences

2. Manual Setup

# 1. Install dependencies
uv sync

# 2. Copy and edit secrets file
cp mcp_agent_secrets.yaml.example mcp_agent.secrets.yaml
# Edit the file with your API keys

# 3. Test the setup
uv run python -c "from webscraper import WebScraperWorkflow; print('Setup OK')"

Authentication Flow

First Run - Zapier Integration

When you first run the application, the Zapier MCP server will authenticate using the URL you configured in the secrets file. Make sure you have:

1. Created a Zapier account
2. Set up Natural Language Actions
3. Copied the correct MCP URL to your secrets file

Testing

Run the test suite:

# Run all tests
uv run pytest

# Run specific test modules
uv run pytest tests/test_webscraper.py
uv run pytest tests/test_mail_agent.py

Development

Linting and Type Checking

# Run linting
uv run ruff check .

# Run type checking
uv run mypy .

# Auto-fix linting issues
uv run ruff check . --fix

Adding Dependencies

# Add Python dependencies
uv add package_name

# Add development dependencies
uv add --group dev package_name

Troubleshooting

Common Issues

1. "Command not found: uv"

   pip install uv

2. "MCP server failed to start"

   • Ensure Node.js is installed: node --version
   • Clear npm cache: npm cache clean --force

3. "Invalid API key" errors

   • Verify API keys in mcp_agent.secrets.yaml
   • Check for extra spaces or quotes

4. Zapier connection issues

   • Verify your Zapier MCP URL is correct
   • Check that Natural Language Actions is enabled in your Zapier account

5. Streamlit connection errors

   • Check if port 8501 is available
   • Try: uv run streamlit run streamlit.py --server.port 8502

Getting Help

1. Check the logs in logs/ directory for detailed error messages
2. Verify MCP server status in the application
3. Test individual components:

   # Test web scraping
   uv run python -c "from webscraper import WebScraperWorkflow; w = WebScraperWorkflow(); print('Webscraper OK')"
   
   # Test email agent
   uv run python -c "from mail_agent import SimplifiedEmailManagementSystem; e = SimplifiedEmailManagementSystem(); print('Mail Agent OK')"

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests: uv run pytest
5. Run linting: uv run ruff check .
6. Submit a pull request

License

This project is licensed under the Apache License 2.0. See the main mcp-agent repository for details.

Related Links

• mcp-agent Documentation
• Model Context Protocol
• Firecrawl API
• Zapier NLA
• Anthropic Claude