Content Discovery Agent

A FastAPI + LangGraph project for automated content discovery, link building opportunity identification, and topic discovery using AI-powered workflows.

Project Structure

This project follows a modular, agentic architecture:

blog_automation/
├── app/
│   ├── __init__.py
│   ├── main.py                    # FastAPI application entry point
│   │
│   ├── api/                       # API Layer
│   │   ├── __init__.py
│   │   ├── link_creator_routes.py      # Link Creator Discovery API routes
│   │   ├── topic_discovery_routes.py   # Topic Discovery API routes
│   │   ├── common_routes.py            # Common endpoints (health check)
│   │   └── routes.py                   # Deprecated (backwards compatibility)
│   │
│   ├── config/                    # Configuration
│   │   ├── __init__.py
│   │   └── settings.py            # Environment-based settings
│   │
│   ├── models/                    # Data Models (Pydantic & TypedDict)
│   │   ├── __init__.py
│   │   ├── agent_state.py         # Link Creator workflow state schema
│   │   ├── topic_discovery_state.py  # Topic Discovery workflow state schema
│   │   ├── request_models.py      # API request models
│   │   ├── response_models.py     # API response models
│   │   ├── domain_models.py       # Domain-specific models
│   │   └── mongodb_models.py      # MongoDB document models
│   │
│   ├── services/                  # Business Logic Layer
│   │   ├── __init__.py
│   │   ├── orchestration_service.py          # Link Creator workflow orchestrator
│   │   ├── topic_discovery_orchestration_service.py  # Topic Discovery orchestrator
│   │   ├── llm_service.py            # Google Gemini LLM wrapper
│   │   ├── mongo_service.py          # MongoDB service
│   │   ├── google_search_service.py  # Google Search API integration
│   │   ├── crawler_service.py        # Web scraping service
│   │   ├── agent/                 # Workflow Nodes
│   │   │   ├── link_creator/      # Link Creator Discovery nodes
│   │   │   │   ├── generate_search_query.py
│   │   │   │   ├── google_search.py
│   │   │   │   ├── social_page.py
│   │   │   │   ├── skip.py
│   │   │   │   ├── scrape_pages.py
│   │   │   │   ├── verify_blog_page.py
│   │   │   │   ├── validate_link_creator.py
│   │   │   │   ├── store_results.py
│   │   │   │   └── prompt_builders.py
│   │   │   └── topic_discovery/   # Topic Discovery nodes
│   │   │       ├── fetch_link_creators.py
│   │   │       ├── process_link_creators.py
│   │   │       ├── find_common_topics.py
│   │   │       ├── analyze_meta_trends.py
│   │   │       └── save_topic_discovery.py
│   │   └── utils/                 # Service Utilities
│   │       ├── __init__.py
│   │       ├── parsers.py
│   │       └── validators.py
│   │
│   ├── compat/                    # Compatibility Layer
│   │   └── __init__.py
│   │
│   └── utils/                     # General Utilities
│       ├── __init__.py
│       └── visualization.py
│
├── main.py                        # Application entry point (wrapper)
├── requirements.txt               # Python dependencies
├── .env.example                   # Environment variable template
├── .gitignore                     # Git ignore patterns
└── README.md                      # This file

Features

FastAPI: Modern, fast web framework for REST API
LangGraph: Stateful workflow orchestration for agentic AI
Google Gemini: LLM integration for intelligent query generation, content analysis, and topic extraction
MongoDB: Full database integration with persistence for discovery results and topic analysis
Pydantic: Type-safe data validation
Modular Architecture: Separation of concerns with clear layer boundaries
Two Workflows:
- Link Creator Discovery: Finds and validates link building opportunities
- Topic Discovery: Analyzes discovered content to find common topics and trends

Workflow Overview

Link Creator Discovery Workflow

Generate Search Query: Creates optimized search queries based on business context
Google Search: Performs search using generated queries
Social Page Detection: Identifies and routes social media pages
Skip & Feedback Loop: Skips social pages and refines queries (up to MAX_SKIP_COUNT iterations)
Scrape Pages: Extracts content from search results
Verify Blog Page: Validates if pages are actual blog posts (/blog, /blogs patterns)
Validate Link Creator: Extracts and validates contact information from blog pages
Store Results: Saves validated opportunities to MongoDB

Topic Discovery Workflow

Fetch Link Creators: Retrieves previously discovered link creators from MongoDB
Process Link Creators: Analyzes blog content from link creators to extract topics per domain
Find Common Topics: Aggregates topics across domains, deduplicates similar topics, and identifies common themes
Analyze Meta Trends: (Optional) Performs advanced trend analysis on discovered topics
Save Results: Persists topic discovery results to MongoDB with metadata and statistics

Setup

Clone the repository:

git clone <repository-url>
cd blog_automation

Create a virtual environment (recommended):

python -m venv venv
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

Install dependencies:
```
pip install -r requirements.txt
```
Set up environment variables:
- Copy the example environment file:
```
# On Windows:
copy .env.example .env
# On macOS/Linux:
cp .env.example .env
```
- Edit .env and add your actual values (see .env.example for all options)
- Required variables:
  - GOOGLE_API_KEY: Your Google Gemini API key (get from Google AI Studio)
  - MONGODB_URL: MongoDB connection string (default: mongodb://localhost:27017)
- Optional but recommended:
  - GOOGLE_SEARCH_API_KEY + GOOGLE_SEARCH_ENGINE_ID: For better search results

Start MongoDB (if running locally):

# Make sure MongoDB is running on your system
# On Windows (if installed as service):
# MongoDB should start automatically
# On macOS (with Homebrew):
brew services start mongodb-community
# On Linux:
sudo systemctl start mongod

Run the application:

python main.py

Or using uvicorn directly:

uvicorn app.main:app --reload --port 8001

Access the API:
- API Base: http://localhost:8001
- Interactive docs (Swagger): http://localhost:8001/docs
- Alternative docs (ReDoc): http://localhost:8001/redoc
- Health check: http://localhost:8001/api/v1/health

API Endpoints

POST /api/v1/content-discovery

Execute the Link Creator Discovery workflow to find link building opportunities.

Request:

{
  "userId": "user123",
  "size": "medium",
  "companyName": "Example Corp",
  "industry": "Technology",
  "niche": "SaaS",
  "websiteUrl": "https://example.com",
  "offer": "Free trial for 30 days"
}

Response:

{
  "success": true,
  "message": "Content discovery completed successfully",
  "results": [
    {
      "url": "https://example.com/blog/post",
      "name": "John Doe",
      "email": "john@example.com",
      "domain": "example.com",
      "title": "Blog Post Title",
      "description": "Blog post description..."
    }
  ],
  "total_results": 5,
  "execution_time_ms": 1234.56,
  "request_id": "uuid-here"
}

POST /api/v1/topic-discovery

Execute the Topic Discovery workflow to analyze discovered link creators and find common topics.

Request:

{
  "userId": "user123"
}

Response:

{
  "success": true,
  "message": "Topic discovery completed successfully",
  "data": {
    "userId": "user123",
    "totalLinkCreators": 10,
    "totalDomains": 8,
    "commonTopics": [
      {
        "topic": "AI-powered content creation",
        "count": 5,
        "domains": ["domain1.com", "domain2.com"]
      }
    ],
    "executionTimeMs": 2345.67,
    "timestamp": "2024-01-01T00:00:00Z"
  },
  "execution_time_ms": 2345.67,
  "request_id": "uuid-here"
}

GET /api/v1/health

Health check endpoint.

Response:

{
  "status": "healthy",
  "service": "Content Discovery Agent"
}

GET /

Root endpoint with API information and available endpoints.

Configuration

All configuration is managed through environment variables. See .env.example for a complete template with all available options and descriptions.

Required Variables

GOOGLE_API_KEY: Your Google Gemini API key (required for LLM operations)

MongoDB Configuration

MONGODB_URL: MongoDB connection string (default: mongodb://localhost:27017)
MONGODB_DATABASE: Database name (default: blog_automation)
MONGODB_COLLECTION: Collection for link creator results (default: discovery_results)
MONGODB_TOPIC_DISCOVERY_COLLECTION: Collection for topic discovery results (default: topicDiscovery)

Optional Variables

GOOGLE_SEARCH_API_KEY + GOOGLE_SEARCH_ENGINE_ID: For enhanced Google Search API results
MODEL_NAME: LLM model name (default: gemini-2.0-flash-lite)
TEMPERATURE: LLM temperature 0.0-1.0 (default: 0.7)
MAX_SCRAPE_ITERATIONS: Maximum pages to scrape (default: 10, lower for testing)
MAX_SKIP_COUNT: Maximum social page skip iterations (default: 10)
LOG_LEVEL: Logging level - DEBUG, INFO, WARNING, ERROR (default: INFO)

Features & Capabilities

Link Creator Discovery

✅ Automated search query generation using AI
✅ Google Search integration (with Custom Search API support)
✅ Social media page detection and filtering
✅ Web scraping with content extraction
✅ Blog page verification (URL pattern matching)
✅ Contact information extraction and validation
✅ MongoDB persistence with full document storage

Topic Discovery

✅ Link creator data retrieval from MongoDB
✅ AI-powered topic extraction from blog content
✅ Domain-level topic analysis
✅ Intelligent topic deduplication (merges similar/subset topics)
✅ Common topic aggregation across domains
✅ Metadata and statistics tracking
✅ MongoDB persistence with structured topic data

Technical Features

✅ LangGraph workflow orchestration with state management
✅ Type-safe models with Pydantic validation
✅ Comprehensive error handling and logging
✅ Async/await support for better performance
✅ CORS middleware for API access
✅ Interactive API documentation (Swagger/ReDoc)

Architecture Principles

Separation of Concerns: Clear boundaries between Models, Services, Nodes, and API layers
Dependency Injection: Factory functions for node creation with dependency injection
Type Safety: Pydantic models for API validation, TypedDict for LangGraph state management
Modularity: Each component has a single, well-defined responsibility
State-Driven: Central state flows through all nodes in the workflow
Error Resilience: Comprehensive error handling with graceful degradation
Scalability: Async/await patterns for better performance and resource utilization

Testing

Using Postman or cURL

Link Creator Discovery:

curl -X POST http://localhost:8001/api/v1/content-discovery \
  -H "Content-Type: application/json" \
  -d '{
    "userId": "test123",
    "size": "medium",
    "companyName": "Example Corp",
    "industry": "Technology",
    "niche": "SaaS",
    "websiteUrl": "https://example.com",
    "offer": "Free trial for 30 days"
  }'

Topic Discovery (after link creators are discovered):

curl -X POST http://localhost:8001/api/v1/topic-discovery \
  -H "Content-Type: application/json" \
  -d '{
    "userId": "test123"
  }'

Troubleshooting

Common Issues

MongoDB Connection Error:
- Ensure MongoDB is running: mongosh or check service status
- Verify MONGODB_URL in .env is correct
- Check database name (will be normalized to lowercase)
Google API Key Error:
- Verify GOOGLE_API_KEY is set in .env
- Check API key is valid and has quota remaining
- Ensure key has access to Gemini API
Empty Results from Topic Discovery:
- Make sure Link Creator Discovery has been run first for the same userId
- Check MongoDB to verify data was saved correctly
- Review logs for any errors during topic extraction
Port Already in Use:
- Default port is 8001 (to avoid conflict with crawler on 8000)
- Change FASTAPI_PORT in .env or modify main.py

Notes

Database names are automatically normalized to lowercase in MongoDB
Topic deduplication uses intelligent similarity matching (80% threshold)
Set MAX_SCRAPE_ITERATIONS=3 for faster testing during development
All API endpoints support CORS (configure appropriately for production)
Interactive API docs available at /docs endpoint

Navigation Menu

Uh oh!

Repository files navigation

Content Discovery Agent

Project Structure

Features

Workflow Overview

Link Creator Discovery Workflow

Topic Discovery Workflow

Setup

API Endpoints

POST /api/v1/content-discovery

POST /api/v1/topic-discovery

GET /api/v1/health

GET /

Configuration

Required Variables

MongoDB Configuration

Optional Variables

Features & Capabilities

Link Creator Discovery

Topic Discovery

Technical Features

Architecture Principles

Testing

Using Postman or cURL

Troubleshooting

Common Issues

Notes

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
app		app
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
main.py		main.py
requirements.txt		requirements.txt

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Content Discovery Agent

Project Structure

Features

Workflow Overview

Link Creator Discovery Workflow

Topic Discovery Workflow

Setup

API Endpoints

POST /api/v1/content-discovery

POST /api/v1/topic-discovery

GET /api/v1/health

GET /

Configuration

Required Variables

MongoDB Configuration

Optional Variables

Features & Capabilities

Link Creator Discovery

Topic Discovery

Technical Features

Architecture Principles

Testing

Using Postman or cURL

Troubleshooting

Common Issues

Notes

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages