Competitive Intelligence Aggregator

Multi-source AI monitoring pipeline that wakes up every morning, scans news and social for fintech topics, and sends you a summary email. Set it and forget it.

Architecture

Scheduled Python jobs pull from NewsAPI, Reddit, and Twitter APIs. GPT-4 filters for relevance and generates summaries. Brevo email API delivers formatted daily reports. Deployed on Heroku with scheduled tasks.

Key Outcomes

• Multi-source data aggregation across news, social media, and forums
• AI-powered relevance filtering reduces noise by 90%+
• Automated daily email delivery via Brevo with structured sections (product launches, industry news, regulatory changes, word on the street)

Tech Stack

• Language: Python
• AI: OpenAI GPT-4
• Data Sources: NewsAPI, Reddit API (PRAW), Twitter/X API (Tweepy)
• Email: Brevo (formerly Sendinblue)
• Deployment: Heroku with Heroku Scheduler

Prerequisites

• Python 3.9+
• API keys for: NewsAPI, Reddit, Twitter/X, OpenAI, Brevo

Setup

1. Clone the repository:

   git clone https://github.com/immuneeb/competitive-intel.git
   cd competitive-intel

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Create your environment file:

   cp .env.template .env

5. Add your API keys to .env:

   Variable	Where to get it
   NEWSAPI_KEY	newsapi.org
   REDDIT_CLIENT_ID	reddit.com/prefs/apps
   REDDIT_CLIENT_SECRET	Same as above
   TWITTER_API_KEY	developer.twitter.com
   TWITTER_API_SECRET	Same as above
   TWITTER_ACCESS_TOKEN	Same as above
   TWITTER_ACCESS_TOKEN_SECRET	Same as above
   OPENAI_API_KEY	platform.openai.com
   BREVO_API_KEY	brevo.com
   BREVO_SENDER_EMAIL	Your verified Brevo sender address
   EMAIL_RECIPIENTS	Comma-separated list of recipient emails

Usage

Run the competitive intelligence report manually:

python index.py

Run the interactive market research tool:

python market_research.py

Heroku Deployment

1. Deploy the app to Heroku
2. Set all environment variables via heroku config:set
3. Add the Heroku Scheduler add-on
4. Create a scheduled job to run python index.py daily

Configuration

Edit TOPICS_OF_INTEREST in index.py to monitor different topics. The default configuration tracks fintech and credit card products.