🌪️ Disaster Monitoring API

A real-time disaster monitoring system that scrapes social media and earthquake data to provide early warnings and situational awareness.

🚀 Features

• Multi-Source Data Collection: Scrapes Twitter/X accounts and earthquake APIs
• AI-Powered Classification: Uses Groq LLaMA for disaster relevance detection
• Real-Time Processing: Concurrent data collection and processing
• Geographic Intelligence: Automatic location extraction and coordinate mapping
• MongoDB Storage: Scalable database with duplicate detection
• RESTful API: Flask-based endpoints for data access

Project Structure

backend/
├── core/                   # Core application logic
│   ├── main.py            # Flask API server
│   ├── classify.py        # AI classification
│   ├── mongo.py          # Database operations  
│   ├── dict_coordinates.py # Location services
│   ├── logger.py         # Logging configuration
│   └── validators.py     # Data validation
├── scrapers/              # Data collection modules
│   ├── x_scrap.py        # Twitter scraper
│   └── seismo_scrap.py   # Earthquake scraper
├── utils/                 # Utility tools
│   ├── pop_data.py       # Database population
│   └── clear.py          # Database cleanup
├── config/                # Configuration files
│   ├── config.py         # Application settings
│   ├── requirements.txt  # Dependencies
│   └── twitter_cookies1.json # Session cookies
├── data/                  # Data storage
│   ├── earthquake/       # Earthquake data
│   └── tweets/          # Tweet data
├── unused/               # Legacy/unused files
└── run.py               # Application entry point

Installation

1. Install Dependencies

   pip install -r config/requirements.txt

2. Configure Chrome Driver

   • Download ChromeDriver and place in C:\\Program Files\\Driver\\chromedriver.exe
   • Or update CHROME_DRIVER_PATH in config/config.py

3. Configure Database

   • Update MongoDB connection string in config/config.py

Usage

Quick Start

python run.py

API Endpoints

• GET /health - Health check endpoint
• GET /fetch_socials - Collect disaster data from all sources

Example Response

{
  "status": "success",
  "message": "Collected 18 total records",
  "results": {
    "x_data": [...],
    "earthquake_data": [...]
  }
}

Configuration

Key settings in config/config.py:

• Database: MongoDB connection and collection settings
• API Keys: Groq AI API configuration
• Scrapers: Twitter accounts, limits, timeouts
• Paths: Data directories, Chrome driver location

Data Flow

1. Collection: Concurrent scraping from Twitter and earthquake APIs
2. Classification: AI-powered disaster type and severity classification
3. Enhancement: Geographic coordinate addition via OpenStreetMap
4. Storage: MongoDB storage with duplicate prevention
5. Response: JSON API response with collected data

Key Improvements Made

✅ Code Organization

• Separated concerns into logical modules
• Configuration management system
• Consistent project structure

✅ Error Handling

• Comprehensive try-catch blocks
• Graceful degradation on failures
• Input validation and sanitization

✅ Logging System

• Structured logging with different levels
• File and console output
• Request/operation tracking

✅ Performance

• Connection pooling for MongoDB
• Concurrent data collection
• Optimized Selenium operations

✅ Maintainability

• Clear separation of configuration
• Modular function design
• Comprehensive documentation

✅ Production Ready

• Health check endpoints
• Proper error responses
• Environment-based configuration

Dependencies

• Flask: Web API framework
• Selenium: Web scraping automation
• BeautifulSoup: HTML parsing
• PyMongo: MongoDB driver
• OpenAI: Groq AI integration
• Requests: HTTP client

Development

Running Tests

pytest tests/

Code Formatting

black .
flake8 .

Database Operations

from utils.clear import clear_database
from utils.pop_data import populate_data

License

This project is for educational and disaster monitoring purposes.