Django Monitoring System with Celery & Redis

A robust monitoring application built with Django that leverages Celery for asynchronous task processing and Redis as a high-performance message broker. This system runs continuous background monitoring tasks to track user metrics in real-time.

🌟 Key Features

Backend Architecture

• Django framework providing a solid web foundation
• Celery integration for distributed task processing
• Redis serving as both message broker and result backend
• Automated periodic monitoring with configurable intervals

Monitoring Capabilities

• Real-time user count tracking
• Background task execution without blocking the main application
• Scalable task distribution across multiple workers
• Comprehensive logging for monitoring and debugging

---

🛠️ Installation Guide

Prerequisites

Ensure you have Python 3.8+ and pip installed on your system.

Step 1: Project Setup

# Clone the repository
git clone <repository-url>
cd monitor_project

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install project dependencies
pip install -r requirements.txt

Step 2: Redis Installation & Configuration

For macOS users:

brew install redis
brew services start redis

For Ubuntu/Debian users:

sudo apt update
sudo apt install redis-server
sudo systemctl start redis-server
sudo systemctl enable redis-server

For Windows users: Download and install Redis from the official website or use Windows Subsystem for Linux (WSL).

Verify Redis installation:

redis-cli ping
# Expected response: PONG

Step 3: Database Configuration

# Apply database migrations
python manage.py migrate

# Create a superuser (optional)
python manage.py createsuperuser

Step 4: Launch the Application

Terminal 1 - Start Celery Worker & Beat:

celery -A monitor_project worker --beat --scheduler django --loglevel=info

Terminal 2 - Start Django Development Server:

python manage.py runserver

---

📈 System Monitoring

Once the system is running, the monitoring task will automatically execute every 10 seconds. You'll see output similar to:

[2024-XX-XX XX:XX:XX,XXX: INFO/MainProcess] [CELERY MONITOR] Current user count: 12
[2024-XX-XX XX:XX:XX,XXX: INFO/MainProcess] [CELERY MONITOR] System status: Active
[2024-XX-XX XX:XX:XX,XXX: INFO/MainProcess] [CELERY MONITOR] Current user count: 15

Customizing Monitoring Intervals

To modify the monitoring frequency, update the periodic task configuration in your Django settings or Celery beat schedule.

---

📋 Dependencies

Core Requirements:

• Django>=4.2.0 - Web framework
• celery>=5.3.0 - Distributed task queue
• redis>=4.5.0 - In-memory data structure store
• django-celery-beat - Database-backed periodic tasks

Development Tools:

• django-extensions - Additional Django management commands
• python-dotenv - Environment variable management

---

🚀 Production Deployment

For production environments, consider:

• Using a dedicated Redis instance or cluster
• Configuring Celery with multiple workers
• Implementing proper logging and monitoring
• Setting up process supervision (systemd, supervisor)
• Using a production WSGI server like Gunicorn

---

🔧 Troubleshooting

Common Issues:

1. Redis Connection Error: Ensure Redis server is running on the default port (6379)
2. Celery Tasks Not Executing: Verify that the Celery worker is started with the correct project configuration
3. Database Errors: Run python manage.py migrate to ensure all migrations are applied

Health Check:

# Test Redis connectivity
redis-cli ping

# Check Celery worker status
celery -A monitor_project inspect active

---

📄 License

This project is open source and available under the MIT License.