Custom CI/CD Pipeline Tool

A minimal, educational CI/CD pipeline tool built from scratch using Python with file-based persistence.

Features

• 🔗 Webhook Listener: Receives GitHub push events via HTTP server
• 📋 Job Queue: File-based job management with status tracking
• 📄 Pipeline Parser: Reads .cicd.yml configuration files
• ⚡ Executor: Runs pipeline steps using subprocess
• 📊 Dashboard: CLI interface to view jobs and logs
• 🔒 Security Scanner: Basic security scanning of repositories

Quick Setup & Demo

Step 1: Install Dependencies

pip install -r requirements.txt

Step 2: Start the CI/CD Server

python start_server.py

You should see:

Starting CI/CD Pipeline Server...
Webhook listener: http://localhost:8080/webhook
Starting job executor...

Step 3: Test with Sample Job (New Terminal)

python debug_test.py

This creates a test job and triggers the webhook.

Step 4: View Jobs and Logs

# View recent jobs
python simple_dashboard.py jobs

# View specific job logs
python simple_dashboard.py logs <job_id>

Step-by-Step Demo Walkthrough

1. First Run - See It Work

# Terminal 1: Start server
python start_server.py

# Terminal 2: Create a test job
python debug_test.py

# Terminal 3: Check results
python simple_dashboard.py jobs

2. Understanding the Output

After running the demo, you'll see:

• Server logs: Job processing in real-time
• Job creation: New job added to queue
• Dashboard: Job status and execution logs

3. Check Generated Files

# View persistent job storage
type jobs.json

# Check cloned repositories
dir workspace

Pipeline Configuration

Create a .cicd.yml file in your repository:

name: "My Pipeline"
steps:
  - name: "setup"
    run: "python --version"
  
  - name: "install"
    run: "pip install -r requirements.txt"
  
  - name: "test"
    run: "python -m pytest"
  
  - name: "build"
    run: "python setup.py sdist"

Real GitHub Integration

Local Testing (Recommended)

1. Expose server: Install ngrok and run ngrok http 8080
2. Add webhook: GitHub repo → Settings → Webhooks → Add webhook
3. URL: https://your-ngrok-url.ngrok.io/webhook
4. Content-type: application/json
5. Events: Just push events

Understanding the Components

Core Files

• start_server.py - Main server entry point
• simple_dashboard.py - CLI dashboard for viewing jobs
• debug_test.py - Webhook testing script
• shared_queue.py - Shared job queue instance

Core Modules

• core/webhook_listener.py - HTTP webhook server
• core/executor.py - Pipeline execution engine
• core/file_queue.py - File-based job queue
• core/pipeline_parser.py - Git operations & .cicd.yml parser
• core/security_scanner.py - Basic security scanning
• models/job.py - Job data model

Generated Files

• jobs.json - Persistent job storage (auto-created)
• workspace/ - Git repositories (auto-created)

Architecture Flow

GitHub Push → Webhook → Job Queue → Executor → Pipeline Steps → Results
     ↓           ↓          ↓           ↓            ↓           ↓
  Payload    HTTP POST   jobs.json   Git Clone   Run Commands  Logs

Common Commands Reference

# Start the CI/CD server
python start_server.py

# Test webhook functionality
python debug_test.py

# View all jobs
python simple_dashboard.py jobs

# View specific job logs
python simple_dashboard.py logs <job_id>

# View recent 5 jobs only
python simple_dashboard.py jobs 5

Job Status Flow

1. queued → Job created, waiting for processing
2. running → Job being executed by server
3. done → Job completed successfully ✅
4. failed → Job failed during execution ❌

Project Structure

cicd_scripting/
├── core/                    # Core pipeline components
│   ├── executor.py         # Runs pipeline steps
│   ├── file_queue.py       # Job queue management
│   ├── pipeline_parser.py  # Git & .cicd.yml handling
│   ├── security_scanner.py # Security scanning
│   └── webhook_listener.py # HTTP webhook server
├── models/
│   └── job.py              # Job data structure
├── config/
│   └── settings.py         # Configuration
├── tests/
│   └── sample_files/       # Test configurations
├── sample_repo/
│   └── .cicd.yml          # Example pipeline config
├── start_server.py         # 🚀 Main server
├── simple_dashboard.py     # 📊 View jobs & logs
├── debug_test.py          # 🧪 Test webhook
├── shared_queue.py        # Shared job queue
└── requirements.txt       # Dependencies

Troubleshooting

Server won't start?

• Check if port 8080 is available
• Ensure all dependencies are installed

Jobs failing?

• Check git is installed and accessible
• Verify repository URLs are correct
• Check job logs: python simple_dashboard.py logs <job_id>

No webhook response?

• Ensure server is running on localhost:8080
• Check firewall settings for local testing

Pipeline Templates

Use pre-built templates for common project types:

# List available templates
python generate_template.py list

# Generate Next.js template
python generate_template.py nextjs/basic

# Generate Python template
python generate_template.py python/basic

Available Templates:

• nextjs/basic - Simple Next.js projects
• nextjs/full - Comprehensive Next.js with testing
• nextjs/deployment - Production deployment pipeline
• python/basic - Python projects with pytest
• python/django - Django web applications
• nodejs/express - Express.js applications

See TEMPLATES.md for detailed documentation.

Performance Benchmarking

Monitor and optimize pipeline performance:

# Start server with benchmarking
python start_server_benchmark.py

# View performance overview
python benchmarking/benchmark_dashboard.py overview

# Analyze step performance
python benchmarking/benchmark_dashboard.py steps

# Identify bottlenecks
python benchmarking/benchmark_dashboard.py bottlenecks

See benchmarking/README.md for detailed documentation.

Next Steps

1. Use templates: Generate .cicd.yml with python generate_template.py
2. Monitor performance: Use benchmarking tools to optimize pipelines
3. Add real repositories: Update webhook payloads
4. Deploy to server: Use ngrok for GitHub integration
5. Extend functionality: Add notifications, artifacts, etc.

Quick Demo Video

1. Terminal 1: python start_server.py (Server starts)
2. Terminal 2: python debug_test.py (Creates job)
3. Terminal 1: Watch job processing logs
4. Terminal 2: python simple_dashboard.py jobs (View results)
5. Terminal 2: python simple_dashboard.py logs <job_id> (View details)

Expected Output: Job processes successfully with Python version check, file listing, and echo command.

---

Built with ❤️ for learning CI/CD concepts