GitLab Webhook FastAPI Server

A production-ready FastAPI server for receiving, validating, and storing GitLab webhook events in PostgreSQL. This server provides endpoints for core GitLab webhook events (Push, Merge Request, and Issue) with an extensible architecture for adding more event types.

Features

• ✅ Core GitLab Webhooks: Push, Merge Request, and Issue events
• ✅ PostgreSQL Storage: Persistent storage of all webhook events with JSONB payloads
• ✅ Token Authentication: Secure webhook verification using GitLab secret tokens
• ✅ Automatic API Documentation: Interactive Swagger UI and ReDoc
• ✅ Health Check Endpoint: Monitor server and database status
• ✅ Docker Support: Easy deployment with Docker Compose
• ✅ Database Migrations: Version-controlled schema changes with Alembic
• ✅ Extensible Architecture: Easy to add new webhook event types
• ✅ Comprehensive Logging: Structured logging for debugging and monitoring
• ✅ Type Safety: Full Pydantic validation for webhook payloads

Quick Start

Prerequisites

• Python 3.11+
• PostgreSQL 15+
• Docker & Docker Compose (optional, for containerized deployment)

Installation

1. Clone the repository

git clone <repository-url>
cd gitlab-webhook-server

2. Set up environment

cp .env.example .env
# Edit .env with your configuration

3. Install dependencies

pip install -r requirements.txt

4. Start PostgreSQL (using Docker)

docker-compose up -d postgres

5. Initialize database

alembic upgrade head

6. Run the server

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

The server will be available at:

• API: http://localhost:8000
• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Using Docker Compose

For a complete setup with PostgreSQL:

docker-compose up --build

This will start both the FastAPI application and PostgreSQL database.

Configuration

Create a .env file with the following variables:

# Database Configuration
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/gitlab_webhooks

# GitLab Configuration
GITLAB_WEBHOOK_SECRET=your-secret-token-here

# Application Configuration
APP_NAME=GitLab Webhook Server
APP_VERSION=1.0.0
LOG_LEVEL=INFO

# CORS Configuration
CORS_ORIGINS=["*"]

API Endpoints

Webhook Endpoints

Endpoint	Method	Description	GitLab Event
/webhooks/push	POST	Handle push events	Push Hook
/webhooks/merge-request	POST	Handle merge request events	Merge Request Hook
/webhooks/issue	POST	Handle issue events	Issue Hook

Utility Endpoints

Endpoint	Method	Description
/health	GET	Health check endpoint
/	GET	API information
/docs	GET	Swagger UI documentation
/redoc	GET	ReDoc documentation

Configuring GitLab Webhooks

1. Navigate to your GitLab project
2. Go to Settings > Webhooks
3. Add webhook URL: http://your-server:8000/webhooks/{event-type}
4. Enter your secret token (must match GITLAB_WEBHOOK_SECRET)
5. Select trigger events:
   • Push events → /webhooks/push
   • Merge request events → /webhooks/merge-request
   • Issues events → /webhooks/issue
6. Click Add webhook

Example Configuration

URL: http://your-server.com:8000/webhooks/push
Secret Token: your-secret-token-here
Trigger: Push events
SSL verification: Enable (recommended for production)

Testing

Manual Testing with curl

Test the push webhook:

curl -X POST http://localhost:8000/webhooks/push \
  -H "Content-Type: application/json" \
  -H "X-Gitlab-Token: your-secret-token" \
  -H "X-Gitlab-Event: Push Hook" \
  -d @test_payloads/push_event.json

Expected response:

{
  "status": "success",
  "message": "Push event received and stored",
  "event_id": "550e8400-e29b-41d4-a716-446655440000"
}

Using the Interactive API Docs

1. Navigate to http://localhost:8000/docs
2. Click on an endpoint to expand it
3. Click "Try it out"
4. Add the X-Gitlab-Token header
5. Paste a webhook payload
6. Click "Execute"

See WEBHOOK_EXAMPLES.md for complete example payloads.

Database Schema

The server uses a single table to store all webhook events:

CREATE TABLE webhook_events (
    id UUID PRIMARY KEY,
    event_type VARCHAR(50) NOT NULL,
    event_name VARCHAR(100) NOT NULL,
    project_id INTEGER,
    project_name VARCHAR(255),
    payload JSONB NOT NULL,
    headers JSONB,
    received_at TIMESTAMP NOT NULL,
    processed BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP NOT NULL
);

CREATE INDEX idx_event_type ON webhook_events(event_type);
CREATE INDEX idx_project_id ON webhook_events(project_id);
CREATE INDEX idx_created_at ON webhook_events(created_at);

Querying Webhook Events

-- View recent webhooks
SELECT id, event_type, project_name, created_at 
FROM webhook_events 
ORDER BY created_at DESC 
LIMIT 10;

-- Count events by type
SELECT event_type, COUNT(*) 
FROM webhook_events 
GROUP BY event_type;

-- Get specific event details
SELECT payload 
FROM webhook_events 
WHERE id = 'your-event-id';

-- Find all push events for a project
SELECT * 
FROM webhook_events 
WHERE event_type = 'push' 
  AND project_id = 15;

Project Structure

gitlab-webhook-server/
├── app/
│   ├── __init__.py
│   ├── main.py                 # FastAPI application
│   ├── config.py               # Configuration management
│   ├── database/
│   │   ├── __init__.py
│   │   ├── connection.py       # Database setup
│   │   └── models.py           # SQLAlchemy models
│   ├── routers/
│   │   ├── __init__.py
│   │   ├── webhooks.py         # Webhook endpoints
│   │   └── health.py           # Health check
│   ├── schemas/
│   │   ├── __init__.py
│   │   ├── push.py             # Push event schema
│   │   ├── merge_request.py    # MR event schema
│   │   └── issue.py            # Issue event schema
│   └── utils/
│       ├── __init__.py
│       ├── auth.py             # Token verification
│       └── logging.py          # Logging config
├── alembic/                    # Database migrations
├── tests/                      # Test files
├── .env.example                # Environment template
├── docker-compose.yml          # Docker setup
├── Dockerfile                  # Container image
├── requirements.txt            # Python dependencies
├── ARCHITECTURE.md             # Architecture documentation
├── IMPLEMENTATION_GUIDE.md     # Detailed implementation guide
├── WEBHOOK_EXAMPLES.md         # Example webhook payloads
└── README.md                   # This file

Architecture

The server follows a clean, modular architecture:

graph TB
    GL[GitLab Server] -->|Webhook POST| API[FastAPI Server]
    API -->|Validate Token| AUTH[Auth Middleware]
    AUTH -->|Route Event| ROUTER[Webhook Router]
    ROUTER -->|Parse & Store| DB[(PostgreSQL)]
    API -->|Health Check| HEALTH[Health Endpoint]
    API -->|Auto Docs| DOCS[Swagger/ReDoc]

Loading

For detailed architecture information, see ARCHITECTURE.md.

Adding New Webhook Types

The architecture is designed for easy extensibility. To add a new webhook type:

1. Create Pydantic schema in app/schemas/new_event.py
2. Add endpoint in app/routers/webhooks.py:

@router.post("/new-event", status_code=status.HTTP_200_OK)
async def handle_new_event(
    request: Request,
    event: NewEventSchema,
    db: Session = Depends(get_db),
    token: str = Depends(verify_gitlab_token)
):
    webhook_event = WebhookEvent(
        event_type="new_event",
        event_name=event.event_name,
        project_id=event.project.id,
        project_name=event.project.name,
        payload=event.model_dump(),
        headers=dict(request.headers),
        received_at=datetime.utcnow()
    )
    
    db.add(webhook_event)
    db.commit()
    
    return {"status": "success", "event_id": str(webhook_event.id)}

3. Configure in GitLab with URL: /webhooks/new-event

No database migration needed! The JSONB payload field stores any event structure.

Security Considerations

Token Verification

All webhook endpoints require a valid X-Gitlab-Token header that matches the configured secret:

X-Gitlab-Token: your-secret-token

Production Recommendations

1. Use HTTPS: Always use TLS/SSL in production
2. Strong Secrets: Generate cryptographically secure tokens
3. Network Security: Restrict access to webhook endpoints
4. Rate Limiting: Implement rate limiting for webhook endpoints
5. Monitoring: Set up alerts for failed authentications
6. Database Security: Use strong database credentials
7. Environment Variables: Never commit secrets to version control

Monitoring & Observability

Health Check

Monitor server health:

curl http://localhost:8000/health

Response:

{
  "status": "ok",
  "timestamp": "2024-01-01T12:00:00.000000",
  "database": "healthy"
}

Logging

The server provides structured logging:

2024-01-01 12:00:00 - app.routers.webhooks - INFO - Received push event for project: MyProject
2024-01-01 12:00:00 - app.routers.webhooks - INFO - Stored push event with ID: 550e8400-e29b-41d4-a716-446655440000

Configure log level in .env:

LOG_LEVEL=INFO  # DEBUG, INFO, WARNING, ERROR, CRITICAL

Troubleshooting

Common Issues

Issue: Connection refused when connecting to PostgreSQL

Solution: Ensure PostgreSQL is running and the DATABASE_URL is correct

docker-compose up -d postgres

---

Issue: 401 Unauthorized when testing webhooks

Solution: Verify the X-Gitlab-Token header matches GITLAB_WEBHOOK_SECRET

---

Issue: 422 Validation Error when receiving webhooks

Solution: Check the webhook payload structure matches the Pydantic schema. Enable debug logging to see the full error.

---

Issue: Database migration fails

Solution: Reset the database and run migrations again:

alembic downgrade base
alembic upgrade head

Development

Running Tests

pytest tests/ -v

Database Migrations

Create a new migration:

alembic revision --autogenerate -m "Description of changes"

Apply migrations:

alembic upgrade head

Rollback migration:

alembic downgrade -1

Code Style

The project follows PEP 8 style guidelines. Format code with:

black app/
isort app/

Performance Considerations

• Connection Pooling: SQLAlchemy connection pool (10 connections, 20 max overflow)
• Async Operations: FastAPI async endpoints for better concurrency
• Database Indexes: Optimized queries on event_type, project_id, and created_at
• JSONB Storage: Efficient storage and querying of webhook payloads

Deployment

Docker Deployment

Build and deploy with Docker:

docker build -t gitlab-webhook-server .
docker run -p 8000:8000 --env-file .env gitlab-webhook-server

Production Deployment

For production deployments:

1. Use a managed PostgreSQL service (AWS RDS, Google Cloud SQL)
2. Deploy behind a reverse proxy (Nginx, Traefik)
3. Enable HTTPS with valid SSL certificates
4. Set up monitoring and alerting
5. Configure log aggregation
6. Implement backup strategies
7. Use container orchestration (Kubernetes, Docker Swarm)

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

License

[Your License Here]

Support

For issues and questions:

• GitHub Issues: [Your Repository URL]
• Documentation: See ARCHITECTURE.md and IMPLEMENTATION_GUIDE.md
• Examples: See WEBHOOK_EXAMPLES.md

Roadmap

Future enhancements:

• Add remaining GitLab webhook types (Pipeline, Deployment, Wiki, etc.)
• Implement webhook replay functionality
• Add webhook filtering and routing rules
• Create admin dashboard for webhook management
• Add webhook event streaming (WebSocket/SSE)
• Implement retry mechanism for failed processing
• Add HMAC signature verification
• Create webhook analytics and reporting
• Add GraphQL API for querying webhook events
• Implement webhook event archival strategy

Acknowledgments

Built with:

• FastAPI - Modern web framework
• SQLAlchemy - SQL toolkit and ORM
• Pydantic - Data validation
• Alembic - Database migrations
• PostgreSQL - Database system