High-Throughput Webhook Processor

Project Overview

A scalable webhook processing service built with Node.js and Express. This service receives webhook events via HTTP, validates them, processes them through a configurable in-memory FIFO queue with worker pool, and exposes Prometheus-compatible metrics for monitoring. Designed for high throughput with built-in rate limiting, retry logic, and graceful shutdown capabilities.

Technologies Used

Node.js
Express.js
Pino (structured logging)
Prometheus client (metrics)
Jest & Supertest (testing)
Dotenv (configuration)

System Architecture

The service follows a queue-based architecture:

HTTP Server: Receives webhook events via POST requests
HTTP-level rate limiting: In-memory request threshold (429 on overload)
Validation Layer: Ensures payloads match expected schema
Queue: In-memory FIFO queue with configurable size limits
Worker Pool: Configurable concurrency for processing webhooks
Queue overload protection: Backpressure via bounded FIFO queue
Retry Logic: Failed webhooks are retried with configurable delay
Metrics: Prometheus-compatible metrics for monitoring

sequenceDiagram
    participant Client
    participant Server as HTTP Server
    participant RateLimiter
    participant Validation
    participant Queue
    participant Worker
    participant Processor
    participant Metrics
    participant Prometheus
    
    Client->>Server: POST /webhook
    Server->>RateLimiter: check request count
    alt below threshold
        RateLimiter->>Validation: pass through
    else above threshold
        RateLimiter-->>Client: 429 Too Many Requests
    end
    Validation->>Queue: Add to FIFO queue
    Server-->>Client: 202 Accepted
    Queue->>Worker: Dequeue when worker available
    Worker->>Processor: Process webhook
    Processor-->>Worker: Processing result
    Worker->>Metrics: Record processing metrics
    Prometheus->>Metrics: Scrape /metrics endpoint

Features

Configurable concurrency for webhook processing
HTTP-level rate limiting: in-memory request threshold (429 on overload)
Queue overload protection: backpressure via bounded FIFO queue
Comprehensive metrics for monitoring and alerting
Structured JSON logging for better observability
Graceful shutdown handling for zero downtime deployments
Configurable retry logic for failed webhooks
Health check endpoint for monitoring
Prometheus-compatible metrics endpoint

Project Structure

├── src/
│   ├── middleware/
│   │   ├── validateJson.js
│   │   ├── errorHandler.js
│   │   └── logging.js
│   ├── routes/
│   │   ├── webhook.js
│   │   ├── metrics.js
│   │   └── health.js
│   ├── services/
│   │   ├── queue.js      
│   │   ├── processor.js
│   │   └── workerPool.js
│   └── utils/
│       ├── metrics.js
│       └── logger.js
│   ├── config.js
│   ├── index.js
│   ├── server.js
└── tests/
    ├── health.test.js
    ├── webhook.test.js
    ├── gracefulShutdown.test.js
    ├── metrics.test.js
    ├── queue.test.js
    ├── processor.test.js
    └── webhookRateLimit.test.js
├── package.json
├── jest.config.mjs
├── load-test.js
├── README.md

Prerequisites

Node.js (v14 or higher)
npm (v6 or higher)

Installation

Clone the repository and install dependencies:

git clone https://github.com/SamMintah/webhook_processor
cd webhook_processor
npm install

Configuration

The application can be configured using the following environment variables:

Variable	Description	Default
`PORT`	HTTP server port	`3000`
`LOG_LEVEL`	Logging level (debug, info, warn, error)	`info`
`CONCURRENCY`	Maximum number of concurrent webhook processing tasks	`10`
`OVERLOAD_THRESHOLD`	Maximum queue size before rejecting new webhooks	`100`
`PROCESSING_TIME_MS`	Processing time in ms (for simulation)	`1000`
`FAILURE_RATE`	Simulated failure rate (0-1)	`0.1`
`MAX_RETRIES`	Maximum number of retries for failed webhooks	`1`
`RETRY_DELAY_MS`	Delay between retry attempts in ms	`2000`

Running the Service

Start the server in production mode:

npm start

For development with auto-restart:

npm run dev

Run the test suite:

npm test

Run load tests:

npm run load-test

API Endpoints

POST /webhook

Receives webhook events for processing.

Headers:

Content-Type: application/json

Request Body:

{
  "event": "user.created",
  "data": {
    "id": "123",
    "name": "John Doe"
  }
}

Responses:

202 Accepted: Webhook accepted for processing
400 Bad Request: Invalid payload
429 Too Many Requests: Queue is full, try again later

Example:

curl -X POST http://localhost:3000/webhook \
  -H "Content-Type: application/json" \
  -d '{"event":"user.created","data":{"id":"123","name":"John Doe"}}'

GET /metrics

Returns Prometheus-compatible metrics.

Example:

curl http://localhost:3000/metrics

Example Response:

# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 6.3391

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 1.015978

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 7.355078

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1745277103

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 65175552

# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0

# HELP nodejs_eventloop_lag_min_seconds The minimum recorded event loop delay.
# TYPE nodejs_eventloop_lag_min_seconds gauge
nodejs_eventloop_lag_min_seconds 0.008593408

# HELP nodejs_eventloop_lag_max_seconds The maximum recorded event loop delay.
# TYPE nodejs_eventloop_lag_max_seconds gauge
nodejs_eventloop_lag_max_seconds 0.117702655

# HELP nodejs_eventloop_lag_mean_seconds The mean of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_mean_seconds gauge
nodejs_eventloop_lag_mean_seconds 0.01099094367336187

# HELP nodejs_eventloop_lag_stddev_seconds The standard deviation of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_stddev_seconds gauge
nodejs_eventloop_lag_stddev_seconds 0.00592045845358732

# HELP nodejs_eventloop_lag_p50_seconds The 50th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p50_seconds gauge
nodejs_eventloop_lag_p50_seconds 0.010387455

# HELP nodejs_eventloop_lag_p90_seconds The 90th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p90_seconds gauge
nodejs_eventloop_lag_p90_seconds 0.011263999

# HELP nodejs_eventloop_lag_p99_seconds The 99th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p99_seconds gauge
nodejs_eventloop_lag_p99_seconds 0.017661951

# HELP nodejs_active_resources Number of active resources that are currently keeping the event loop alive, grouped by async resource type.
# TYPE nodejs_active_resources gauge
nodejs_active_resources{type="TTYWrap"} 2
nodejs_active_resources{type="TCPServerWrap"} 1
nodejs_active_resources{type="TCPSocketWrap"} 1
nodejs_active_resources{type="Immediate"} 1

# HELP nodejs_active_resources_total Total number of active resources.
# TYPE nodejs_active_resources_total gauge
nodejs_active_resources_total 5

# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="WriteStream"} 2
nodejs_active_handles{type="Server"} 1
nodejs_active_handles{type="Socket"} 1

# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 4

# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge

# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 0

# HELP nodejs_heap_size_total_bytes Process heap size from Node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 10711040

# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 9651040

# HELP nodejs_external_memory_bytes Node.js external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 2575049

# HELP nodejs_heap_space_size_total_bytes Process heap space size total from Node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 0
nodejs_heap_space_size_total_bytes{space="new"} 1048576
nodejs_heap_space_size_total_bytes{space="old"} 8081408
nodejs_heap_space_size_total_bytes{space="code"} 1310720
nodejs_heap_space_size_total_bytes{space="shared"} 0
nodejs_heap_space_size_total_bytes{space="new_large_object"} 0
nodejs_heap_space_size_total_bytes{space="large_object"} 270336
nodejs_heap_space_size_total_bytes{space="code_large_object"} 0
nodejs_heap_space_size_total_bytes{space="shared_large_object"} 0

# HELP nodejs_heap_space_size_used_bytes Process heap space size used from Node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 0
nodejs_heap_space_size_used_bytes{space="new"} 390680
nodejs_heap_space_size_used_bytes{space="old"} 7845896
nodejs_heap_space_size_used_bytes{space="code"} 1161152
nodejs_heap_space_size_used_bytes{space="shared"} 0
nodejs_heap_space_size_used_bytes{space="new_large_object"} 0
nodejs_heap_space_size_used_bytes{space="large_object"} 262160
nodejs_heap_space_size_used_bytes{space="code_large_object"} 0
nodejs_heap_space_size_used_bytes{space="shared_large_object"} 0

# HELP nodejs_heap_space_size_available_bytes Process heap space size available from Node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 0
nodejs_heap_space_size_available_bytes{space="new"} 640200
nodejs_heap_space_size_available_bytes{space="old"} 94656
nodejs_heap_space_size_available_bytes{space="code"} 0
nodejs_heap_space_size_available_bytes{space="shared"} 0
nodejs_heap_space_size_available_bytes{space="new_large_object"} 1048576
nodejs_heap_space_size_available_bytes{space="large_object"} 0
nodejs_heap_space_size_available_bytes{space="code_large_object"} 0
nodejs_heap_space_size_available_bytes{space="shared_large_object"} 0

# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v20.17.0",major="20",minor="17",patch="0"} 1

# HELP nodejs_gc_duration_seconds Garbage collection duration by kind, one of major, minor, incremental or weakcb.
# TYPE nodejs_gc_duration_seconds histogram
nodejs_gc_duration_seconds_bucket{le="0.001",kind="minor"} 0
nodejs_gc_duration_seconds_bucket{le="0.01",kind="minor"} 19
nodejs_gc_duration_seconds_bucket{le="0.1",kind="minor"} 19
nodejs_gc_duration_seconds_bucket{le="1",kind="minor"} 19
nodejs_gc_duration_seconds_bucket{le="2",kind="minor"} 19
nodejs_gc_duration_seconds_bucket{le="5",kind="minor"} 19
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="minor"} 19
nodejs_gc_duration_seconds_sum{kind="minor"} 0.0537080210149288
nodejs_gc_duration_seconds_count{kind="minor"} 19
nodejs_gc_duration_seconds_bucket{le="0.001",kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="0.01",kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="0.1",kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="1",kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="2",kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="5",kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="incremental"} 8
nodejs_gc_duration_seconds_sum{kind="incremental"} 0.004048804000020027
nodejs_gc_duration_seconds_count{kind="incremental"} 8
nodejs_gc_duration_seconds_bucket{le="0.001",kind="major"} 0
nodejs_gc_duration_seconds_bucket{le="0.01",kind="major"} 7
nodejs_gc_duration_seconds_bucket{le="0.1",kind="major"} 8
nodejs_gc_duration_seconds_bucket{le="1",kind="major"} 8
nodejs_gc_duration_seconds_bucket{le="2",kind="major"} 8
nodejs_gc_duration_seconds_bucket{le="5",kind="major"} 8
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="major"} 8
nodejs_gc_duration_seconds_sum{kind="major"} 0.03212324298918247
nodejs_gc_duration_seconds_count{kind="major"} 8

# HELP webhook_total_received Total number of webhook requests received
# TYPE webhook_total_received counter
webhook_total_received 3212

# HELP webhook_total_processed Total number of webhook requests successfully processed
# TYPE webhook_total_processed counter
webhook_total_processed 1209

# HELP webhook_too_many_requests Total number of webhook requests rejected due to queue overload
# TYPE webhook_too_many_requests counter
webhook_too_many_requests 1988

# HELP webhook_queue_length Current length of the webhook processing queue
# TYPE webhook_queue_length gauge
webhook_queue_length 9

# HELP webhook_processing_time_ms Webhook processing time in milliseconds
# TYPE webhook_processing_time_ms histogram
webhook_processing_time_ms_bucket{le="10"} 0
webhook_processing_time_ms_bucket{le="50"} 0
webhook_processing_time_ms_bucket{le="100"} 7
webhook_processing_time_ms_bucket{le="200"} 514
webhook_processing_time_ms_bucket{le="500"} 1198
webhook_processing_time_ms_bucket{le="1000"} 1209
webhook_processing_time_ms_bucket{le="2000"} 1209
webhook_processing_time_ms_bucket{le="5000"} 1209
webhook_processing_time_ms_bucket{le="+Inf"} 1209
webhook_processing_time_ms_sum 271348
webhook_processing_time_ms_count 1209

# HELP webhook_queue_time_ms Time items spend in the queue in ms
# TYPE webhook_queue_time_ms histogram
webhook_queue_time_ms_bucket{le="10"} 10
webhook_queue_time_ms_bucket{le="50"} 10
webhook_queue_time_ms_bucket{le="100"} 10
webhook_queue_time_ms_bucket{le="200"} 17
webhook_queue_time_ms_bucket{le="500"} 39
webhook_queue_time_ms_bucket{le="1000"} 83
webhook_queue_time_ms_bucket{le="+Inf"} 1224
webhook_queue_time_ms_sum 2327855
webhook_queue_time_ms_count 1224

GET /health

Returns the health status of the service.

Example:

curl http://localhost:3000/health

Example Response:

{
  "status": "ok",
  "uptime": 1234.56,
  "timestamp": "2025-04-21T12:34:56.789Z"
}

Testing

The service includes a comprehensive test suite using Jest and Supertest.

Running Tests

# Run all tests
npm test

# Run tests with coverage
npm run test:coverage

Example Test with Supertest

import request from 'supertest';
import { app } from '../src/server.js';

describe('Webhook API', () => {
  it('should accept a valid webhook', async () => {
    const response = await request(app)
      .post('/webhook')
      .send({
        event: 'user.created',
        data: { id: '123', name: 'Test User' }
      });
    
    expect(response.status).toBe(202);
  });
});

Load Testing

The service includes a load testing script to simulate high traffic scenarios.

Running Load Tests

# Run the load test script
npm run load-test

The load test script sends a configurable number of concurrent webhook requests and reports:

Success/failure rates
Response times (min, max, average)
Throughput (requests per second)
Queue overflow events

You can customize the load test parameters by modifying the environment variables:

LOAD_TEST_CONCURRENCY: Number of concurrent requests
LOAD_TEST_TOTAL: Total number of requests to send
LOAD_TEST_RATE: Rate limiting (requests per second)

Logging

The application uses Pino for structured JSON logging, providing detailed information for production environments and monitoring systems.

Configuration

Logging can be configured using the following environment variables:

Variable	Description	Default
`LOG_LEVEL`	Logging level (debug, info, warn, error)	`info`
`NODE_ENV`	Environment setting that affects log formatting	`production`

Log Formats

Production Mode: By default, logs are output in JSON format for easy parsing by log aggregation tools.
Development Mode: When running in development mode (NODE_ENV=development), logs are pretty-printed for better readability.

Pretty Printing

For human-readable logs during development:

Ensure pino-pretty is installed as a dev dependency:
```
npm install --save-dev pino-pretty
```
Run the application in development mode:
```
NODE_ENV=development npm run dev
```

Log Structure

All logs include:

Timestamp in ISO format
Log level
Request ID (for HTTP requests)
Processing time (for HTTP requests)
User agent (for HTTP requests)
Custom application context

Example Log Output

JSON format (production):

{"level":30,"time":"2025-04-21T12:34:56.789Z","pid":1234,"hostname":"server","reqId":"550e8400-e29b-41d4-a716-446655440000","req":{"method":"POST","url":"/webhook","headers":{}},"msg":"Webhook received"}

Pretty-printed format (development):

[2025-04-21 12:34:56.789] INFO (webhook-processor/1234): Webhook received
    reqId: 550e8400-e29b-41d4-a716-446655440000
    req: {
      "method": "POST",
      "url": "/webhook"
    }

Metrics and Monitoring

The application exposes Prometheus-compatible metrics at the /metrics endpoint. These metrics include:

Standard Node.js metrics (memory, CPU, etc.)
Webhook-specific metrics:
- Total webhooks received
- Webhook processing duration
- Success/failure counts
- Retry counts
Queue metrics:
- Current queue size
- Items being processed
- Queue overflow events

These metrics can be scraped by Prometheus and visualized using Grafana or similar tools.

Performance Characteristics

Throughput: The service can handle approximately 500 requests per second with default concurrency settings (10 workers). This can be scaled up by increasing the CONCURRENCY setting.
Latency: Average queue wait time is typically under 50ms at moderate load. P95 queue wait times can reach 200ms during high traffic periods.
Scaling Knobs:
- Increasing CONCURRENCY improves throughput but may increase CPU usage
- Adjusting OVERLOAD_THRESHOLD controls memory usage and backpressure sensitivity
- Tuning PROCESSING_TIME_MS and RETRY_DELAY_MS affects overall system latency
Resource Usage: Memory usage scales linearly with queue size and is typically 100-200MB under normal operation.

Deployment Notes

Containerization: A simple Dockerfile is recommended for deployment:

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
CMD ["node", "src/index.js"]

Environment Configuration:
- Use .env files for local development
- In production, use orchestration platform secrets (Kubernetes Secrets, AWS Parameter Store, etc.)
- All configuration options are available as environment variables
Graceful Shutdown:
- The service handles SIGTERM and SIGINT signals to drain the queue before shutting down
- In Kubernetes, configure preStop hooks to allow for graceful termination:
```
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
```
- Set appropriate termination grace periods (30s recommended)
Monitoring Setup:
- Configure Prometheus to scrape the /metrics endpoint
- Example Prometheus configuration:
```
scrape_configs:
  - job_name: 'webhook-processor'
    scrape_interval: 15s
    static_configs:
      - targets: ['webhook-processor:3000']
```
- Set up alerts for queue size, error rates, and processing latency

Design Decisions and Trade-offs

Queue-based Architecture

The service uses an in-memory queue to decouple webhook reception from processing. This allows the HTTP server to respond quickly while processing happens asynchronously, improving throughput and user experience.

Concurrency Model

The worker pool model with configurable concurrency allows the service to be tuned for different workloads and hardware. This provides a balance between maximizing throughput and preventing system overload.

Rate Limiting Strategy

The service implements two layers of rate limiting:

HTTP-level rate limiting: An in-memory counter tracks incoming request rates and rejects requests with 429 status code when the threshold is exceeded.
Queue-based backpressure: When the queue reaches the configured threshold, new requests are rejected with a 429 status code, encouraging clients to implement exponential backoff.

This dual approach provides protection at both the HTTP server level and the processing queue level.

Retry Logic

Failed webhook processing is automatically retried with configurable delays and maximum attempts. This improves reliability without requiring clients to implement their own retry logic.

Observability

The service prioritizes observability through structured logging, detailed metrics, and a health check endpoint. This makes it easier to monitor, troubleshoot, and optimize in production environments.

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
src		src
tests		tests
.gitignore		.gitignore
README.md		README.md
jest.config.mjs		jest.config.mjs
load-test.js		load-test.js
package-lock.json		package-lock.json
package.json		package.json

Folders and files

Latest commit

History

Repository files navigation

High-Throughput Webhook Processor

Project Overview

Technologies Used

System Architecture

Features

Project Structure

Prerequisites

Installation

Configuration

Running the Service

API Endpoints

POST /webhook

GET /metrics

GET /health

Testing

Running Tests

Example Test with Supertest

Load Testing

Running Load Tests

Logging

Configuration

Log Formats

Pretty Printing

Log Structure

Example Log Output

Metrics and Monitoring

Performance Characteristics

Deployment Notes

Design Decisions and Trade-offs

Queue-based Architecture

Concurrency Model

Rate Limiting Strategy

Retry Logic

Observability

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages