ParamCollector Gateway

Backend API for collecting and querying HTTP parameter statistics from Burp Suite extensions.

Architecture

Burp Suite Extension
  → HTTP Gateway API
  → Redis batch deduplication
  → ClickHouse AggregatingMergeTree

Services

• Spring Boot Gateway API — HTTP API layer.
• Redis — atomic batch deduplication via SET NX EX.
• ClickHouse — analytics storage for aggregate states.

Authentication

All protected API endpoints require the following header:

X-API-Token: <token>

Public endpoints:

GET /api/v1/health
GET /actuator/health
GET /v3/api-docs
GET /swagger-ui/index.html

Environment Variables

SERVER_PORT=8000

API_TOKEN=dev-paramcollector-token

CLICKHOUSE_JDBC_URL=jdbc:clickhouse://127.0.0.1:8123/paramcollector?clickhouse.jdbc.v1=true
CLICKHOUSE_USER=paramcollector
CLICKHOUSE_PASSWORD=paramcollector_pass

REDIS_HOST=127.0.0.1
REDIS_PORT=6379

clickhouse.jdbc.v1=true is required for JDBC batch insert through input(...).

ClickHouse Tables

collections

Stores collection metadata.

collection_id UUID
name String
description String

stats_agg

Stores aggregate states.

collection_id UUID
param_name String
location LowCardinality(String)

hosts_state AggregateFunction(uniqCombined64, UInt64)
endpoints_state AggregateFunction(uniqCombined64, UInt64)
methods_state AggregateFunction(uniqCombined64, UInt64)
contexts_state AggregateFunction(uniqExact, UInt64)

Batch Deduplication

Redis is used with atomic SET NX EX.

Flow:

1. Reserve batch_id as processing.
2. Insert observations into ClickHouse.
3. Mark batch_id as accepted.
4. If insert fails, delete Redis key to allow retry.

API Endpoints

Health

GET /api/v1/health

Collections

POST /api/v1/collections
GET /api/v1/collections?page=0&size=20
DELETE /api/v1/collections/{collectionId}
POST /api/v1/collections/{collectionId}/cleanup

Observations

POST /api/v1/collections/{collectionId}/observations/batch

Statistics

GET /api/v1/collections/{collectionId}/params
GET /api/v1/collections/{collectionId}/wordlist

Query Parameters

GET /api/v1/collections

Parameter	Default	Description
page	0	Zero-based page number.
size	20	Page size.

GET /api/v1/collections/{collectionId}/params

Parameter	Default	Description
page	0	Zero-based page number.
size	20	Page size.
sort	score	Sort field: score, unique_hosts, unique_endpoints, unique_methods, unique_contexts, param_name.
direction	desc	Sort direction: asc or desc.
locations	omitted	Optional list of locations, for example query,json,body_form,multipart.

GET /api/v1/collections/{collectionId}/wordlist

Parameter	Default	Description
limit	1000	Maximum number of parameters to export.
locations	omitted	Optional list of locations, for example query,json,body_form,multipart.

Examples

Create Collection

curl -H "X-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST http://127.0.0.1:8000/api/v1/collections \
  -d '{
    "name": "test collection",
    "description": "local development"
  }'

List Collections

curl -H "X-API-Token: $API_TOKEN" \
  "http://127.0.0.1:8000/api/v1/collections?page=0&size=20"

Upload Observations Batch

curl -H "X-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST http://127.0.0.1:8000/api/v1/collections/<COLLECTION_ID>/observations/batch \
  -d '{
    "batchId": "11111111-1111-1111-1111-111111111111",
    "items": [
      {
        "hostHash": 100,
        "endpointHash": 200,
        "contextHash": 300,
        "methodHash": 400,
        "location": "query",
        "paramName": "returnUrl"
      }
    ]
  }'

Query Parameter Statistics

curl -H "X-API-Token: $API_TOKEN" \
  "http://127.0.0.1:8000/api/v1/collections/<COLLECTION_ID>/params?page=0&size=20&sort=score&direction=desc&locations=query,json"

Download Wordlist

curl -OJ -H "X-API-Token: $API_TOKEN" \
  "http://127.0.0.1:8000/api/v1/collections/<COLLECTION_ID>/wordlist?limit=10000&locations=query,json,body_form"

Cleanup Collection

curl -H "X-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST http://127.0.0.1:8000/api/v1/collections/<COLLECTION_ID>/cleanup \
  -d '{
    "keepTop": 10000,
    "location": "query"
  }'

Swagger UI

Swagger UI is available at:

http://127.0.0.1:8000/swagger-ui/index.html

Use Authorize and set X-API-Token.

Current Limitations

• ClickHouse does not enforce strict uniqueness for collection_id.
• collection_id is generated server-side using UUID.randomUUID().
• Cleanup uses ClickHouse mutations and may not complete instantly.
• Redis batch deduplication keys currently use TTL-based retention.
• stats_agg stores aggregate states only, not raw HTTP requests.
• Parameter names are stored case-sensitive.
• Wordlist export is streamed from the database result set.