gitlab-logger

Ships GitLab projects, pipelines, jobs, and full job traces into an existing Grafana Loki instance, so CI history stays searchable in Grafana after GitLab's own retention expires.

How it works

Three producers feed one collector pool:

• Webhook — a group-level GitLab webhook delivers pipeline and job events in near real time.
• Discovery — every DISCOVERY_INTERVAL, the configured groups are walked to pick up new projects.
• Sweeper — every SWEEP_INTERVAL, the most recent PIPELINES_PER_PROJECT pipelines of every known project are checked for jobs that were missed. This also performs the initial backfill. Projects are swept concurrently, up to SWEEP_CONCURRENCY at a time.

A SQLite ledger at STATE_PATH records which jobs have shipped, so webhook and sweep deliveries of the same job collapse into one. It also records when a settled pipeline's jobs have all shipped, so later sweeps skip re-listing that pipeline's jobs entirely — this is what keeps steady-state sweeps cheap.

Only jobs in a terminal state (success, failed, canceled, skipped) are collected — a running job's trace is incomplete.

Setup

Prerequisites

• Docker with the Compose plugin.
• A reachable Loki instance, and a Grafana with that Loki configured as a datasource.
• A self-managed GitLab instance, and permission to create a group access token and a group webhook.

1. Create a GitLab token

Group → Settings → Access tokens → Add new token

• Scope: read_api — nothing more. The service only issues GET requests.
• Role: Reporter or above on every group you want collected.

A personal access token works too, but a group token limits the blast radius and does not vanish when someone leaves.

If you configure several groups in GITLAB_GROUPS, the token must be able to read all of them.

To collect every group instead (see GITLAB_GROUPS=* below), the token must be able to see every group on the instance. A group access token only sees its own group, so * generally needs a personal access token with read_api — or an admin token to cover the whole instance.

2. Configure

git clone git@github.com:kpanuragh/gitlab_logger.git
cd gitlab_logger
cp .env.example .env
$EDITOR .env

Five values are required; everything else has a working default:

GITLAB_URL=https://git.example.com
GITLAB_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx
GITLAB_GROUPS=platform,infra          # comma-separated, subgroups included
LOKI_URL=http://loki:3100
WEBHOOK_SECRET=$(openssl rand -hex 32)

The service refuses to start if any of them is missing, naming the one it wants. .env is excluded from git and from the Docker build context — keep it that way.

Collecting every group. Set GITLAB_GROUPS=* instead of a list to collect every top-level group the token can see, with no other groups allowed alongside it. The group list is re-resolved every DISCOVERY_INTERVAL (default 1h), so a group created after startup is picked up automatically — no config edit, no restart. A group removed or renamed similarly drops out of collection on the next successful discovery pass; a discovery pass that fails to even list the groups changes nothing, so a transient outage never wipes what was already found. Note that * walks groups, so projects living in personal namespaces (a user's own project, not inside any group) are never included.

3. Prepare Loki before the first run

Do this before starting the service. The first sweep backfills history, and those entries carry their real timestamps — often months old. Loki rejects anything older than reject_old_samples_max_age (default 168h, one week), so with stock settings most of your backfill is silently dropped.

In your Loki config:

limits_config:
  reject_old_samples: false           # or raise reject_old_samples_max_age
  max_global_streams_per_user: 20000  # default 5000; see the note below
  allow_structured_metadata: true     # required; see below
  discover_service_name: []           # stop Loki inventing a service_name label
  discover_log_levels: false
  max_query_length: 4380h             # default 30d1h is too short to query CI history
  max_query_lookback: 4380h

ingester:
  max_chunk_age: 8760h                # out-of-order window is half of this

max_query_length counts the window plus the range selector. A 30-day dashboard using [$__range] costs 60 days, so the 30d1h default rejects a 30-day dashboard with the query time range exceeds the limit.

Two separate age limits, and the second one is the one that bites. reject_old_samples governs age relative to now. Separately, an entry may not be written more than max_chunk_age / 2 behind the newest entry already in its stream. At the 2h default that window is one hour — so as soon as a project's most recent pipeline is ingested, every older pipeline for that project is refused with entry too far behind, and the backfill fails even though reject_old_samples is off. A year of max_chunk_age allows roughly six months of backfill.

Structured metadata is mandatory, not optional. This service attaches job_id, pipeline_id, sha and friends as structured metadata, which Loki only accepts on schema v13 with the tsdb store. On an older schema every push fails with a 400 and nothing is ingested at all.

Stream count. Job traces are labelled by project, job_name, stage and status, so the stream count is roughly projects × distinct job names × statuses. Several hundred projects can pass Loki's default 5000 limit. If you would rather not raise it, drop job_name and stage from the labels in internal/transform/stream.go — they remain queryable as structured metadata.

4. Run

docker compose up -d --build
docker compose logs -f gitlab-logger

Check it came up:

curl -s localhost:8080/healthz    # "ok"    — process is alive
curl -s localhost:8080/readyz     # "ready" — GitLab and Loki are both reachable

If /readyz returns 503 it names the dependency that failed. The most common causes are a wrong LOKI_URL from inside the container, or a token missing read_api.

5. Confirm data is arriving

Discovery and the first sweep start immediately. Within a minute or so the logs show discovery complete with a project count, then job shipped lines.

In Grafana, against your Loki datasource:

{kind="job_trace"}

Then a real one — the last hour of failures across everything:

{kind="pipeline_event"} | json | event="job_finished" | status="failed"

The first sweep is the expensive one: it walks up to PIPELINES_PER_PROJECT pipelines for every project. Depending on how much history you have, a full backfill can take a while and is bounded by GITLAB_RPS. Steady state is far cheaper — completed pipelines are recorded in the ledger and skipped from then on.

6. Register the webhook

Everything works without this — the sweeper collects on its own — but a webhook cuts collection lag from up to SWEEP_INTERVAL down to seconds.

In GitLab: Group → Settings → Webhooks → Add new webhook

• URL: http://<host-reachable-from-gitlab>:8080/webhook
• Secret token: the exact value of WEBHOOK_SECRET
• Trigger: tick Pipeline events and Job events

Use "Test → Job events" to confirm delivery — GitLab shows the response code inline. A 401 means the secret does not match. To see the service side, set LOG_LEVEL=debug and watch for webhook handled, which logs at debug level.

One group-level webhook covers every project in that group, including ones created later.

7. Keep it running

docker compose up -d          # survives reboots via restart: unless-stopped

The ledger lives in the state volume. Do not delete it casually — it is what stops the service re-shipping everything it has already collected.

Trying it locally

docker-compose.yml above is the production path: it runs the collector alone, against a Loki you already operate. For evaluating gitlab-logger without standing up your own Grafana/Loki first, docker-compose.stack.yml brings up the whole thing in one shot — Loki, Grafana, and the collector — with Grafana pre-provisioned so CI data is visible with no manual datasource or dashboard setup.

docker compose -f docker-compose.stack.yml up -d --build

Then open Grafana at http://localhost:3001 (admin / admin — override the password with GRAFANA_ADMIN_PASSWORD) and look at the "GitLab CI" dashboard. The stack's Loki is pre-configured with the schema and limits this service needs (see deploy/loki/loki-config.yaml), so there is nothing to tune before the first run.

This stack still reads .env for GitLab credentials, but LOKI_URL is overridden in docker-compose.stack.yml to point at the in-stack Loki regardless of what .env says. It uses its own named volumes, so it never shares a ledger with a docker-compose.yml deployment running alongside it.

Note that a GitLab webhook cannot reach a locally-running collector — GitLab has no route to localhost on your machine — so a local run depends entirely on the sweeper to pick up data. Give it a few minutes after startup; see step 5 above for what to expect.

Tear it down (including its data) with:

docker compose -f docker-compose.stack.yml down -v

Troubleshooting

Symptom	Likely cause
/readyz 503 naming loki	LOKI_URL unreachable from inside the container — localhost there is not your host
/readyz 503 naming gitlab	Token lacks read_api, or GITLAB_URL is wrong
Starts, but no data in Grafana	Backfill rejected as too old — see step 3
Only recent jobs appear	Same cause: reject_old_samples_max_age
429 warnings in the logs	Lower GITLAB_RPS; the service backs off and retries on its own
Webhook shows 401 in GitLab	WEBHOOK_SECRET mismatch
Sweeps never finish	Raise SWEEP_CONCURRENCY and GITLAB_RPS, or lower PIPELINES_PER_PROJECT
job abandoned after max attempts	A job failed MAX_ATTEMPTS times. Fix the cause, then DELETE FROM jobs WHERE state='dead'; in the ledger to let it retry

Configuration

Variable	Default	Purpose
GITLAB_URL	required	Base URL, e.g. https://git.example.com
GITLAB_TOKEN	required	Token with read_api scope
GITLAB_GROUPS	required	Comma-separated group paths, or * for every group the token can see
LOKI_URL	required	e.g. http://loki:3100
LOKI_TENANT_ID	unset	Sets X-Scope-OrgID for multi-tenant Loki
WEBHOOK_SECRET	required	Compared against X-Gitlab-Token
WEBHOOK_ADDR	:8080	Listen address
PIPELINES_PER_PROJECT	100	Sweep/backfill depth per project
SWEEP_CONCURRENCY	8	Projects swept in parallel
DISCOVERY_INTERVAL	1h	Group re-walk interval
SWEEP_INTERVAL	15m	Safety-net poll interval
WORKERS	4	Concurrent job collectors
GITLAB_RPS	10	Client-side API rate limit
MAX_TRACE_BYTES	5242880	Trace cap; the middle is elided past this
MAX_ATTEMPTS	3	Attempts before a job is marked dead (see below)
RETRY_DELAY	15s	Delay before retrying a job collection failure
STATE_PATH	/data/state.db	SQLite ledger
LOG_LEVEL	info	debug, info, warn, error

Tuning for a large number of projects (200+)

The two knobs that matter at this scale are SWEEP_CONCURRENCY and GITLAB_RPS:

• SWEEP_CONCURRENCY controls how many projects are swept in parallel. Sweeping is latency-bound — most of the time is spent waiting on GitLab's response, not on CPU — so raising this hides that latency instead of leaving most of a sweep interval idle.
• GITLAB_RPS is a single client-side rate limiter shared across every goroutine, sweeper and collector alike. Raising SWEEP_CONCURRENCY therefore cannot push more requests per second at GitLab than GITLAB_RPS allows; it only lets more of those requests be in flight at once. Raise GITLAB_RPS itself if you want a higher ceiling, and check with GitLab about the API rate limit applied to your token first.

With 200+ projects, size SWEEP_INTERVAL so a full sweep round comfortably finishes within it, and see the first note under Operational notes below: the first sweep after startup is far more expensive than every one after it.

Querying in Grafana

Read a job's log:

{kind="job_trace", project="platform/api", job_name="unit-tests", status="failed"}

One specific job:

{kind="job_trace", project="platform/api"} | job_id="40231"

Failed jobs per project, last hour:

sum by (project) (
  count_over_time({kind="pipeline_event"} | json | event="job_finished" | status="failed" [1h])
)

p95 job duration by job name:

quantile_over_time(0.95,
  {kind="pipeline_event"} | json | event="job_finished"
  | unwrap duration_sec [1h]
) by (project)

Search all failures for a message:

{kind="job_trace", status="failed"} |= "connection refused"

Label design

Labels are deliberately few, because Loki performance depends on keeping stream count low:

• Labels: kind, gitlab, group, project, plus job_name, stage, and status on job traces.
• Structured metadata (filter with | job_id="…"): job_id, pipeline_id, sha, ref, user, runner, web_url.

If your Loki is shared and tight on stream limits, dropping job_name and stage from the labels in internal/transform/stream.go reduces stream count substantially.

Development

go test ./... -race
go build ./...

The end-to-end test starts a real Loki container and needs Docker:

go test -tags e2e ./test/e2e/ -v -timeout 10m

CI

Every push and pull request runs .github/workflows/ci.yml:

• test — build, go vet, a gofmt -l check, and the full test suite under -race.
• e2e — the Docker-dependent end-to-end test above, on a ubuntu-latest runner (Docker is preinstalled there).
• publish — only after test and e2e both pass, and only on pushes to main or v* tags (never on pull requests), a multi-arch (linux/amd64, linux/arm64) image is built and pushed to ghcr.io/kpanuragh/gitlab_logger.

Pull the published image instead of building locally:

docker pull ghcr.io/kpanuragh/gitlab_logger:latest

services:
  gitlab-logger:
    image: ghcr.io/kpanuragh/gitlab_logger:latest
    restart: unless-stopped
    env_file: .env
    ports:
      - "8080:8080"
    volumes:
      - state:/data

volumes:
  state:

The package defaults to private on its first publish. Make it public once, either from the repo's Packages page or:

gh api --method PATCH /user/packages/container/gitlab_logger -f visibility=public

Operational notes

• The first sweep is the expensive one; steady state is cheap. On a fresh deployment (or after a gap), the sweeper lists every pipeline's jobs to backfill history — this is the costly path. Once a settled pipeline's jobs have all shipped, the ledger marks it done and later sweeps skip re-listing its jobs entirely, so steady-state sweeps only re-check pipelines that are still open or new. Expect the first sweep after startup, or after adding a large group, to take noticeably longer and issue far more GitLab API calls than every sweep after it.
• A pipeline with no created_at and no updated_at gets no pipeline-level event. It cannot be de-duplicated without a timestamp to key it on, so emitting one would just accumulate duplicate lines in Loki on every sweep. Its jobs and traces are still collected and shipped normally; only the pipeline_finished record for that pipeline is skipped.
• A job that fails MAX_ATTEMPTS times is marked dead and stops being retried. Its pipeline, however, is deliberately NOT sealed: a settled pipeline is only marked done once every terminal job has actually shipped, so a pipeline containing a dead job is re-listed on every sweep. This costs one extra job listing per sweep for that pipeline, but it means clearing the dead row from the SQLite ledger (or fixing whatever made the job unrecoverable) lets the next sweep pick it up again, instead of losing it and its pipeline permanently. Dead jobs should be rare; check logs for "job abandoned after max attempts" to find them.
• Restarts are safe. The ledger persists in the /data volume; work resumes where it stopped.
• Delivery is at-least-once. A job is only marked shipped after Loki accepts it, so a crash mid-push results in a re-send. Loki de-duplicates identical entries within a stream.
• Single replica. The SQLite ledger assumes one writer. Scale with WORKERS and SWEEP_CONCURRENCY, not with replicas.