Async Jobs

Async Jobs API

Reference for the background job queue in eagle-api. It runs long sysadmin operations off the request path. One job type exists: CSV export of a project's documents.

No API route creates a job. The API only reads jobs that already exist in the queue. To export a project's documents, run tools/export-project-docs.js (see Files below).

How it works

Agenda (MongoDB-backed queue, agendaJobs collection)
    │
    ▼
One pod claims the job atomically (findOneAndUpdate on lockedAt)
    │
    ▼
exportProjectDocs() → result stored in job.attrs.data.result
    │
    ▼
GET /api/jobs/:id/download → streams CSV

eagle-api runs several load-balanced pods, so the queue has to be shared. Agenda.js v5 stores jobs in the existing MongoDB connection and claims each one with a single findOneAndUpdate({lockedAt: null}), so exactly one pod runs it. No Redis or SQS to operate.

Agenda polls every 5 seconds. A pod that dies mid-job releases its claim after defaultLockLifetime (5 minutes), and the next pod to poll retries the job.

The job type is the Agenda job name. Job data holds projectId, includeAll (default false, Published documents only), requestedBy and requestedAt. The CSV is stored inline at job.attrs.data.result.csv. Project CSVs run 50–500 KB, well under MongoDB's 16 MB document limit.

Endpoints

All require a sysadmin Bearer token. A sysadmin may read any job regardless of owner; other users only their own (api/controllers/job.js).

Method	Path	Returns
GET	/api/jobs	200 — the caller's 20 most recent jobs
GET	/api/jobs/{jobId}	200 — status record
GET	/api/jobs/{jobId}/download	200 text/csv; 202 if still queued or running; 422 if failed

Status record

{
  "jobId": "683f1a2b3c4d5e6f7a8b9c0d",
  "type": "project-doc-export",
  "status": "running",
  "projectId": "588511ffaaecd9001b8291b7",
  "requestedBy": "someone@example.com",
  "createdAt": "2026-06-15T14:30:00.000Z",
  "completedAt": null,
  "progress": null,
  "hasResult": false,
  "filename": null
}

Status is derived from the Agenda record, not stored: failed if failedAt is set, else completed if lastFinishedAt is set, else running if lockedAt is set, else queued.

Clients poll /api/jobs/{jobId} every few seconds until status is completed or failed, then call the download path.

project-doc-export

Exports a project's documents to CSV. Columns:

Document Name, Legislation, Document Type, Author, Milestone, Project Phase, Document Date, Upload Date, Description, Labels

Runs in roughly 2–30 seconds depending on document count.

Files

File	Purpose
api/helpers/jobQueue.js	Agenda setup and job definitions
api/controllers/job.js	Controller: protectedOptions, listJobs, getJobStatus, downloadJobResult
api/helpers/export-docs-helper.js	CSV generation, shared with the CLI tool
api/swagger/swagger.yaml	Route definitions under /jobs
tools/export-project-docs.js	CLI wrapper

The agendaJobs collection

Each document carries:

• name — job type
• data — parameters, result, and requestedBy
• lockedAt — when a pod claimed it; null means unclaimed
• lastFinishedAt — completion timestamp
• failedAt / failReason — error detail

Finished records stay in the collection. No route deletes them.

Related pages

• Bulk Document Download — a proposed second job type
• API Architecture