m-bulk-manager-go

Bulk and async process orchestration service. Manages long-running batch jobs: tracks progress in MySQL, routes work through RabbitMQ, and dispatches items to target applications over HTTP.

Business

Bulk actions (mass updates, exports, notifications, etc.) are modeled as bulk processes. Each process has a unique key, item count, target service, and lifecycle status (created → processing → finished / failed / paused).

Capability	Description
Process registry	Create, list, filter, update, and delete bulk processes
Lifecycle control	Start, pause, resume, cancel; enforce valid status transitions
Progress tracking	Aggregate processed / failed counts and per-item error logs
Work dispatch	Consume per-job RabbitMQ queues and call target /api/bulk/consume endpoints
Chunk mode	Process items one-by-one (chunk_size=1) or in Redis-backed batches (chunk_size>1)
Parent/child pipelines	Auto-start child processes when a parent finishes
Notifications	Notify external live-notification service on create/update/delete
Tagging	Attach searchable tags to processes for filtering and routing
Service registry	Resolve target service URLs from configuration or schedulable-service records

Primary consumers: workflow applications that publish bulk work and poll process status via the REST API.

Technical

Two binaries share one module and Docker image:

Binary	Role
cmd/api	HTTP API on :8012 + embedded bulk.process inform consumer
cmd/worker	Fixed worker pool: discovers processing jobs, acquires Redis lease, consumes bulk.tmp-{key}

Compared to a pod-per-job model, v1 uses a worker pool with Redis leases so only one worker handles a given process key at a time. No Kubernetes Pod API on the hot path.

                    ┌─────────────────────────────┐
                    │  Client (bulk producer app) │
                    └──────────────┬──────────────┘
                                   │ REST
                                   ▼
                    ┌─────────────────────────────┐
                    │         cmd/api               │
                    │  /v1/bulkProcesses          │
                    │  + bulk.process consumer    │
                    └──────┬──────────────┬───────┘
                           │              │
              ┌────────────┘              └────────────┐
              ▼                                        ▼
         ┌─────────┐                            ┌───────────┐
         │  MySQL  │                            │ RabbitMQ  │
         │ state   │                            │ tmp +     │
         └────┬────┘                            │ inform Q  │
              │                                 └─────┬─────┘
              │ poll processing                     │
              ▼                                       │
    ┌─────────────────────┐      HTTP consume       │
    │   cmd/worker × N    │ ───────────────────────►│ Target app
    │  lease + tmp queue  │                         │
    └──────────┬──────────┘                         │
               ▼                                     │
          ┌─────────┐                                │
          │  Redis  │◄── chunk lists, leases ────────┘
          └─────────┘

Layered layout

cmd/api, cmd/worker
internal/domain          entities, value objects, policies, repository interfaces
internal/application     use cases, DTOs, ports
internal/infrastructure  MySQL, Redis, RabbitMQ, HTTP clients, notifications
internal/transport       HTTP handlers, routing, middleware
internal/bootstrap       composition root (manual DI)
internal/worker          supervisor loop, lease, consume
pkg/bulkprocess          shared RabbitMQ naming helpers

Features

Area	Feature
API	GET/POST/PUT/DELETE /v1/bulkProcesses, list errors, meta/health/version
Worker	Supervisor poll, Redis lease (NX + renew), item and chunk listeners
Messaging	Per-key tmp queues bulk.tmp-{key}, inform queue bulk.process, binding cleanup on finish
Persistence	bulk_processes, errors, tags, notifications — same MySQL schema as legacy manager
Integrations	RabbitMQ, Redis, MySQL, HTTP target services, live notifications
Ops	Graceful shutdown, structured stdout logs, health endpoints, Helm chart, Docker Compose
Quality	Unit + E2E tests, golangci-lint, coverage reports, load-test tooling
Deferred (v2)	Schedulable services/endpoints API, cron jobs, runtime Kubernetes orchestration

Stack

Layer	Technology
Language	Go 1.22
HTTP	net/http (stdlib)
Database	MySQL (go-sql-driver/mysql)
Cache / lease	Redis (go-redis/v9)
Messaging	RabbitMQ (amqp091-go)
Packaging	Multi-stage Docker (distroless), Helm
Local dev	Docker Compose, Makefile

Quick start

cp .env.example .env
make up          # MySQL, Redis, RabbitMQ
make dev-api     # API on :8012
make dev-worker  # worker pool (second terminal)

Full stack in containers:

make up-app
make smoke

Documentation (./docs)

Document	Contents
docs/README.md	Documentation index
docs/go-architecture.md	Layers, use cases, sequences, folder map
docs/orchestration-report.md	End-to-end orchestration lifecycle
docs/api-go.md	HTTP API reference
docs/sequences.md	Mermaid sequence diagrams
docs/domain.md	Domain model and business rules
docs/integrations.md	Queues, Redis keys, HTTP contracts
docs/deployment.md	Build, env vars, Docker, Kubernetes
docs/parity-report.md	Behavioral parity notes
docs/debt.md	Known gaps and Phase 2 items
docs/adr/	Architecture Decision Records
api/openapi.yaml	OpenAPI specification