Skip to content

feat: add Docker health check and graceful shutdown with in-flight request draining #165

@BigBen-7

Description

@BigBen-7

Description

The Dockerfile has no HEALTHCHECK instruction — Docker cannot detect when the container is unhealthy and restart it. Additionally, on_shutdown only shuts down the ETL scheduler but does not wait for in-flight HTTP requests to complete, meaning a SIGTERM during an active ETL run or DB write can corrupt data mid-transaction.

Requirements & context

  • Add HEALTHCHECK CMD curl -f http://localhost:8000/health || exit 1 to the Dockerfile with --interval=30s --timeout=5s --retries=3
  • Update on_shutdown to: stop accepting new connections, wait up to 30 seconds for in-flight requests to complete, then shut down the scheduler with wait=True
  • Add a SHUTDOWN_TIMEOUT_SECONDS env var (default 30) to Settings
  • Configure Uvicorn --timeout-graceful-shutdown in run.py to match SHUTDOWN_TIMEOUT_SECONDS
  • Test the health check by starting the container and verifying docker inspect --format="{{.State.Health.Status}}" returns healthy

Suggested execution

git checkout -b feat/docker-healthcheck-graceful-shutdown
  • Update Dockerfile
  • Update src/main.py on_shutdown
  • Update run.py
  • Update src/config.py
  • Document in README.md

Guidelines

  • PR must include: Closes #[issue_id]
  • Timeframe: 48 hours

Metadata

Metadata

Assignees

Labels

Stellar WaveIssues in the Stellar wave programenhancementNew feature or improvementinfrastructureInfrastructure and DevOps

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions