Describe the feature request and use-case
When using CREMA with the github-runner KEDA scaler to autoscale Cloud Run worker pools running ephemeral GitHub Actions runners, CREMA should account for in-progress jobs — not just queued jobs — when calculating the target instance count. This would prevent CREMA from scaling down worker pool instances that are actively executing jobs.
Is your feature request related to a problem? Please describe.
CREMA scales down Cloud Run worker pool instances while GitHub Actions jobs are still running on them, causing "The self-hosted runner lost communication" errors and failed workflow runs.
The root cause: the KEDA github-runner scaler only counts jobs in queued status. Once a runner picks up a job, it moves to in_progress and is no longer counted. CREMA then sees 0 queued jobs, calculates a target of 0 instances, and calls UpdateWorkerPool with manualInstanceCount: 0. Cloud Run terminates running instances to reach the target — killing jobs mid-execution.
Timeline of a typical failure:
- 3 jobs queued → CREMA scales pool to 3 instances
- 3 runners start and each pick up a job → queue is now 0
- CREMA polls, sees 0 queued → recommends 0 instances
- After stabilization window, CREMA sets manualInstanceCount: 0
- Cloud Run sends SIGTERM to running instances → jobs killed → GitHub reports "lost communication"
This is a known pattern across KEDA scalers (see kedacore/keda#6719, kedacore/keda#2901, kedacore/keda#7368), but is particularly impactful for Cloud Run worker pools because there is no graceful drain mechanism — Cloud Run gives only 10 seconds between SIGTERM and SIGKILL.
Describe the solution you'd like
CREMA should support a scaling mode aware of ephemeral job lifecycles, where scale-down only happens when instances are truly idle. Possible approaches:
-
Include in-progress jobs in the metric — When using the github-runner scaler, count both queued and in_progress jobs when calculating the target instance count. This way CREMA would maintain the instance count while jobs are still running and only scale to 0 when all jobs have completed.
-
Webhook-driven scaling — Support a workflow_job webhook receiver (in addition to polling) that tracks job state transitions (queued → in_progress → completed). Scale up on queued, maintain count while in_progress > 0, scale down only when all jobs reach completed.
-
Instance-aware scale-down — Before reducing manualInstanceCount, check whether the instances to be removed are still actively running work. Cloud Run worker pools with ephemeral runners will self-terminate when their job completes — CREMA only needs to scale up, and can let natural instance exits handle scale-down.
Describe alternatives you've considered
-
Increasing stabilizationWindowSeconds for scale-down (e.g., 3600s). This reduces the probability of killing running jobs but doesn't eliminate it, and keeps idle replacement instances running (costing money) until the window expires. This is the workaround we currently use.
-
KEDA ScaledJob instead of ScaledObject — ScaledJob is designed for ephemeral workloads and won't kill running jobs during scale-down. However, CREMA doesn't support ScaledJob; it only sets manualInstanceCount via the Cloud Run Admin API.
-
Disabling scale-down entirely in the CREMA behavior config (scaleDown.policies.value: 0). This doesn't work because Cloud Run manual scaling restarts instances to maintain the target count after ephemeral runners self-exit, leaving idle instances running indefinitely.
Additional context
- We are running ephemeral GitHub Actions runners (--ephemeral flag) on Cloud Run worker pools (8 CPU, 32Gi RAM, europe-west3)
- Runner auto-update is disabled (RUNNER_DISABLEUPDATE=true) to prevent a separate issue where update-triggered exits kill running jobs
- The SIGTERM cleanup trap (config.sh remove) has been removed because it causes TaskAgentJobStillRunningException — ephemeral runners already de-register on job completion
- Our jobs typically run 5–15 minutes (Maven test suites), with a 60-minute timeout
- CREMA version: us-central1-docker.pkg.dev/cloud-run-oss-images/crema-v1/autoscaler:1.0
- KEDA scaler: github-runner with targetWorkflowQueueLength: 1, pollingInterval: 30
- The Google Cloud tutorial states CREMA “scales to zero when all running jobs have completed” — but this is not the actual behavior when using the polling-based github-runner scaler
Describe the feature request and use-case
When using CREMA with the github-runner KEDA scaler to autoscale Cloud Run worker pools running ephemeral GitHub Actions runners, CREMA should account for in-progress jobs — not just queued jobs — when calculating the target instance count. This would prevent CREMA from scaling down worker pool instances that are actively executing jobs.
Is your feature request related to a problem? Please describe.
CREMA scales down Cloud Run worker pool instances while GitHub Actions jobs are still running on them, causing "The self-hosted runner lost communication" errors and failed workflow runs.
The root cause: the KEDA github-runner scaler only counts jobs in queued status. Once a runner picks up a job, it moves to in_progress and is no longer counted. CREMA then sees 0 queued jobs, calculates a target of 0 instances, and calls UpdateWorkerPool with manualInstanceCount: 0. Cloud Run terminates running instances to reach the target — killing jobs mid-execution.
Timeline of a typical failure:
This is a known pattern across KEDA scalers (see kedacore/keda#6719, kedacore/keda#2901, kedacore/keda#7368), but is particularly impactful for Cloud Run worker pools because there is no graceful drain mechanism — Cloud Run gives only 10 seconds between SIGTERM and SIGKILL.
Describe the solution you'd like
CREMA should support a scaling mode aware of ephemeral job lifecycles, where scale-down only happens when instances are truly idle. Possible approaches:
Include in-progress jobs in the metric — When using the github-runner scaler, count both queued and in_progress jobs when calculating the target instance count. This way CREMA would maintain the instance count while jobs are still running and only scale to 0 when all jobs have completed.
Webhook-driven scaling — Support a workflow_job webhook receiver (in addition to polling) that tracks job state transitions (queued → in_progress → completed). Scale up on queued, maintain count while in_progress > 0, scale down only when all jobs reach completed.
Instance-aware scale-down — Before reducing manualInstanceCount, check whether the instances to be removed are still actively running work. Cloud Run worker pools with ephemeral runners will self-terminate when their job completes — CREMA only needs to scale up, and can let natural instance exits handle scale-down.
Describe alternatives you've considered
Increasing stabilizationWindowSeconds for scale-down (e.g., 3600s). This reduces the probability of killing running jobs but doesn't eliminate it, and keeps idle replacement instances running (costing money) until the window expires. This is the workaround we currently use.
KEDA ScaledJob instead of ScaledObject — ScaledJob is designed for ephemeral workloads and won't kill running jobs during scale-down. However, CREMA doesn't support ScaledJob; it only sets manualInstanceCount via the Cloud Run Admin API.
Disabling scale-down entirely in the CREMA behavior config (scaleDown.policies.value: 0). This doesn't work because Cloud Run manual scaling restarts instances to maintain the target count after ephemeral runners self-exit, leaving idle instances running indefinitely.
Additional context