CancellationCleanup service stuck permanently at 100% CPU usage

[gabr1elt]

Bug summary

CancellationCleanup constantly keeps the CPU at 90+% (the service still works tough, and with no delays)

Stopping the service as described in settings ref frees up the cpu (so I assume its that).

logs

The server only ever showed a warning once:

19:21:17.691 | WARNING | prefect.server.services.cancellationcleanup - stop(block=True)was called on CancellationCleanup but more than one loop interval (20.0 seconds) has passed. This usually means something is wrong. Ifstop()was called from inside the loop service, usestop(block=False) instead.

then it never mentioned it again

how to stop it

Resetting the DB is the only way I found to fix it (but obviously not ideal)

Maybe related

My infra crashed a few times and had to do some manual cleanup

also found this

#15231

Version info

Prefect Server: docker.io/prefecthq/prefect:3.4.0-python3.10
Postgres: docker.io/postgres:17.4

Additional context

No response

[zzstoatzz]

hi @gabr1elt - thanks for the issue! can you share whatever details you can on your setup (infra, database settings etc) and load (roughly how much work, task-heavy, what infra, how many workers etc)? that will help pinpoint what the actual issue is

[gabr1elt]

Hi @zzstoatzz

It's a small deployment, less than 100 flows an hour not IO or CPU intensive.
I'm running it on a Podman Pod for now (untill I solve all the issues)
1 Worker
The runners are docker containers (actually podman but should not be any different)

This started since I moved to 3.4.0 (from 3.2.13)

The database has no particular conf changes from the base image

pg_config --version

PostgreSQL 17.4 (Debian 17.4-1.pgdg120+2)

podman version

Version:             3.4.0
API version:         0.8.4
Python version:      3.10.17
Git commit:          c80e4442
Built:               Fri, May 02, 2025 08:04 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         ephemeral
Pydantic version:    2.11.4
Server:
  Database:          postgresql
Integrations:
  prefect-redis:     0.2.2

Server conf

# prefect.toml

[server]
logging_level = "INFO"
analytics_enabled = false

[server.api]
host = "0.0.0.0"
port = 4200
csrf_protection_enabled = true
cors_allowed_origins = "http://127.0.0.1:4200"

[server.ui]
api_url = "http://127.0.0.2:4200/api"

[server.database]
connection_url = "postgresql+asyncpg://prefect:${PREFECT_SERVER_DATABASE_PASSWORD}@127.0.0.1:5432/prefect"

[server.deployments]
concurrency_slot_wait_seconds = 10

Worker conf

# prefect.toml

[api]
url = "http://127.0.1.1:4200/api"

[worker.webserver]
host = "0.0.0.0"
port = 4300

[runner]
heartbeat_frequency = 30

anything else, let me know

[zzstoatzz]

excellent, thanks!