Flow run limit reached for worker even when nothing is "Running" and flows get accumulated in "Late"

[steinskeeper]

Bug summary

Problem

Worker shows Flow run limit reached; 15 flow runs in progress even when no flow is running and all the flows gets accumulated in Late.

What fixes it temporarily

Only upon closing the worker and starting again do flows from Late start running.

When is this behaviour triggred

Not sure, but haven't had this issue before. Seems recent.

Additional details

• Worker is started using prefect worker start -p "my-pool" -n "dannylinux" --limit 15
• No concurrency limits are set anywhere else.

Version info

prefect version
Version:             3.2.13
API version:         0.8.4
Python version:      3.13.2
Git commit:          12800297
Built:               Fri, Mar 14, 2025 8:37 PM
OS/Arch:             linux/x86_64
Profile:             local
Server type:         cloud
Pydantic version:    2.10.6

Additional context

Not sure if it is relevant, but this is one error before the continuous loop of Flow run limit reached

 /usr/lib/python3.13/asyncio/selector_events.py:869: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=6 read=idle write=<idle, bufsize=0>>
0|run      |   _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
0|run      | ResourceWarning: Enable tracemalloc to get the object allocation traceback

[lelouvincx]

Can you show the database log?

[steinskeeper]

Can you show the database log?

@lelouvincx where do I find this?

[lelouvincx]

Can you show the database log?

@lelouvincx where do I find this?

First set the environment variable PREFECT_API_DATABASE_ECHO to true

PREFECT_API_DATABASE_ECHO='True'

Then reproduce your steps and capture the log on Prefect server.

[steinskeeper]

These are the logs that go on a loop which I get with logging set to DEBUG and Database Echo set to True

prefect.runner - Checking for cancelled flow runs...
prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
prefect.workers.process.dannylinux - Querying for flow runs scheduled before 2025-03-28 18:17:51.158559+00:00

prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-c3596ab5-b6e5-45a9-8d20-42aafb60e8a5')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x775cd5aeb240>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x775cd095a990>>)
prefect.runner - Checking for cancelled flow runs... ( MULTIPLE TIMES )

prefect.workers.process.dannylinux - Discovered 200 scheduled_flow_runs
prefect.workers.process.dannylinux - Flow run limit reached; 15 flow runs in progress.

AGAIN
prefect.utilities.services.critical_service_loop ( MULTIPLE TIMES )
prefect.runner - Checking for cancelled flow runs...
prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend'

[wangxing-git]

I have encountered a similar issue before. The possible cause might be that some resources, such as thread pools, were not properly cleaned up. As a result, the process running the flow did not exit and was still waiting, even though the flow run status was marked as completed. You might want to check whether you are cleaning up these resources when the flow finishes.

[willhcr]

I am also seeing this issue (with a limit of 8). This is new since upgrading to 3.x.

Flow run limit reached; 8 flow runs in progress.

But I have confirmed there are only 2 prefect processes running in ps.

Version Info

Version:             3.3.5
API version:         0.8.4
Python version:      3.12.10
Git commit:          db4b7a33
Built:               Thu, Apr 17, 2025 09:25 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.10.6
Integrations:
  prefect-azure:     0.4.3
  prefect-docker:    0.6.3

[austinweisgrau]

We are seeing a similar issue when a work queue has a concurrency limit set on it in Prefect Cloud within a ECS push pool.