Flow Run continues to run after cancelled

[gscholtes-relativity]

Bug summary

A Flow Run in the running state can be cancelled (via API call to Prefect) and the Flow Run will continue to run, including new Tasks which start after the cancellation time.

Similar issues:

• Prefect run runtime not stopped when run is canceled #16939 this issues mentions something about failing tasks (details unclear but in our case no failed tasks). In their case the flow run is cancelled via UI, so possibly a similar root cause?
• Job stuck in Cancelling state #16001 this is for runs stuck in the Cancelling state. In contrast, we saw a run successfully enter the Cancelled state but then continue to run.

Version info

Version:             3.1.3
API version:         0.8.4
Python version:      3.10.12
Git commit:          39b6028c
Built:               Tue, Nov 19, 2024 3:25 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         cloud
Pydantic version:    2.9.2
Integrations:
  prefect-azure:     0.4.2

Additional context

Some additional details:

• In the Prefect UI and logs, we can see a prefect.flow-run.Cancelled log at the correct time
• The Start and End time in the UI show the true start of the run and the End time is the cancellation time. Duration is based on start to cancel time.
• New tasks Start and End after the cancellation time. Visible in logs and in UI

I will provide screenshots as possible (all from the same Flow Run):

Full run time visualized:

Start and End agree with cancelled log but not above screenshot:

The logs for the final Tasks of this Flow Run, which were scheduled to start, started, and ended after the cancel time and entered a completed state:

Reproduction steps

I have not identified a reliable way to reproduce this. Any insight appreciated.

[desertaxle]

Thanks for the bug report @gscholtes-relativity! Would you be able to share the payload of the canceled event for this flow run? I hope it will show the actor claiming they canceled the flow run. Could you also provide details about how this flow run was started (i.e. was it run according to a schedule, or was it started another way)?

[137137137]

@desertaxle Hi Alex, we're running into the same issue (can reproduce, happens every time on every flow / deployment)

We are self hosting the server and starting our flows via .serve in another container. Here's what our configuration looks like:

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: python_app
    command: python -m app.main
    env_file:
      - .env
    depends_on:
      - prefect_server
    networks:
      - logging
    volumes:
      - ./app:/app/app
      - ./data:/app/data

  prefect_server:
    image: prefecthq/prefect:3-latest
    container_name: prefect_server
    user: "0"
    command: ["prefect", "server", "start", "--host=0.0.0.0"]
    ports:
      - "4200:4200" 
    volumes:
      - prefect_data:/data
    env_file:
      - .env
    environment:
      PREFECT_HOME: /data
      PREFECT_API_URL: "/api"
      PREFECT_UI_URL: "/"
      PREFECT_SERVER_API_HOST: "0.0.0.0"
    networks:
      - logging

In our python_app container, we are initializing our deployments as such:

def setup_deployments():
    from x import get_deployment as x_deployment
    from y import get_deployment as y_deployment
    from z import get_deployment as z_deployment
    q = x_deployment()
    w = y_deployment()
    e= z_deployment()
    return q, w, e

def main():
    main_logger = logger.get_child("main")
    main_logger.info("Starting application")
    if not initialize_application():
        main_logger.error("Application initialization failed")
        sys.exit(1)
    api_url = os.environ.get("PREFECT_API_URL", "http://prefect_server:4200/api")
    if not wait_for_prefect(api_url):
        main_logger.error("Prefect API at %s did not become available within the timeout period.", api_url)
        sys.exit(1)
    try:
        deployments = setup_deployments()
        main_logger.info("Starting Prefect deployments")
        serve(*deployments)
        main_logger.info("Application main execution completed successfully")
    except Exception as e:
        main_logger.error("Failed to start deployments: %s", e)
        sys.exit(1)

if __name__ == "__main__":
    main()

The flows show as cancelled in the UI, but proceed to run in the background.

In these flows, we typically have hundreds of artifacts.

[zangell44]

I haven't been able to reproduce this yet, might be missing something your setup.

@137137137 if you're still seeing this, would you be able to share your flow code and the prefect version you're running?