refactor(core): operation cleanup

[michael-johnston]

Operation cleanup logic was mixed together (a single function for cleaning up everything was required/was different for different scenarios) and also mixed with signal handling (a shutdown could be successful operation or a signal).

This made it impossible to correctly handle operation cleanup when there were nested operations (see #200)

In this PR

• cleanup code required by different operations/functions is separated
• A single signal handler exists that is generic
  • operations register their cleanup requirements with the handler so they are cleaned-up
• shutdown global now only indicates if a signal was received NOT if an operation finished

New behaviour:

• When an explore operation finishes it only cleans up the ray actors it has created. It does not
  - shutdown ray
  - remove global resource like the cleaner
  - set shutdown global (which is gone)
• When orchestrate() (called via ado create operation)) exits it cleans up ResourceCleaner and ray
• When general operation finishes there is no set cleanup as there are no ray resources created. It does not:
  - shutdown ray
  - remove global resource like the cleaner
  - set shutdown global (which is gone)
• If operations are nested there should no longer be issues with shutdown being set or resources deleted
• If SIGTERM is received all active operations will be cleaned up
• If SIGINT is received during an operation an InterruptedOperationError is raised with the operation id
  - previously nothing was raised and the operation exited via the normal route

Note: If an operation is called directly that creates Ray actors then the caller is responsible for cleaning up the ResourceCleaner and shutting down ray i.e. the same responsibilities orchestrate has.

[DRL-NextGen]

Checks Summary

Last run: 2025-12-05T13:40:35.163Z

Code Risk Analyzer vulnerability scan found 2 vulnerabilities:

Severity	Identifier	Package	Details	Fix
🔷 Medium	CVE-2025-50181	urllib3	urllib3 redirects are not disabled when retries are disabled on PoolManager instantiation GHSA-pq67-6m6q-mj2v urllib3:2.3.0->kubernetes:34.1.0	2.5.0
🔷 Medium	CVE-2025-50182	urllib3	urllib3 does not control redirects in browsers and Node.js GHSA-48p4-8xcf-vxj5 urllib3:2.3.0->kubernetes:34.1.0	2.5.0

[AlessandroPomponio]

Just naming suggestions

[michael-johnston]

@danielelotito can you try Trim with this branch?

The main things to check are

1. That when Trim finishes "Successfully" both it and the child randomwalk have status SUCCESS (in ado get operations) and that ado show related for the Trim operation shows the randomwalk and vice versa
   • c.f. bug: nested operations cause the outer operation to think a signal has been sent #200 randomwalk would be success and Trim would be failed and Trim would not have randomwalk as a related op (IIRC)
2. That if you run Trim and interrupt the randomwalk (CTRL-C) that everything shutdowns gracefully - both operations are marked FAILED but both still are shown as related

[danielelotito]

Will do!

[michael-johnston]

Also need to check with vllm_performance that resource cleanup is working.