OCTRL-966: Kill tasks from destroyed environments during reconciliation

[Copilot]

Problem

When a mesos-slave reappears after network connectivity issues, it may advertise tasks that belong to environments which were destroyed while the slave was offline. The existing reconciliation logic only killed tasks that were not in the roster, but tasks from destroyed environments remained running even though their environments no longer existed.

This led to:

• Resource waste from orphaned tasks
• Potential conflicts when redeploying environments
• Inconsistent system state

Solution

Enhanced the reconciliation logic in the task manager to detect and terminate orphaned tasks during mesos agent reconnection:

Key Changes

1. Environment Existence Validation: Added EnvironmentExistsFunc type and integration allowing the task manager to check if environments still exist during reconciliation.

2. Enhanced Reconciliation Logic: Modified handleMessage() to check both:

   • Tasks not in roster (existing behavior)
   • Tasks in roster but from destroyed environments (new behavior)

3. Automatic Cleanup: Orphaned tasks are automatically killed via mesos and removed from the roster.

4. Comprehensive Logging: Added detailed logging at operator level (IL_Ops) for visibility when orphaned tasks are detected and terminated.

Implementation Details

The solution maintains full backward compatibility by:

• Preserving existing reconciliation behavior for tasks not in roster
• Gracefully handling cases where environment checker is not available
• Using dependency injection to avoid tight coupling between managers

Code Structure

• core/task/manager.go: Enhanced reconciliation logic with environment validation
• core/globalstate.go: Integration of environment existence checker

Example Scenario

1. Environment A is running with tasks on agents 1, 2, 3
2. Agent 2 goes offline due to network issues
3. Operator destroys Environment A (tasks on agents 1, 3 are killed)
4. Agent 2 comes back online and reports its tasks as still running
5. NEW: Reconciliation detects these tasks belong to destroyed Environment A
6. NEW: Tasks are automatically killed and removed from roster

Testing

• All existing tests continue to pass
• Full project builds successfully
• Backward compatibility maintained
• No regression in existing reconciliation behavior

This implementation successfully addresses OCTRL-966 by ensuring that tasks from destroyed environments are automatically cleaned up during mesos agent reconciliation, maintaining system consistency and preventing resource waste.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.