Add artifact persistence and recovery system to kernel runtime

[Edwardvaneechoud]

Summary

This PR implements a comprehensive artifact persistence and recovery system for the kernel runtime, enabling automatic disk-based storage of published artifacts and manual/lazy recovery mechanisms.

Key Changes

Core Models (kernel/models.py)

• Added RecoveryMode enum (lazy, eager, none) to control recovery behavior
• Added RecoveryStatus model to track recovery progress and state
• Added CleanupRequest and CleanupResult models for artifact cleanup operations
• Added ArtifactPersistenceInfo model to expose persistence configuration and statistics

KernelManager Enhancements (kernel/manager.py)

• Persistence environment variables: Modified start_kernel() to inject persistence configuration into Docker containers:
  • KERNEL_ID: Kernel identifier
  • PERSISTENCE_ENABLED: Enable/disable persistence
  • PERSISTENCE_PATH: Disk location for persisted artifacts
  • RECOVERY_MODE: Recovery strategy (lazy by default)
• New proxy methods:
  • recover_artifacts(): Trigger manual recovery from persisted storage
  • get_recovery_status(): Query current recovery state
  • cleanup_artifacts(): Remove old persisted artifacts by age or name
  • get_persistence_info(): Retrieve persistence stats and configuration

API Routes (kernel/routes.py)

• Added four new endpoints:
  • POST /{kernel_id}/recover: Manually trigger artifact recovery
  • GET /{kernel_id}/recovery-status: Check recovery progress
  • POST /{kernel_id}/cleanup: Clean up old artifacts
  • GET /{kernel_id}/persistence: Get persistence info and stats
• All endpoints include authorization checks and proper error handling

Module Exports (kernel/__init__.py)

• Exported new models and types for public API consumption

Docker Image (kernel_runtime/Dockerfile)

• Added cloudpickle>=3.0.0 dependency for artifact serialization

Testing

• Unit tests (test_artifact_persistence_integration.py):
  • Model validation and serialization
  • KernelManager proxy methods with mocked HTTP responses
  • Docker environment variable injection verification
• Integration tests (test_kernel_persistence_integration.py):
  • End-to-end persistence lifecycle (publish → persist → recover)
  • Artifact accessibility across node executions
  • Cleanup operations by age and name
  • Recovery status tracking
  • Persistence metadata in API responses

Implementation Details

• Lazy recovery: Artifacts are persisted automatically on publish; recovery loads them on-demand
• State validation: All persistence operations verify kernel is in IDLE or EXECUTING state
• HTTP timeouts: Configured appropriately per operation (120s for recovery, 60s for cleanup, 30s for status)
• Error handling: RuntimeError for invalid kernel state, HTTPException for API errors
• Backward compatible: Persistence is opt-in via environment variables; existing code unaffected

Testing Coverage

• Model instantiation and serialization
• KernelManager proxy methods with mocked responses
• Docker environment variable injection
• Full Docker-based integration tests (marked with @pytest.mark.kernel)
• Artifact persistence through node re-execution cycles

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 60.55046% with 43 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
flowfile_core/flowfile_core/kernel/routes.py	16.66%	40 Missing ⚠️
flowfile_core/flowfile_core/kernel/manager.py	91.66%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!