[Phase 1 Issue #5] ORSA Plugin Architecture - Remediation Execution Engine

[remyluslosius]

Overview

Implement the ORSA (OpenWatch Remediation and Security Automation) Plugin Architecture to execute remediation content extracted from SCAP rules. This enables automated security hardening after compliance scans identify failed checks.

Phase 1 Progress: 4/7 tasks completed → 5/7 after this issue

Problem Statement

After scanning identifies failed compliance rules, operators need to remediate them. Currently:

• Rules have remediation field (from PR [Phase 1] Enhanced SCAP Converter with Variable and Remediation Extraction #97) with Ansible/Bash content
• No execution engine exists to apply remediations
• No tracking of remediation status or rollback capability
• No integration with scan results

Solution: ORSA Plugin Architecture

Pluggable remediation executor framework supporting multiple automation engines:

• Ansible: Execute playbooks for configuration management
• Bash: Execute shell scripts for simple fixes
• Future: Terraform, Kubernetes manifests, Python scripts

Requirements

Functional Requirements

1. Execute remediation content from ComplianceRule.remediation field
2. Support multiple executor types (Ansible, Bash, future extensibility)
3. Track remediation execution status (pending/running/completed/failed)
4. Store remediation results with stdout/stderr
5. Rollback support for failed remediations
6. Dry-run mode for preview
7. Integration with scan results

Non-Functional Requirements

• Async execution for long-running remediations
• Idempotent execution
• Secure credential handling
• Audit logging of all remediation actions
• Resource isolation (containers/sandboxes)

Architecture

Components

┌─────────────────────────────────────────────────────────────┐
│                   Remediation API                            │
│  POST /remediate/execute                                     │
│  GET  /remediate/{id}                                        │
│  POST /remediate/rollback                                    │
└─────────────────────────────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│              RemediationOrchestrator                         │
│  - Query rule remediation content                            │
│  - Route to appropriate executor                             │
│  - Track execution status                                    │
│  - Store results in MongoDB                                  │
└─────────────────────────────────────────────────────────────┘
                           │
                           ▼
         ┌─────────────────┴─────────────────┐
         │                                    │
         ▼                                    ▼
┌──────────────────┐              ┌──────────────────┐
│ AnsibleExecutor  │              │  BashExecutor    │
│ - ansible-playbook│              │  - bash -c       │
│ - Inventory mgmt │              │  - Script        │
│ - Vault decrypt  │              │    validation    │
└──────────────────┘              └──────────────────┘
         │                                    │
         └─────────────────┬─────────────────┘
                           ▼
                ┌──────────────────────┐
                │  RemediationResult   │
                │  MongoDB Document    │
                └──────────────────────┘

Implementation Tasks

1. MongoDB Models (backend/app/models/remediation_models.py)

class RemediationStatus(str, Enum):
    PENDING = "pending"
    RUNNING = "running"
    COMPLETED = "completed"
    FAILED = "failed"
    ROLLED_BACK = "rolled_back"

class RemediationTarget(BaseModel):
    type: ScanTargetType  # ssh_host, kubernetes, local
    identifier: str
    credentials: Optional[Dict[str, str]]

class RemediationResult(Document):
    remediation_id: str
    rule_id: str
    rule_title: str
    executor_type: str  # ansible, bash, terraform
    target: RemediationTarget
    status: RemediationStatus
    started_at: Optional[datetime]
    completed_at: Optional[datetime]
    dry_run: bool
    
    # Execution details
    content_executed: str
    variables_applied: Dict[str, str]
    stdout: Optional[str]
    stderr: Optional[str]
    exit_code: Optional[int]
    
    # Tracking
    executed_by: str
    rollback_available: bool
    rollback_content: Optional[str]
    
    class Settings:
        name = "remediation_results"

2. Base Executor (backend/app/services/remediators/base_executor.py)

class BaseRemediationExecutor(ABC):
    @abstractmethod
    async def execute(
        self,
        content: str,
        target: RemediationTarget,
        variables: Dict[str, str],
        dry_run: bool = False
    ) -> RemediationExecutionResult:
        pass
    
    @abstractmethod
    async def rollback(
        self,
        remediation_id: str,
        rollback_content: str,
        target: RemediationTarget
    ) -> RemediationExecutionResult:
        pass
    
    @abstractmethod
    def validate_content(self, content: str) -> bool:
        pass

3. Ansible Executor (backend/app/services/remediators/ansible_executor.py)

Features:

• Execute Ansible playbooks from string content
• Dynamic inventory generation
• Variable substitution
• Ansible Vault support for encrypted credentials
• Check mode (dry-run) support
• Idempotency via Ansible modules

4. Bash Executor (backend/app/services/remediators/bash_executor.py)

Features:

• Execute bash scripts from string content
• Variable expansion
• Script validation (syntax check)
• Timeout support
• SSH execution for remote targets
• Restricted mode (no network access in dry-run)

5. Executor Factory (backend/app/services/remediators/__init__.py)

class RemediationExecutorFactory:
    _executors = {
        'ansible': AnsibleExecutor,
        'bash': BashExecutor,
    }
    
    @classmethod
    def get_executor(cls, executor_type: str) -> BaseRemediationExecutor:
        pass

6. Remediation Orchestrator (backend/app/services/remediation_orchestrator_service.py)

class RemediationOrchestrator:
    async def execute_remediation(
        self,
        rule_id: str,
        target: RemediationTarget,
        variable_overrides: Dict[str, str] = None,
        dry_run: bool = False,
        executed_by: str = None
    ) -> RemediationResult:
        # 1. Query rule from MongoDB
        # 2. Extract remediation content
        # 3. Determine executor type
        # 4. Execute remediation
        # 5. Store result
        pass
    
    async def execute_bulk_remediation(
        self,
        scan_id: str,
        rule_filter: Dict[str, Any] = None,
        dry_run: bool = False
    ) -> List[RemediationResult]:
        # Remediate all failed rules from a scan
        pass

7. API Endpoints (backend/app/api/v1/endpoints/remediation_api.py)

@router.post("/execute", response_model=RemediationResult)
async def execute_remediation(request: RemediationRequest, ...):
    pass

@router.post("/execute-bulk", response_model=List[RemediationResult])
async def execute_bulk_remediation(request: BulkRemediationRequest, ...):
    pass

@router.get("/{remediation_id}", response_model=RemediationResult)
async def get_remediation_result(remediation_id: str, ...):
    pass

@router.post("/{remediation_id}/rollback", response_model=RemediationResult)
async def rollback_remediation(remediation_id: str, ...):
    pass

Example Usage

Execute Single Remediation

curl -X POST http://localhost:8000/api/v1/remediation/execute \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "rule_id": "xccdf_com.hanalyx.openwatch_rule_accounts_tmout",
    "target": {
      "type": "ssh_host",
      "identifier": "prod-web-01.example.com",
      "credentials": {
        "username": "root",
        "ssh_key": "..."
      }
    },
    "variable_overrides": {
      "var_accounts_tmout": "300"
    },
    "dry_run": false
  }'

Bulk Remediation (Failed Rules from Scan)

curl -X POST http://localhost:8000/api/v1/remediation/execute-bulk \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "scan_id": "550e8400-e29b-41d4-a716-446655440000",
    "rule_filter": {
      "status": "fail",
      "severity": ["high", "critical"]
    },
    "dry_run": true
  }'

Rollback Remediation

curl -X POST http://localhost:8000/api/v1/remediation/{remediation_id}/rollback \
  -H "Authorization: Bearer $TOKEN"

Security Considerations

1. Credential Encryption: Store SSH keys/passwords encrypted in MongoDB
2. Execution Isolation: Run remediations in containers/VMs when possible
3. Audit Logging: Log all remediation actions with user attribution
4. Approval Workflow: Require manual approval for production systems
5. Dry-Run First: Always test with dry_run=true before production
6. Rollback Safety: Generate rollback content before execution

Testing Strategy

Unit Tests

• Executor validation methods
• Variable substitution
• Content parsing
• Error handling

Integration Tests

• Ansible playbook execution against test VM
• Bash script execution
• Rollback functionality
• Bulk remediation

E2E Tests

• Scan → Identify Failed Rules → Remediate → Re-scan → Verify Pass

Dependencies

• ansible-core: Ansible playbook execution
• paramiko: SSH connectivity for bash executor
• Existing: motor, beanie, fastapi

Estimated Effort

5-7 days

• Models: 0.5 day
• Base executor: 0.5 day
• Ansible executor: 2 days
• Bash executor: 1 day
• Orchestrator: 1 day
• API: 0.5 day
• Testing: 1.5 days

Acceptance Criteria

• Base executor interface defined
• Ansible executor executes playbooks from string content
• Bash executor executes scripts with variable substitution
• Remediation orchestrator integrates with MongoDB rules
• API endpoints functional with auth
• Dry-run mode works correctly
• Rollback capability implemented
• Bulk remediation from scan results works
• Unit tests >80% coverage
• Integration tests pass
• Documentation updated

Related Issues

• Depends on: Enhanced SCAP Converter with Variable Extraction #96 (PR [Phase 1] Enhanced SCAP Converter with Variable and Remediation Extraction #97 - remediation field), MongoDB-Based Scan Service with Multi-Scanner Routing #100 (PR [Phase 1 Issue #4] MongoDB-Based Scan Service with Multi-Scanner Routing #101 - scan results)
• Blocks: chore(deps): bump actions/upload-artifact from 3 to 4 #6 (Scan Configuration API), chore(deps): bump peter-evans/create-pull-request from 5 to 7 #7 (Frontend UI)

References

• ComplianceAsCode remediation templates: https://github.com/ComplianceAsCode/content/tree/master/shared/templates
• Ansible Best Practices: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html
• XCCDF Remediation Specification: https://csrc.nist.gov/schema/xccdf/1.2

[remyluslosius]

Completed - ORSA Plugin Architecture and Remediation Execution Engine have been fully implemented.

Evidence:
Comprehensive remediation services exist:

• remediation_orchestrator_service.py (17,677 bytes) - Orchestration engine
• remediation_recommendation_engine.py (44,482 bytes) - Recommendation engine
• remediation_workflow_service.py (34,346 bytes) - Workflow management
• remediation_system_adapter.py (25,919 bytes) - System integration adapter
• remediation_scheduler_service.py (25,940 bytes) - Scheduling service
• post_remediation_verification_service.py (30,349 bytes) - Verification
• bulk_remediation_service.py (31,000 bytes) - Bulk operations

API Routes:

• bulk_remediation_routes.py (24,370 bytes)
• remediation_callback.py (6,267 bytes)

Related Commits:

• b2ac68e: security: Complete Phase 1 verification with comprehensive testing
• bf1a811: Phase 2: Critical security fixes
• 4ebfce7: Documentation: Complete security remediation summary

Implemented Features:

• ✅ Pluggable remediation executor framework
• ✅ Ansible & Bash automation engine support
• ✅ Remediation status tracking
• ✅ Workflow orchestration
• ✅ Post-remediation verification
• ✅ Bulk remediation operations

Phase 1 Issue #5 completed successfully.

Closed as completed.