Implement GitHub Actions workflow to publish policies

[DakshithaS]

Description

Create a GitHub Actions workflow to automate the policy publishing process. The workflow should validate policy structure and versioning, package policy artifacts, and publish immutable policy versions to the configured artifact store, with clear logging and failure handling.

[DakshithaS]

📌 Proposed Policy Publishing Flow

Related Discussion:
🔗 [Policy Hub] Policy Contribution Flow

---

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub Repo (Policy Hub)
    participant Maint as WSO2 Maintainers
    participant WF as GitHub Workflows
    participant S3 as Artifact Storage
    participant Hub as Policy Hub

    Note over Dev: Developer creates a new policy locally

    Dev->>Dev: Implement policy + run local validation

    alt Local validation passes
        Dev->>GH: Submit changes (open PR)
    else Validation fails
        Dev->>Dev: Fix issues and retry
    end

    GH-->>Maint: Notify maintainers of new PR
    Maint->>GH: Review PR and validate quality

    alt PR approved
        Maint->>GH: Merge PR into main branch
    else PR rejected
        GH-->>Dev: PR closed without merging
    end

    Note over GH,WF: After merge, automation begins

    GH->>WF: Trigger GitHub CI workflows
    WF->>WF: Detect new policy versions
    WF->>WF: Validate merged policy version

    alt Validation succeeds
        WF->>S3: Upload policy artifact
        S3-->>WF: Artifact stored successfully

        WF->>Hub: Register policy in Policy Hub
        Hub-->>WF: Policy successfully added

        WF->>GH: ✓ Workflow successful
    else Validation fails
        WF->>GH: ✗ Workflow failed
        GH-->>Maint: Notify maintainers of validation failure
    end

Loading

[DakshithaS]

flowchart TD
    Start([Developer Creates GitHub Release]) --> Initialize[Initialize Job]

    Initialize --> CheckBaseline{Check .state/baseline.sha}
    CheckBaseline -->|Exists & Valid| UseBaseline[Use Existing Baseline]
    CheckBaseline -->|Missing/Invalid| CreateBaseline[Create from Root Commit]
    UseBaseline --> DetectPolicies[Detect Policies Job]
    CreateBaseline --> DetectPolicies

    DetectPolicies --> GitDiff[Git Diff: baseline..HEAD]
    GitDiff --> FilterPolicies[Filter: policies/*/v*/]
    FilterPolicies --> ReadDeliveryState[Read .state/delivered.json]
    ReadDeliveryState --> FilterDelivered[Filter Out Already Delivered Policies]
    FilterDelivered --> BuildMatrix[Build Matrix Config]
    BuildMatrix --> CheckCount{Any Policies Need Processing?}

    CheckCount -->|No| Summary[Summary Job]
    CheckCount -->|Yes| MatrixValidateCheck[Matrix Validate and Check Jobs]

    MatrixValidateCheck --> ForEachPolicy[For Each Policy in Parallel]
    ForEachPolicy --> CheckAPIExists[Check Policy Hub API]

    CheckAPIExists --> APICall[GET policy by name and version]
    APICall --> APIResponse{API Response}

    APIResponse -->|200 Exists| PolicyInAPI[Policy Exists in API]
    APIResponse -->|404 Not Found| PolicyNotInAPI[Policy Not in API]
    APIResponse -->|Error| APIError[API Error - Proceed]

    PolicyInAPI --> UpdateLocalState[Mark Policy as Delivered]
    UpdateLocalState --> UploadStateAPI[Upload State Artifact]
    UploadStateAPI --> SkipProcessing[Skip Validation and Publishing]

    PolicyNotInAPI --> ValidatePolicy[Validate Policy Structure]
    APIError --> ValidatePolicy
    ValidatePolicy --> ValidationResult{Validation Passed?}

    ValidationResult -->|Yes| ReadyToPublish[Ready for Publishing]
    ValidationResult -->|No| ValidationFailed[Validation Failed]

    ReadyToPublish --> MatrixPublish[Matrix Publish Jobs]
    ValidationFailed --> FinalizeJobs{All Matrix Jobs Complete?}
    SkipProcessing --> FinalizeJobs

    MatrixPublish --> ForEachPublish[For Each Policy to Publish]
    ForEachPublish --> ShouldDeliver{Should Deliver Policy?}

    ShouldDeliver -->|No| SkipPublish[Skip Publishing]
    ShouldDeliver -->|Yes| PreparePayload[Prepare API Payload]

    PreparePayload --> ReadMetadata[Read metadata.json]
    ReadMetadata --> ReadDefinition[Read policy-definition.yaml]
    ReadDefinition --> AddIdempotency[Add Release SHA as Idempotency Key]
    AddIdempotency --> CallPublishAPI[POST Policy to API]

    CallPublishAPI --> PublishResponse{API Response}

    PublishResponse -->|Success| PublishSuccess[Published Successfully]
    PublishResponse -->|Already Exists| PublishConflict[Already Exists]
    PublishResponse -->|Failure| PublishFailed[Publishing Failed]

    PublishSuccess --> UpdateDeliveryState[Update Delivery State]
    PublishConflict --> UpdateDeliveryState
    PublishFailed --> LogPublishFailure[Log Failure for Retry]

    UpdateDeliveryState --> RecordDelivery[Record Version and Timestamp]
    RecordDelivery --> UploadStatePublish[Upload State Artifact]

    UploadStatePublish --> FinalizeJobs
    LogPublishFailure --> FinalizeJobs
    SkipPublish --> FinalizeJobs

    FinalizeJobs -->|Yes| FinalizeState[Finalize State Job]
    FinalizeState --> DownloadAllArtifacts[Download State Artifacts]
    DownloadAllArtifacts --> MergeStates[Merge delivered.json Files]
    MergeStates --> WriteDeliveryState[Write Final delivered.json]
    WriteDeliveryState --> CheckAllSuccess{All Policies Successful?}

    CheckAllSuccess -->|Yes| AdvanceBaseline[Update baseline.sha]
    CheckAllSuccess -->|No| KeepBaseline[Keep baseline.sha]

    AdvanceBaseline --> CommitChanges[Commit and Push Changes]
    KeepBaseline --> CommitChanges
    CommitChanges --> Summary

    Summary --> GenerateReport[Generate Workflow Summary]
    GenerateReport --> CheckOverallResults{Overall Result}

    CheckOverallResults -->|All Success| SuccessReport[Release Successful]
    CheckOverallResults -->|Some Failed| PartialReport[Partial Success]
    CheckOverallResults -->|All Failed| FailureReport[Release Failed]

    SuccessReport --> End([Workflow Complete])
    PartialReport --> End
    FailureReport --> End

Loading

[DakshithaS]

Update - 18-12-2025

flowchart TD
    Start([Developer Creates GitHub Release]) --> Initialize[Initialize Job]

    Initialize --> CheckBaseline{Check .state/baseline.sha}
    CheckBaseline -->|Exists & Valid| UseBaseline[Use Existing Baseline]
    CheckBaseline -->|Missing/Invalid| CreateBaseline[Create from Root Commit]
    UseBaseline --> DetectPolicies[Detect Policies Job]
    CreateBaseline --> DetectPolicies

    DetectPolicies --> GitDiff[Git Diff: baseline..HEAD]
    GitDiff --> FilterPolicies[Filter: policies/*/v*/]
    FilterPolicies --> ReadDeliveryState[Read .state/delivered.json]
    ReadDeliveryState --> FilterDelivered[Filter Out Already Delivered Policies]
    FilterDelivered --> BuildMatrix[Build Matrix Config]
    BuildMatrix --> CheckCount{Any Policies Need Processing?}

    CheckCount -->|No| Summary[Summary Job]
    CheckCount -->|Yes| MatrixValidate[Matrix Validation Jobs]

    MatrixValidate --> ForEachPolicy[For Each Policy in Parallel]
    ForEachPolicy --> ValidatePolicy[Validate Policy Structure]
    ValidatePolicy --> ValidationResult{Validation Passed?}

    ValidationResult -->|No| ValidationFailed[Validation Failed]
    ValidationResult -->|Yes| ReadyToPublish[Ready for Publishing]

    ReadyToPublish --> MatrixPublish[Matrix Publish Jobs]
    ValidationFailed --> FinalizeJobs{All Matrix Jobs Complete?}

    MatrixPublish --> ForEachPublish[For Each Policy to Publish]
    ForEachPublish --> ShouldDeliver{Should Deliver Policy?}

    ShouldDeliver -->|No| SkipPublish[Skip Publishing]
    ShouldDeliver -->|Yes| PrepareMultipart[Prepare Multipart Request]

    PrepareMultipart --> ReadMetadata[Read metadata.json]
    ReadMetadata --> ReadDefinition[Read policy-definition.yaml]
    ReadDefinition --> ReadZip[Read Policy ZIP File]
    ReadZip --> BuildMultipart[Build multipart/form-data payload]
    BuildMultipart --> AddIdempotency[Add Release SHA as Idempotency Key]
    AddIdempotency --> CallPublishAPI[POST Policy Publish API]

    CallPublishAPI --> PublishResponse{API Response}

    PublishResponse -->|200/201 Success| PublishSuccess[Published Successfully]
    PublishResponse -->|409 Conflict| PublishExists[Policy Already Exists]
    PublishResponse -->|Failure| PublishFailed[Publishing Failed]

    PublishSuccess --> UpdateDeliveryState[Update Delivery State]
    PublishExists --> UpdateDeliveryState
    PublishFailed --> LogPublishFailure[Log Failure for Retry]

    UpdateDeliveryState --> RecordDelivery[Record Version and Timestamp]
    RecordDelivery --> UploadStatePublish[Upload State Artifact]

    UploadStatePublish --> FinalizeJobs
    LogPublishFailure --> FinalizeJobs
    SkipPublish --> FinalizeJobs

    FinalizeJobs -->|Yes| FinalizeState[Finalize State Job]
    FinalizeState --> DownloadAllArtifacts[Download State Artifacts]
    DownloadAllArtifacts --> MergeStates[Merge delivered.json Files]
    MergeStates --> WriteDeliveryState[Write Final delivered.json]
    WriteDeliveryState --> CheckAllSuccess{All Policies Successful?}

    CheckAllSuccess -->|Yes| AdvanceBaseline[Update baseline.sha]
    CheckAllSuccess -->|No| KeepBaseline[Keep baseline.sha]

    AdvanceBaseline --> CommitChanges[Commit and Push Changes]
    KeepBaseline --> CommitChanges
    CommitChanges --> Summary

    Summary --> GenerateReport[Generate Workflow Summary]
    GenerateReport --> CheckOverallResults{Overall Result}

    CheckOverallResults -->|All Success| SuccessReport[Release Successful]
    CheckOverallResults -->|Some Failed| PartialReport[Partial Success]
    CheckOverallResults -->|All Failed| FailureReport[Release Failed]

    SuccessReport --> End([Workflow Complete])
    PartialReport --> End
    FailureReport --> End

Loading

[DakshithaS]

⚠️ Note : The above diagrams are outdated, as the policy repository structure and the policy resolution flow have been completely changed.