Workflow missing on Run.enqueue after snaphots

[jyeshe]

Workflow is missing on Multi passed to Run.enqueue :

[error] GenServer #PID<0.1433.0> terminating
** (FunctionClauseError) no function clause matching in anonymous fn/1 in increment_runs/1
    (0.2.12) anonymous fn(%{run: %Lightning.Run{__meta__: #Ecto.Schema.Metadata<:loaded, "runs">, id: "16a9ebd1-1188-4dcf-8ec7-b106c3506d23", work_order_id: "872ee83d-76f7-48ab-8eb3-c6cd8be3ea40", work_order: %Lightning.WorkOrder{__meta__: #Ecto.Schema.Metadata<:loaded, "work_orders">, id: "872ee83d-76f7-48ab-8eb3-c6cd8be3ea40", state: :success, last_activity: ~U[2024-04-25 15:32:42.575391Z], workflow_id: "c079f878-1b68-4450-8f2e-5c79bc5fb21c", workflow: #Ecto.Association.NotLoaded<association :workflow is not loaded>, snapshot_id: "6bb65f70-48fd-4eb9-a842-66c40e22770d", snapshot: #Ecto.Association.NotLoaded<association :snapshot is not loaded>, trigger_id: "cae544ab-03dc-4ccc-a09c-fb4edb255d7a", trigger: #Ecto.Association.NotLoaded<association :trigger is not loaded>, dataclip_id: "7a10a2fc-b73b-47f8-a228-7802ea5bf817", dataclip: #Ecto.Association.NotLoaded<association :dataclip is not loaded>, runs: #Ecto.Association.NotLoaded<association :runs is not loaded>, jobs: #Ecto.Association.NotLoaded<association :jobs is not loaded>, inserted_at: ~U[2024-04-25 15:32:42.338195Z], updated_at: ~U[2024-04-25 15:32:42.338195Z]}, starting_job_id: "9c09787b-f04b-43b8-b11a-2fe67d7052d2", starting_job: %Lightning.Workflows.Job{__meta__: #Ecto.Schema.Metadata<:loaded, "jobs">, id: "9c09787b-f04b-43b8-b11a-2fe67d7052d2", body: "fn(state => state);\n", name: "Transform data to FHIR standard", adaptor: "@openfn/language-http@latest", project_credential_id: nil, project_credential: #Ecto.Association.NotLoaded<association :project_credential is not loaded>, credential: #Ecto.Association.NotLoaded<association :credential is not loaded>, workflow_id: "c079f878-1b68-4450-8f2e-5c79bc5fb21c", workflow: #Ecto.Association.NotLoaded<association :workflow is not loaded>, project: #Ecto.Association.NotLoaded<association :project is not loaded>, steps: #Ecto.Association.NotLoaded<association :steps is not loaded>, delete: nil, inserted_at: ~U[2024-04-25 15:32:42.291453Z], updated_at: ~U[2024-04-25 15:32:42.295026Z]}, starting_trigger_id: nil, starting_trigger: #Ecto.Association.NotLoaded<association :starting_trigger is not loaded>, created_by_id: "73859d6d-3831-4582-bff4-af60beb5fd71", created_by: #Lightning.Accounts.User<__meta__: #Ecto.Schema.Metadata<:loaded, "users">, id: "73859d6d-3831-4582-bff4-af60beb5fd71", first_name: "Amy", last_name: "Admin", email: "demo@openfn.org", confirmed_at: nil, role: :user, disabled: false, mfa_enabled: false, scheduled_deletion: nil, user_totp: #Ecto.Association.NotLoaded<association :user_totp is not loaded>, credentials: #Ecto.Association.NotLoaded<association :credentials is not loaded>, project_users: #Ecto.Association.NotLoaded<association :project_users is not loaded>, projects: #Ecto.Association.NotLoaded<association :projects is not loaded>, backup_codes: #Ecto.Association.NotLoaded<association :backup_codes is not loaded>, inserted_at: ~N[2024-04-25 15:32:41], updated_at: ~N[2024-04-25 15:32:41], ...>, dataclip_id: "7a10a2fc-b73b-47f8-a228-7802ea5bf817", dataclip: %Lightning.Invocation.Dataclip{__meta__: #Ecto.Schema.Metadata<:loaded, "dataclips">, id: "7a10a2fc-b73b-47f8-a228-7802ea5bf817", body: nil, request: nil, type: :http_request, wiped_at: nil, project_id: "4adf2644-ed4e-4f97-a24c-ab35b3cb1efa", project: #Ecto.Association.NotLoaded<association :project is not loaded>, source_step: #Ecto.Association.NotLoaded<association :source_step is not loaded>, inserted_at: ~U[2024-04-25 15:32:42.326085Z], updated_at: ~U[2024-04-25 15:32:42.326085Z]}, workflow: #Ecto.Association.NotLoaded<association :workflow is not loaded>, snapshot_id: "6bb65f70-48fd-4eb9-a842-66c40e22770d", snapshot: %Lightning.Workflows.Snapshot{__meta__: #Ecto.Schema.Metadata<:loaded, "workflow_snapshots">, id: "6bb65f70-48fd-4eb9-a842-66c40e22770d", workflow_id: "c079f878-1b68-4450-8f2e-5c79bc5fb21c", workflow: #Ecto.Association.NotLoaded<association :workflow is not loaded>, name: "OpenHIE Workflow", lock_version: 9, jobs: [%Lightning.Workflows.Snapshot.Job{id: "9c09787b-f04b-43b8-b11a-2fe67d7052d2", name: "Transform data to FHIR standard", body: "fn(state => state);\n", adaptor: "@openfn/language-http@latest", project_credential: #Ecto.Association.NotLoaded<association :project_credential is not loaded>, project_credential_id: nil, credential: #Ecto.Association.NotLoaded<association :credential is not loaded>, inserted_at: ~U[2024-04-25 15:32:42.291453Z], updated_at: ~U[2024-04-25 15:32:42.295026Z]}, %Lightning.Workflows.Snapshot.Job{id: "66359683-2fb7-4255-9986-430215ce7932", name: "Send to OpenHIM to route to SHR", body: "fn(state => state);\n", adaptor: "@openfn/language-http@latest", project_credential: #Ecto.Association.NotLoaded<association :project_credential is not loaded>, project_credential_id: nil, credential: #Ecto.Association.NotLoaded<association :credential is not loaded>, inserted_at: ~U[2024-04-25 15:32:43.305868Z], updated_at: ~U[2024-04-25 15:32:42.305921Z]}, %Lightning.Workflows.Snapshot.Job{id: "e44030dd-0c23-43ae-b9fa-588edf571c83", name: "Notify CHW upload successful", body: "fn(state => state);\n", adaptor: "@openfn/language-http@latest", project_credential: #Ecto.Association.NotLoaded<association :project_credential is not loaded>, project_credential_id: nil, credential: #Ecto.Association.NotLoaded<association :credential is not loaded>, inserted_at: ~U[2024-04-25 15:32:44.308243Z], updated_at: ~U[2024-04-25 15:32:42.308284Z]}, %Lightning.Workflows.Snapshot.Job{id: "bd1bf006-6053-49c9-80a1-160ec6ccbe73", name: "Notify CHW upload failed", body: "fn(state => state);\n", adaptor: "@openfn/language-http@latest", project_credential: #Ecto.Association.NotLoaded<association :project_credential is not loaded>, project_credential_id: nil, credential: #Ecto.Association.NotLoaded<association :credential is not loaded>, inserted_at: ~U[2024-04-25 15:32:45.310689Z], updated_at: ~U[2024-04-25 15:32:42.310732Z]}], triggers: [%Lightning.Workflows.Snapshot.Trigger{id: "cae544ab-03dc-4ccc-a09c-fb4edb255d7a", comment: nil, custom_path: nil, cron_expression: nil, enabled: true, type: :webhook, inserted_at: ~U[2024-04-25 15:32:42Z], updated_at: ~U[2024-04-25 15:32:42Z]}], edges: [%Lightning.Workflows.Snapshot.Edge{id: "da07fbe5-f7b8-44d5-bfea-0b5b42846ebc", source_job_id: nil, source_trigger_id: "cae544ab-03dc-4ccc-a09c-fb4edb255d7a", target_job_id: "9c09787b-f04b-43b8-b11a-2fe67d7052d2", condition_type: :always, condition_expression: nil, condition_label: nil, enabled: true, inserted_at: ~U[2024-04-25 15:32:42Z], updated_at: ~U[2024-04-25 15:32:42Z]}, %Lightning.Workflows.Snapshot.Edge{id: "43550739-eb59-4e16-87d4-38310c515aab", source_job_id: "9c09787b-f04b-43b8-b11a-2fe67d7052d2", source_trigger_id: nil, target_job_id: "66359683-2fb7-4255-9986-430215ce7932", condition_type: :on_job_success, condition_expression: nil, condition_label: nil, enabled: true, inserted_at: ~U[2024-04-25 15:32:42Z], updated_at: ~U[2024-04-25 15:32:42Z]}, %Lightning.Workflows.Snapshot.Edge{id: "4851b688-7d27-4920-9071-17f05442c2ce", source_job_id: "66359683-2fb7-4255-9986-430215ce7932", source_trigger_id: nil, target_job_id: "e44030dd-0c23-43ae-b9fa-588edf571c83", condition_type: :on_job_success, condition_expression: nil, condition_label: nil, enabled: true, inserted_at: ~U[2024-04-25 15:32:42Z], updated_at: ~U[2024-04-25 15:32:42Z]}, %Lightning.Workflows.Snapshot.Edge{id: "fcea5fbb-7b52-4fde-bd0f-ad9fc1497fed", source_job_id: "66359683-2fb7-4255-9986-430215ce7932", source_trigger_id: nil, target_job_id: "bd1bf006-6053-49c9-80a1-160ec6ccbe73", condition_type: :on_job_failure, condition_expression: nil, condition_label: nil, enabled: true, inserted_at: ~U[2024-04-25 15:32:42Z], updated_at: ~U[2024-04-25 15:32:42Z]}], inserted_at: ~U[2024-04-25 15:32:42.320896Z]}, log_lines: #Ecto.Association.NotLoaded<association :log_lines is not loaded>, steps: [], state: :available, error_type: nil, claimed_at: nil, star (truncated)

Version number

v2.4.2

I have reproduced this locally on main:

• Yes
• No

To Reproduce

1. Retry from job inspector or history page and inspect (test/lightning/runs_test.exs is useful as well)