ReferenceIds on StepFramework steps are incorrectly updated when updating step versions, this leads to a mismatch between the referenceId and StepPackageInputsReferenceId causing deployment failures

[IsaacCalligeros95]

Severity

Sev 1

Version

2023.*

Latest Version

I could reproduce the problem in the latest build

What happened?

When upgrading a Step Framework step inputs that set a referenceId are being updated to a new v4 while the StepPackageInputsReferenceId is not. At deployment time this causes failures in retrieving package information.

Reproduction

Create a deployment process where the referenceId and StepPackageInputsReferenceId mismatch, in this case, I've updated the deployment process manually using SQL, in Octopus this happens when the step package is upgraded. Note the referenceId end in vv

Update DeploymentProcess Set JSON = '{ "Steps": [ { "Id": "6cd44672-9b61-48c6-9170-04bdc534c616", "Name": "Kustomize", "Slug": "kustomize", "Condition": "Success", "StartTrigger": "StartAfterPrevious", "PackageRequirement": "AfterPackageAcquisition", "Actions": [ { "Id": "d72f5a5d-b204-4214-b057-2021f6ffbebf", "Name": "Kustomize", "Slug": "kustomize", "ActionType": "kustomize", "Notes": null, "WorkerPoolId": "WorkerPools-42", "WorkerPoolVariable": null, "Container": { "Image": "octopuslabs/k8s-workertools", "FeedId": "Feeds-1021" }, "IsDisabled": false, "IsRequired": false, "Environments": [], "ExcludedEnvironments": [], "Channels": [], "TenantTags": [], "Packages": [], "GitDependencies": [ { "Name": "", "RepositoryUri": "https://github.com/OctopusSamples/OctoPetShop.git", "DefaultBranch": "master", "FilePathFilters": [], "GitCredentialType": "Library", "GitCredentialId": "GitCredentials-1", "StepPackageInputsReferenceId": "de2d5028-47de-42b4-bef2-c3b451108854" } ], "Condition": "Success", "Properties": { "Octopus.Action.RunOnServer": "true", "OctopusUseBundledTooling": "False" }, "StepPackageVersion": "2.0.2", "Inputs": { "overlayPath": "k8s/kustomize/overlays/#{Octopus.Environment.Name}", "substituteFiles": { "filesToTarget": "k8s/kustomize/base/kustomization.yaml" }, "kubernetesObjectStatus": { "timeout": 180, "waitForJobs": false, "enabled": true }, "gitSource": { "isAnonymous": false, "gitCredential": "GitCredentials-1", "url": "https://github.com/OctopusSamples/OctoPetShop.git", "defaultBranch": "master", "referenceId": "de2d5028-47de-42b4-bef2-c3b4511088vv" }, "containerImageReferences": [] } } ], "Properties": { "Octopus.Action.TargetRoles": "wsltarget" } } ] }' where Id = 'deploymentprocess-Projects-XXX'

Error and Stacktrace

|       System.NullReferenceException
                    |       at Octopus.Core.StepPackages.PackageInformationRetriever.GetValue(String referenceId, String property) in PackageInformationRetriever.cs:line 24
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.StepPackages.ExtractPackageInstructionContributor.Contribute(JToken packageReference, String referenceId, InstructionContributorContext context)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.StepPackages.PackageManipulationInstructionCoordinator.<>c__DisplayClass3_1.<Generate>b__1(InstructionContributor x)
                    |       at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
                    |       at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
                    |       at System.Collections.Generic.List`1.AddRange(IEnumerable`1 collection)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.StepPackages.PackageManipulationInstructionCoordinator.Generate(String id, String version, InstructionContributorContext context)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.StepPackages.StepPackageStepActionHandler.Execute(IActionHandlerContext context, ITaskLog taskLog, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.ExecuteAction(ActionCommand command, DeploymentTarget inContextOf, TargetManifest targetManifest, ITaskLog taskLog, IActionHandler handler, IActionHandlerContext handlerContext, Nullable`1 executionTimeout, CancellationToken ct)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.<>c__DisplayClass20_0.<Execute
                    |       at Octopus.Server.Orchestration.ServerTasks.ActionRetry.Execute(Func`2 callback, ITaskLog taskLog, TargetManifest targetManifest, IActionAndTargetScopedVariables variablesSnapshot, PlannedAction action, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.ActionRetry.Execute(Func`2 callback, ITaskLog taskLog, TargetManifest targetManifest, IActionAndTargetScopedVariables variablesSnapshot, PlannedAction action, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.<>c__DisplayClass20_0.<Execute
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.Execute(ActionCommand command, DeploymentTarget inContextOf, IExecutor executor, TargetManifest targetManifest, Maybe`1 stagedPackagePath, IEnumerable`1 packageAcquisitionInformation, IActionAndTargetScopedVariables variablesSnapshot, ITaskLog taskLog, IActionHandler handler, Action guidanceExcludeCallback, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.ExecuteOnWorker(ActionCommand command, DeploymentTarget inContextOf, ITaskLog taskLog, IActionHandler handler, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.ExecuteOnWorker(ActionCommand command, DeploymentTarget inContextOf, ITaskLog taskLog, IActionHandler handler, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.Dispatch(ActionCommand command, DeploymentTarget deploymentTarget, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, IActionHandlerResolver actionHandlerResolver, Action guidanceExcludeCallback, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_3.<TryExecuteActionAndInitLoggingContext
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_2.<TryExecuteActionAndInitLoggingContext
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_1.<TryExecuteActionAndInitLoggingContext
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass7_0`4.<Execute
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 50
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 56
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name)
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.TryExecuteActionAndInitLoggingContext(ExecutionPlan plan, ExecutionPlanner planner, PlannedStep step, DeploymentTarget targetContext, PlannedAction action, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, CancellationToken cancellationToken)
17:23:45   Fatal    |       The action Kustomize on WSL target failed

### More Information

This impacts the following steps:
* Deploy/Update Amazon ECS Service
* Kustomize
* Deploy a Bicep Template

### Workaround

Updating these steps in the UI and saving them will update the referenceId and fix this issue. 

* For the Kustomize step, switch between Library and Anonymous, and back, then save.
* For ECS, delete the container references, recreate them, and save.
* For Bicep Templates, switch from the package selector to source code, then back to the package selector, then save.

[IsaacCalligeros95]

The fix to this issue will still require the deployment process to be updated manually to re-sync the referenceIds.

[Octobob]

🎉 The fix for this issue has been released in:

Release stream	Release
2023.2	2023.2.13440
2023.3	2023.3.13155
2023.4	2023.4.7996
2024.1	2024.1.357
2024.2+	all releases