Download docker container image failing with Azure App Service step template - docker command not available error

[Clare-Octopus]

Severity

Blocking anyone that wants to deploy a docker container using a Azure App Service step template

Version

2023.3.13094

Latest Version

I could reproduce the problem in the latest build

What happened?

Creating a project to deploy a docker container using a Deploy an Azure App Service step and selecting the default worker pool (which uses the Octopus Server) is successful and pulls the docker image fine:

Task log of a working step -

ServerTasks-12644.log.txt

However, when using the exact same settings in a step template, the step fails with -

docker command not available
NotSpecified: You will need docker installed and running to pull docker images

Task log of a non working step using a step template with the same settings -

ServerTasks-12642.log.txt

It looks like, in a step, the package is not acquired, whereas using a step template, the package is acquired. I tested using a Linux worker with docker installed and it errors out with the same error.

Reproduction

1. Ensure you have a docker feed in Octopus with docker images in.
2. Create a deployment target to deploy a container on (I used an Azure Web App).
3. Create a project with a Deploy An Azure App Service Step.
4. Configure the step to use the default worker pool.
5. Configure the step to use a container package and pull it from the feed we had in Step 1.
6. Create a release and deploy it to your Azure Web App, see it deploys successfully.
7. Create a step template using the Deploy An Azure App Service Step and use the same docker feed as in Step 1.
8. Attach the step template to the project and configure it to use the default worker pool.
9. Disable the Deploy An Azure App Service Step we created in Step 3 to ensure you only have the step template step enabled.
10. Create a release and deploy it. Notice that the deployment will fail with the docker error at the Acquire Packages stage.

Error and Stacktrace

Task ID:        ServerTasks-12642
Related IDs:    Deployments-2373, Channels-1541, Releases-2292, Projects-1481, Spaces-1, Environments-2
Task status:    Failed
Task queued:    Friday, 02 February 2024 10:58:37 AM +00:00
Task started:   Friday, 02 February 2024 10:58:37 AM +00:00
Task completed: Friday, 02 February 2024 10:58:40 AM +00:00
Task duration:  3 seconds
Server version: 2023.4.8279
Server node:    OCTODC

                    | == Failed: Deploy AzureAppServiceStepTemplate release 0.0.9 to Test ==
10:58:37   Info     |   Step 1: Deploy an Azure App Service is disabled, and will not be executed
10:58:37   Verbose  |   Could not discover Azure Web Application targets for 'DeployAppServiceStepTemplate' due to the following reasons:
10:58:37   Verbose  |   Azure account for discovery not configured. Please configure an Azure account in variable 'Octopus.Azure.Account'.
10:58:37   Verbose  |   Account type of '$AzureOidc' does not support target discovery
10:58:37   Verbose  |   Initiated by: octoadmin
10:58:37   Verbose  |   Guided failure is not enabled for this task
10:58:40   Fatal    |   The deployment failed because one or more steps failed. Please see the deployment log for details.
                    | 
                    |   == Failed: Acquire packages ==
10:58:37   Info     |     Acquiring packages
10:58:37   Info     |     Making a list of packages to acquire
10:58:37   Verbose  |     Package octopussamples/randomquotes v1.0.31 is required by action 'DeployAppServiceStepTemplate'
10:58:37   Verbose  |     Task Acquire octopussamples/randomquotes.1.0.31 on Octopus Server created to acquire octopussamples/randomquotes v1.0.31 for action ('DeployAppServiceStepTemplate')
10:58:37   Verbose  |     No packages are required on any deployment targets
10:58:40   Fatal    |     The step failed: Activity failed with error 'The remote script failed with exit code 1'.
10:58:40   Verbose  |     Acquire Packages completed
                    |   
                    |     Failed: Octopus Server
10:58:40   Fatal    |       The remote script failed with exit code 1
10:58:40   Verbose  |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessful(IActionHandlerResult result) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/SuccessArbitrator.cs:line 12
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 80
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.BuiltInWorkerPackageDownloader.Download(PackageIdentity packageIdentity, IPackageFeed packageFeed, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/BuiltInWorkerPackageDownloader.cs:line 22
                    |       at Octopus.Core.Packages.DeploymentPackageDownloader.<>c__DisplayClass11_0.<<DownloadDependency>b__0>d.MoveNext() in ./source/Octopus.Core/Packages/DeploymentPackageDownloader.cs:line 61
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Core.Packages.DeploymentPackageDownloader.PackageMemento.GetOrAdd(PackageIdentity identifier, Func`1 factory, CancellationToken cancellationToken) in ./source/Octopus.Core/Packages/DeploymentPackageDownloader.cs:line 135
                    |       at Octopus.Core.Packages.DeploymentPackageDownloader.DownloadDependency(ITaskLog log, DependencyIdentity identity, CancellationToken cancellationToken) in ./source/Octopus.Core/Packages/DeploymentPackageDownloader.cs:line 51
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireServerPackageTask.Acquire(AcquiredPackageMap acquiredPackageMap, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireServerPackageTask.cs:line 52
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_3`3.<<Execute>b__5>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 78
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 149
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 79
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_3`3.<<Execute>b__4>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 77
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 50
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 50
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_2`3.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 74
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass6_0`3.<<Execute>b__0>d.MoveNext() in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 99
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3](Func`5 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 102
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3](Func`5 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 102
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_1`3.<<Execute>b__2>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 68
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Orchestration.ServerTasks.ParallelWorkOrderAsync`1.<>c__DisplayClass7_1.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/ParallelWorkOrderAsync.cs:line 61
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Orchestration.ServerTasks.OctoThreadClosureAsync`1.Execute(CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/OctoThreadClosureAsync.cs:line 61
                    |     
                    |       Failed: Download package octopussamples/randomquotes v1.0.31 from Docker Registry  'Octopus Docker Feed' at https://index.docker.io
10:58:38   Verbose  |         Running this script in the Octopus Server security context (NT AUTHORITY\SYSTEM)
10:58:38   Verbose  |         Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-12642
10:58:38   Verbose  |         Executable directory is 'C:\Windows\system32\WindowsPowershell\v1.0'
10:58:38   Verbose  |         Executable name or full path: 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe'
10:58:38   Verbose  |         No user context provided. Running as current user.
10:58:38   Verbose  |         Starting 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in working directory 'C:\Octopus\Work\20240202105838-12642-24' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
10:58:38   Verbose  |         Process 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in 'C:\Octopus\Work\20240202105838-12642-24' exited with code 0
10:58:38   Verbose  |         Using Calamari.win-x64 26.0.16
10:58:38   Verbose  |         Running this script in the Octopus Server security context (NT AUTHORITY\SYSTEM)
10:58:38   Verbose  |         Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-12642
10:58:38   Verbose  |         Executable directory is 'C:\Windows\system32\WindowsPowershell\v1.0'
10:58:38   Verbose  |         Executable name or full path: 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe'
10:58:38   Verbose  |         No user context provided. Running as current user.
10:58:38   Verbose  |         Starting 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in working directory 'C:\Octopus\Work\20240202105838-12642-25' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
10:58:38   Verbose  |         Calamari Version: 26.0.16
10:58:38   Verbose  |         Environment Information:
10:58:38   Verbose  |         OperatingSystem: Microsoft Windows NT 10.0.20348.0
10:58:38   Verbose  |         OsBitVersion: x64
10:58:38   Verbose  |         Is64BitProcess: True
10:58:38   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
10:58:38   Verbose  |         MachineName: OCTODC
10:58:38   Verbose  |         ProcessorCount: 8
10:58:38   Verbose  |         CurrentDirectory: C:\Octopus\Work\20240202105838-12642-25
10:58:38   Verbose  |         TempDirectory: C:\Windows\TEMP\
10:58:38   Verbose  |         HostProcess: Calamari (9496)
10:58:39   Verbose  |         Feed type provided `Docker` using DockerImagePackageDownloader
10:58:39   Verbose  |         Setting Proxy Environment Variables
10:58:39   Verbose  |         Name                           Value
10:58:39   Verbose  |         ----                           -----
10:58:39   Verbose  |         PSVersion                      5.1.20348.1850
10:58:39   Verbose  |         PSEdition                      Desktop
10:58:39   Verbose  |         PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
10:58:39   Verbose  |         BuildVersion                   10.0.20348.1850
10:58:39   Verbose  |         CLRVersion                     4.0.30319.42000
10:58:39   Verbose  |         WSManStackVersion              3.0
10:58:39   Verbose  |         PSRemotingProtocolVersion      2.3
10:58:39   Verbose  |         SerializationVersion           1.1.0.1
10:58:39   Verbose  |         PowerShell Environment Information:
10:58:39   Verbose  |         OperatingSystem: Microsoft Windows NT 10.0.20348.0
10:58:39   Verbose  |         OsBitVersion: x64
10:58:39   Verbose  |         Is64BitProcess: True
10:58:39   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
10:58:39   Verbose  |         MachineName: OCTODC
10:58:39   Verbose  |         ProcessorCount: 8
10:58:39   Verbose  |         CurrentDirectory: C:\Octopus\Work\20240202105838-12642-25
10:58:39   Verbose  |         CurrentLocation: C:\Octopus\Work\20240202105838-12642-25
10:58:39   Verbose  |         TempDirectory: C:\Windows\TEMP\
10:58:39   Verbose  |         HostProcess: powershell (9988)
10:58:40   Verbose  |         TotalPhysicalMemory: 9124808 KB
10:58:40   Verbose  |         AvailablePhysicalMemory: 4771960 KB
10:58:40   Info     |         Unable to connect to docker service
10:58:40   Error    |         NotSpecified: You will need docker installed and running to pull docker images
10:58:40   Error    |         At C:\Octopus\Work\20240202105838-12642-25\Bootstrap.Octopus.DockerLogin.ps1:855 char:2
10:58:40   Error    |         +     . '.\Octopus.DockerLogin.ps1'
10:58:40   Error    |         +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10:58:40   Error    |         at <ScriptBlock>, C:\Octopus\Work\20240202105838-12642-25\Octopus.DockerLogin.ps1: line 29
10:58:40   Error    |         at <ScriptBlock>, C:\Octopus\Work\20240202105838-12642-25\Bootstrap.Octopus.DockerLogin.ps1: line 855
10:58:40   Error    |         at <ScriptBlock>, <No file>: line 1
10:58:40   Error    |         at <ScriptBlock>, <No file>: line 1
10:58:40   Error    |         Failed to download package octopussamples/randomquotes v1.0.31 from feed: 'https://index.docker.io'
10:58:40   Error    |         Unable to log in Docker registry
10:58:40   Verbose  |         Process 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in 'C:\Octopus\Work\20240202105838-12642-25' exited with code 1
10:58:40   Verbose  |         Exit code: 1
                    |       
                    |   == Skipped: Step 2: DeployAppServiceStepTemplate ==
10:58:40   Info     |     Step "DeployAppServiceStepTemplate" runs only when all previous steps succeeded; skipping
                    |   
                    |   == Success: Release packages ==
10:58:40   Info     |     There are no packages to be released.
10:58:40   Verbose  |     Release Packages completed

More Information

Customer Ticket (Internal) - https://octopus.zendesk.com/agent/tickets/170951
Slack Discussion (Internal) - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1706873944485759

Workaround

Use the standard Deploy An Azure App Service Step instead of a Step Template.

[octoreleasebot]

Release Note: Fix Azure App Service steps failing to deploy when the step is added from a step template or cloned from an existing step, and a Docker feed is used

[Octobob]

🎉 The fix for this issue has been released in:

Release stream	Release
2024.3	2024.3.4814
2024.4+	all releases