Publish-BCContainerApp HTTP Error 503 when using -useDevEndpoint parameter

[JeroenHems]

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!

Describe the issue
I am trying to deploy Business Central containers to an Azure VM. The container deployment goes without any problems, but when I try to publish an app to the container, I receive a HTTP 503 error.

Scripts used to create container and cause the issue

$token = "*********************************************************************"
$licenseFile = "https://.../bc.flf
$credential = Get-Credential BCAdmin
$version = "18"
$languageCode = "nl"
$scripts = @('C:\_Temp\SupportFiles\Install-Certificate.ps1','C:\_Temp\SupportFiles\certificate.pfx')
$artifactUrl = Get-BCArtifactUrl -storageAccount "bcinsider" `
                                 -type "Sandbox"
                                 -select "Latest"
                                 -country $languageCode
                                 -version $version
                                 -sasToken $token

New-BCContainer -containerName DEV`
                -accept_eula `
                -memoryLimit 8G `
                -useTraefik `
                -PublicDnsName "$($alternateHostName).$($dnsDomain)" `
                -updateHosts `
                -credential $credential `
                -artifactUrl $artifactUrl `
                -isolation process `
                -includeAL `
                -includeTestLibrariesOnly `
                -includeTestToolkit `
                -licenseFile $licenseFile `
                -assignPremiumPlan `
                -auth NavUserPassword `
                -authenticationEmail $globalAdminEmail `
                -useSSL `
                -multitenant `
                -myScripts $scripts

$appFolder = "C:\_temp\solutionfiles\Trade365\Enterprise"

Publish-BCContainerApp -containerName DEV `
                       -appFile "$($appFolder)\Dependency\GAC Business Solutions_License_18.1.24830.0.app" `
                       -skipVerification `
                       --install `
                       --sync `
                       --credential $credential `
                       --useDevEndpoint

Full output of scripts

// New-BCContainer output

BcContainerHelper is version 2.0.14
BcContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Datacenter - ltsc2019
Docker Client Version is 19.03.14
Docker Server Version is 19.03.14
Removing C:\ProgramData\BcContainerHelper\Extensions\DEV
Downloading application artifact /sandbox/18.4.27785.0/nl
Downloading C:\Users\Adm_JeroenHems\AppData\Local\Temp\6\dece34b4-cd51-44aa-b0e8-f38b8e5c6cf6.zip
Unpacking application artifact to tmp folder
using Expand-Archive
Downloading platform artifact /sandbox/18.4.27785.0/platform
Downloading C:\Users\Adm_JeroenHems\AppData\Local\Temp\6\adb1b9e2-0540-43c1-8ccc-21fe24d348df.zip
Unpacking platform artifact to tmp folder
using Expand-Archive
Downloading Prerequisite Components
Downloading c:\bcartifacts.cache\sandbox\18.4.27785.0\platform\Prerequisite Components\Open XML SDK 2.5 for Microsoft Office\OpenXMLSDKv25.msi
Downloading c:\bcartifacts.cache\sandbox\18.4.27785.0\platform\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
Downloading c:\bcartifacts.cache\sandbox\18.4.27785.0\platform\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
Fetching all docker images
Fetching all docker volumes
Enabling SSL as otherwise all clients will see mixed HTTP / HTTPS request, which will cause problems e.g. on the mobile and modern windows clients
Using image mcr.microsoft.com/businesscentral:10.0.17763.1999
PublicDnsName is trd365-bc18.gaccloud.com
Creating Container DEV
Version: 18.4.27785.0-NL
Style: sandbox
Multitenant: Yes
Platform: 18.0.27769.0
Generic Tag: 1.0.1.7
Container OS Version: 10.0.17763.1999 (ltsc2019)
Host OS Version: 10.0.17763.2029 (ltsc2019)
WARNING: Host OS and Base Image Container OS doesn't match and process isolation is specified. If you encounter issues, you could try to specify -isolation hyperv
Using process isolation
Using locale nl-NL
Adding special CheckHealth.ps1 to enable Traefik support
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file https://gaccourselabstorage.blob.core.windows.net/scriptextensions/bc.flf
Downloading C:\ProgramData\BcContainerHelper\Extensions\DEV\my\license.flf
Additional Parameters:
-e webserverinstance=DEV
-e publicdnsname=trd365-bc18.gaccloud.com
-l "traefik.protocol=https"
-l "traefik.web.frontend.rule=PathPrefix:/DEV"
-l "traefik.web.port=443"
-l "traefik.soap.frontend.rule=PathPrefix:/DEVsoap;ReplacePathRegex: ^/DEVsoap(.*) /BC$1"
-l "traefik.soap.port=7047"
-l "traefik.rest.frontend.rule=PathPrefix:/DEVrest;ReplacePathRegex: ^/DEVrest(.*) /BC$1"
-l "traefik.rest.port=7048"
-l "traefik.dev.frontend.rule=PathPrefix:/DEVdev;ReplacePathRegex: ^/DEVdev(.*) /BC$1"
-l "traefik.dev.port=7049"
-l "traefik.snap.frontend.rule=PathPrefix:/DEVsnap;ReplacePathRegex: ^/DEVsnap(.*) /BC$1"
-l "traefik.snap.port=7083"
-l "traefik.dl.frontend.rule=PathPrefixStrip:/DEVdl"
-l "traefik.dl.port=8080"
-l "traefik.dl.protocol=http"
-l "traefik.enable=true"
-l "traefik.frontend.entryPoints=https"
--env customNavSettings=PublicODataBaseUrl=https://trd365-bc18.gaccloud.com/DEVrest/odata,PublicSOAPBaseUrl=https://trd365-bc18.gaccloud.com/DEVsoap/ws,PublicWebBaseUrl=https://trd365-bc18.gaccloud.com/DEV
Files in C:\ProgramData\BcContainerHelper\Extensions\DEV\my:
- AdditionalOutput.ps1
- certificate.pfx
- CheckHealth.ps1
- Install-Certificate.ps1
- license.flf
- MainLoop.ps1
- SetupNavUsers.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container DEV from image mcr.microsoft.com/businesscentral:10.0.17763.1999
2f6cf6d78feb0e4f01339dd1852cf11acb24408ded4b0d2407159c03bcf5c4dc
Waiting for container DEV to be ready
Using artifactUrl https://bcinsider.azureedge.net/sandbox/18.4.27785.0/nl
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server

Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\18.4.27785.0\platform\ServiceTier\Program Files

c:\dl\sandbox\18.4.27785.0\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\sandbox\18.4.27785.0\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\sandbox\18.4.27785.0\platform\WindowsPowerShellScripts\WebSearch
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Determining Database Collation from c:\dl\sandbox\18.4.27785.0\nl\BusinessCentral-NL.bak

Restoring CRONUS Demo Database

Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS

Exporting Application to CRONUS

Removing Application from tenant

Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Copying Web Client Files
c:\dl\sandbox\18.4.27785.0\platform\WebClient\Microsoft Dynamics NAV
Copying Client Files
c:\dl\sandbox\18.4.27785.0\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\sandbox\18.4.27785.0\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\sandbox\18.4.27785.0\platform\LegacyDlls\systemFolder
Copying ModernDev Files
c:\dl\sandbox\18.4.27785.0\platform
c:\dl\sandbox\18.4.27785.0\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\18.4.27785.0\nl\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\18.4.27785.0\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\18.4.27785.0\nl\Extensions
Copying Applications
C:\dl\sandbox\18.4.27785.0\platform\Applications
Copying Applications.NL
C:\dl\sandbox\18.4.27785.0\nl\Applications.NL
Starting Business Central Service Tier

Importing license file

Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files

Attaching files as new Database default

Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False

Sync'ing Tenant

Tenant is Operational
Stopping Business Central Service Tier
Installation took 168 seconds
Installation complete
Initializing...
Setting host.containerhelper.internal to 172.30.160.1 in container hosts file
Starting Container
Hostname is DEV
PublicDnsName is trd365-bc18.gaccloud.com

Using NavUserPassword Authentication
Creating Self Signed Certificate

Self Signed Certificate Thumbprint B4B769848DCEE35201E5A8C9F4C00835956FA7D4
Modifying Service Tier Config File with Instance Specific Settings
Modifying Service Tier Config File with settings from environment variable
Setting PublicODataBaseUrl to https://trd365-bc18.gaccloud.com/DEVrest/odata
Setting PublicSOAPBaseUrl to https://trd365-bc18.gaccloud.com/DEVsoap/ws
Setting PublicWebBaseUrl to https://trd365-bc18.gaccloud.com/DEV
Starting Service Tier

Registering event sources
Creating DotNetCore Web Server Instance

Enabling Financials User Experience
Using license file 'c:\run\my\license.flf'
Import License

Dismounting Tenant

Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False

Sync'ing Tenant

Tenant is Operational
Creating http download site
Setting SA Password and enabling SA
Creating bcadmin as SQL User and add to sysadmin
Creating SUPER user

Assign Premium plan for BCADMIN
Container IP Address: 172.30.163.125
Container Hostname  : DEV
Container Dns Name  : trd365-bc18.gaccloud.com
Web Client          : https://trd365-bc18.gaccloud.com/DEV/?tenant=default
Dev. Server         : https://trd365-bc18.gaccloud.com
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting DEV to 172.30.163.125 in host hosts file
Setting DEV-default to 172.30.163.125 in host hosts file
Setting DEV-default to 172.30.163.125 in container hosts file

Files:
http://trd365-bc18.gaccloud.com:8080/ALLanguage.vsix
http://trd365-bc18.gaccloud.com:8080/certificate.cer

Container Total Physical Memory is 32.0Gb
Container Free Physical Memory is 25.9Gb

Initialization took 63 seconds
Ready for connections!

Reading CustomSettings.config from DEV
PS C:\_scripts> TerminatingError(Invoke-Command): "The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Because the session state for session WinRM2, bb388166-d3d3-4778-b4b4-a77d1553d46f, f5555f7ace72f14c34b9d222adc4505712bbc66db8781d51249e18521b8e620e is not equal to Open, you cannot run a command  in the session.  The session state is Broken."
Creating Desktop Shortcuts for DEV
INFO: Synchronizing Test Runner on default
App successfully synchronized
INFO: Installing Test Runner on default
App successfully installed
INFO: Synchronizing Any on default
App successfully synchronized
INFO: Installing Any on default
App successfully installed
INFO: Synchronizing Library Assert on default
App successfully synchronized
INFO: Installing Library Assert on default
App successfully installed
INFO: Synchronizing Permissions Mock on default
App successfully synchronized
INFO: Installing Permissions Mock on default
App successfully installed
INFO: Synchronizing Library Variable Storage on default
App successfully synchronized
INFO: Installing Library Variable Storage on default
App successfully installed
INFO: Publishing C:\ProgramData\BcContainerHelper\Extensions\DEV\6be65204-b9c3-4686-a51b-2f70d2debdee\Microsoft_System Application Test Library_18.4.27785.0.app
INFO: Synchronizing System Application Test Library on tenant default
INFO: Installing System Application Test Library on tenant default
App Microsoft_System Application Test Library_18.4.27785.0.app successfully published
INFO: Publishing C:\ProgramData\BcContainerHelper\Extensions\DEV\0156d70c-c7f5-4d76-bbe0-5e43942cb0cd\Microsoft_Tests-TestLibraries_18.4.27785.0.app
INFO: Synchronizing Tests-TestLibraries on tenant default
INFO: Installing Tests-TestLibraries on tenant default
App Microsoft_Tests-TestLibraries_18.4.27785.0.app successfully published
TestToolkit successfully imported
INFO: Extracting C:\Applications.NL\Base Application.Source.zip
Creating .net Assembly Reference Folder for VS Code
INFO: Copying DLLs from C:\Windows\assembly to assemblyProbingPath
INFO: Copying DLLs from C:\Windows\Microsoft.NET\assembly to assemblyProbingPath
INFO: Copying DLLs from C:\Program Files\Microsoft Dynamics NAV\180\Service to assemblyProbingPath
INFO: Copying DLLs from C:\Program Files (x86)\Microsoft Dynamics NAV\180\RoleTailored Client to assemblyProbingPath
INFO: Copying DLLs from C:\Test Assemblies\Mock Assemblies to assemblyProbingPath
INFO: Copying DLLs from C:\Program Files (x86)\Open XML SDK to assemblyProbingPath
Container DEV successfully created
Because of Traefik, the following URLs need to be used when accessing the container from outside your Docker host:
Web Client:        https://trd365-bc18.gaccloud.com/DEV?tenant=default&page=
SOAP WebServices:  https://trd365-bc18.gaccloud.com/DEVsoap
OData WebServices: https://trd365-bc18.gaccloud.com/DEVrest
Dev Service:       https://trd365-bc18.gaccloud.com/DEVdev
Snapshot Service:  https://trd365-bc18.gaccloud.com/DEVsnap
File downloads:    https://trd365-bc18.gaccloud.com/DEVdl

Use:
Get-BcContainerEventLog -containerName DEV
 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName DEV
 to get debug information about the container
Enter-BcContainer -containerName DEV
 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName DEV
 to remove the container again
docker logs DEV
 to retrieve information about URL's again

// Publish-BCContainerApp Output

Disabling SSL Verification
Publishing GAC Business Solutions_License_18.1.24830.0.app to https://172.30.163.125:7049/BC/dev/apps?SchemaUpdateMode=synchronize&tenant=default
PS C:\_scripts> TerminatingError(ConvertFrom-Json): "Invalid JSON primitive: ."
>> TerminatingError(): "Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
"
PS C:\_scripts> TerminatingError(): "Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
"
>> TerminatingError(): "Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
"
Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>

At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\2.0.14\AppHandling\Publish-NavContainerApp.ps1:245
char:25
+                         throw $message
+                         ~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Status Code Ser.../BODY></HTML>
:String) [], RuntimeException
    + FullyQualifiedErrorId : Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
...

Screenshots
N/A

Additional context

• I have 3 machines with 3 containers each, it happens on all of them.
• Omitting the parameter -useDevEndpoint bypasses the problem, but our developers want to connect to this endpoint from VSCode, so it must be available in the end.
• It still works in Docker Desktop, according to colleagues.
• On older Windows Server 2019 Core machines we run on prem, it still works.

[freddydk]

Could you try to do the same without using traefik?

[JeroenHems]

Same result it seems:

.\TRADE365-18-2.ps1
Disabling SSL Verification
Publishing GAC Business Solutions_License_18.1.24830.0.app to https://172.30.175.82:7049/BC/dev/apps?SchemaUpdateMode=synchronize&tenant=default
Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>
At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\2.0.14\AppHandling\Publish-NavContainerApp.ps1:245
char:25
+                         throw $message
+                         ~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Status Code Ser.../BODY></HTML>
:String) [], RuntimeException
    + FullyQualifiedErrorId : Status Code ServiceUnavailable : Service Unavailable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Service Unavailable</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Service Unavailable</h2>
<hr><p>HTTP Error 503. The service is unavailable.</p>
</BODY></HTML>

[freddydk]

Strange
I would probably have to have access to an environment in a troubleshooting call in order to figure out what this is.
I will be on vacation for the reminder of July, but will be back beginning of August if you haven't found a solution.
Note that I published new generic images that matches the OS you are running yesterday, you could try again, but I don't think that will change anything.

[JeroenHems]

I don't think it will, because the most recent image is from 2 days ago. But I'm rolling it out again, just to be sure. Enjoy your vacation! [Update] - Just tried redeploying the container and apps, same result. I also downloaded the container event logs, but they don't show anything that points to a possible cause.

[JeroenHems]

I have just deployed another VM for one of our trainers. Size B2ms, OS Image Windows Server 2019 Server Core with Containers. Instead of pulling the artifacts from bcinsider, I pulled them from bcartifiacts. And now I am able to publish apps using the dev endpoint.

[JeroenHems]

Using version 2.0.13 of BCContainerHelper the -useDevEndpoint parameter works. So for now I'll use the following commands to install and import BCContainerHelper:

Install-Module BCContainerHelper -RequiredVersion 2.0.13
Import-Module BCContainerHelper -RequiredVersion 2.0.13

[freddydk]

There are no changes to 2.0.14 in this area (issue #1967 was the only fix in 2.0.14) and also not in 2.0.15 preview.
Have you checked whether this is related to traefik?

[JeroenHems]

I was mistaken. I have tried so many things. In the end I tried the version we use on our on-prem machines, 2.04, to get everything up and running. Weren't you on holliday? :) Op 17 jul. 2021 07:02 schreef Freddy Kristiansen ***@***.***>: There are no changes to 2.0.14 in this area (issue #1967<#1967> was the only fix in 2.0.14) and also not in 2.0.15 preview. Have you checked whether this is related to traefik?

[freddydk]

2.0.4 was shipped on February 1st - no changes went into useDevEnvPoint after that.
Did you check whether this was related to traefik?

[JeroenHems]

Yes, first thing I did. Didn't make a difference though. Op 18 jul. 2021 07:11 schreef Freddy Kristiansen ***@***.***>: 2.0.4 was shipped on February 1st - no changes went into useDevEnvPoint after that. Did you check whether this was related to traefik?

[JeroenHems]

We've been doing a lot of troubleshooting, and we have come to the point where we're suspicious of Traefik as well. There is a new image that's being used for Traefik (on top of the stefanscherer image). In combination with BCContainerHelper 2.0.4 it doesn't have an issue, with later versions (we've tested with versions 2.0.9 through 2.0.14) it shows the 503 behavior when trying to publish apps using -useDevEndpoint.

[warlof]

Hello,

I'm facing the same issue apparently but simply when publishing from Visual Studio Code.
After a few investigation time, it appears issue isn't with traefik (or not only), but more with current Business Central image.

We still have working container on the server, which are using the build 10.0.17763.1999 from image mcr.microsoft.com/businesscentral (using hash 9ab098ce53ac)
However, all those which have been renew, are using build 10.0.17763.2061 from image mcr.microsoft.com/businesscentral - and we got error 503

base on a 2061 build, calling this URI https://172.26.16.117:7049/BC/dev/metadata/ is resulting in a 503.
From there, I assume issue isn't coming from traefik.

However, before renewing our containers; we also had change our publicDns from mycustomer-dev-docker.francecentral.cloudapp.azure.com to dev.erp.mycustomer.io. If we're using the old publicDns value on build 2061, we're able to query upper endpoint which return HTTP 200 with config metadata.

[freddydk]

Did you manage to get this working or are you still facing an issue?