Download of container image from ACR fails

[ivansharamok]

I'm running SF runtime version 6.0.232.9494 on Windows Server 2016 VM. I created Container type SF project that points to a container image hosted in Azure Container Registry (ACR). My container image is based on microsoft/windowsservercore:1709 image and has .NET Framework app in it.

ServiceManifest

.............
<CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ContainerHost>
        <ImageName>myregistry.azurecr.io/sample:1.0-1709</ImageName>
      </ContainerHost>
    </EntryPoint>
  </CodePackage>
.............

ApplicationManifest

.............
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="SamplePkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides />
    <Policies>
      <ContainerHostPolicies CodePackageRef="Code">
        <RepositoryCredentials AccountName="myregistry" Password="myPassword" PasswordEncrypted="false"/>
      </ContainerHostPolicies>
    </Policies>
  </ServiceManifestImport>
.............

I tried to deploy my SF Container app to both local (1 Node) cluster and SF cluster (3 Node) hosted in Azure. With both clusters I get the same error message indicating that download of container image has failed. Unfortunately I cannot find more details than this output:

ApplicationName                    : fabric:/Sample102
NodeName                           : _Node_0
AggregatedHealthState              : Error
UnhealthyEvaluations               :
                                     Unhealthy deployed service packages: 100% (2/2).

                                     Unhealthy deployed service package: ApplicationName='fabric:/Sample102',
                                     ServiceManifestName='SamplePkg', ServicePackageActivationId='',
                                     NodeName='_Node_0', AggregatedHealthState='Error'.

                                        Error event: SourceId='System.Hosting', Property='Download:1.0:1.0'.

                                     Unhealthy deployed service package: ApplicationName='fabric:/Sample102',
                                     ServiceManifestName='SamplePkg',
                                     ServicePackageActivationId='05ac8b74-f176-45c9-a8e0-03cb01fd6873',
                                     NodeName='_Node_0', AggregatedHealthState='Error'.

                                        Error event: SourceId='System.Hosting',
                                     Property='ServiceTypeRegistration:SampleType'.

DeployedServicePackageHealthStates :
                                     ServiceManifestName   : SamplePkg
                                     ServicePackageActivationId :
                                     NodeName              : _Node_0
                                     AggregatedHealthState : Error

                                     ServiceManifestName   : SamplePkg
                                     ServicePackageActivationId : 05ac8b74-f176-45c9-a8e0-03cb01fd6873
                                     NodeName              : _Node_0
                                     AggregatedHealthState : Error

HealthEvents                       :
                                     SourceId              : System.Hosting
                                     Property              : Activation
                                     HealthState           : Ok
                                     SequenceNumber        : 131589117961840587
                                     SentAt                : 12/28/2017 5:16:36 AM
                                     ReceivedAt            : 12/28/2017 5:16:36 AM
                                     TTL                   : Infinite
                                     Description           : The application was activated successfully.
                                     RemoveWhenExpired     : False
                                     IsExpired             : False
                                     Transitions           : Warning->Ok = 12/28/2017 5:16:36 AM, LastError = 1/1/0001
                                     12:00:00 AM

I've tried to use all available passwords for ACR that got via CLI command:
az acr credential show --name myregistry

When using local SF cluster I see a bunch of warnings in Event Viewer -> MSFT-Service Fabric -> Admin log:

client-localhost:19080 target 14fc4a794e0-a1ae9601-5ec5-4544-8e3e-21d526c6ea64: dropping message: incoming frame length 1347703880 exceeds limit 4469566

CreateProcessW("dockerd" -H localhost:2375 -H npipe:// --pidfile C:\SfDevCluster\Data\sfdocker.pid) failed. ErrorCode=0x80070002

Activation path for command line '"dockerd" -H localhost:2375 -H npipe:// --pidfile C:\SfDevCluster\Data\sfdocker.pid' not found. This can happen if the code package entry point path is too long.

End(ContainerProvider process Activate): ErrorCode=FABRIC_E_PATH_TOO_LONG

How do I troubleshoot image download error?
What do the Event Viewer warnings mean? Are they related to image download error I'm getting or that's different issue(s)?

[sfmskywalker]

I'm having the exact same issue. Only difference is that I created a SF cluster on Azure.

[ivansharamok]

The issue seems to be related to the size of the container image. I was able to successfully deploy a Linux container app in SF cluster on Azure that is based on a container image about 300Mb in size. I used exact same commands to deploy my Linux container as I used for Windows container. At this point the differences I see are SF cluster type (Linux vs Windows) and container images size ~300Mb for Linux and ~6Gb for Windows.

[arunmohandas196]

I'm facing similiar issue too.
Trying to deploy a docker container with java application in onebox/cloud from mac.
Which fails on container download from ACR.
Not much details on the error log as well except( There was an error during download.Failed to download container image)
My docker container size is ~36Mb
Able to download and run container locally.

Any help will be appreciated on how to debug this issue.

[RajeetN]

What version of docker are you running on?

[sfmskywalker]

17.09.1-ce-win42 (14687)

[RajeetN]

sfmskywalker, you mentioned that you deployed the cluster on Azure? Which gallery image did you use for the cluster? SF requires "ee" version of docker which is installed on Azure gallery images with "*-with containers" in image name.

[Marusyk]

Is it possible to run Linux container in Service Fabric cluster on Windows? I'm getting:

There was an error during download.Failed to download container image
my image

 <ContainerHost>
        <ImageName>rabbitmq:3.6-management</ImageName>
 </ContainerHost>

[mani-ramaswamy]

No, this is not supported. Running Linux containers need a Service Fabric Linux cluster at this time.

[sanjaydebnath]

Same issue with windows image. I have deployed a SF cluster (win server 16 with containers, version 6.0.232.9494) with one node & trying to host an image from ACR. Getting same error as below. I have provided password correctly. Also I tried to point to public docker hub image too, same issue.

Error event: SourceId='System.Hosting', Property='Download:1.0:1.0'.
There was an error during download.Failed to download container image sanjaysrepo.azurecr.io/quotesgenerator:v2

The image (simple asp.net core api app) is build for windows OS in a win 10 machine & published to docker hub & ACR

Update: If I go inside the VM and try to execute the below command (for public docker hub) I get folder not found error after extracting. Seems somehow folders are getting deleted after extract!

PS C:\Users\sanjayd> docker run dsanjay/quotesgenerator:latest
Unable to find image 'dsanjay/quotesgenerator:latest' locally
latest: Pulling from dsanjay/quotesgenerator
407ada6e90de: Extracting 81.04MB/81.04MB
9c9e16cbf19f: Download complete
ccb735991f8c: Download complete
a56cd95cb578: Download complete
8d17ef737799: Download complete
502a96c9272b: Download complete
4297e35a3bfe: Download complete
baef27e01fb3: Download complete
dc4943a2272b: Download complete
3739c9aa2f65: Download complete
dccdba03d3b0: Download complete
d429f24ce170: Download complete
C:\Program Files\Docker\docker.exe: failed to register layer: re-exec error: exit status 1: output: ProcessUtilityVMImage C:\ProgramData\docker\wind
owsfilter\28bdbee861bf15374668f8d96bfdd807e5720e5e864e630f1188a6a23757dfa0\UtilityVM: The system cannot find the path specified.
See 'C:\Program Files\Docker\docker.exe run --help'.

Update 2: Got the issue. SF deploys old win server 16 OS version, so need to use '2.0.5-2.1.4-nanoserver-sac2016' tag to build the image, I was using latest for fall creator's update (1709).