Unable to download container image from private repository

[smokedlinq]

I have a SF (Microsoft.Azure.ServiceFabric.WindowsServer.5.6.220.9494) running the ClusterConfig.Unsecure.DevCluster configuration.

Docker was installed prior and was working and running these same images. Before installed SF I removed all running containers and cleaned up images and private repository logins.

I am trying to deploy an application pointing to my private repo, e.g. myrepo.azurecr.io/sf/myapp

I am able to use the repository credentials specified in the manifest to login to the repo from docker cli.

When SF tries to deploy the container it states:

Error event: SourceId='System.Hosting', Property='Download:1.0:1.0'.
There was an error during download.Failed to download container image myrepo.azurecr.io/sf/myapp

In the admin log I see this sequence of events:

End(BeginDownloadAndActivate): Error=HostingDeploymentInProgress, VersionedServiceTypeId={MyAppType_App10:MyAppPkg:MyAppType,1.0:1.0:131420102687941111}, ActivationContext=551bf757-1e64-45ca-9812-b99f3875df69, ServicePackagePublicActivationId=d87665d6-1421-4c9d-8d36-73442c5d7b80, SequenceNumber=185
DownloadContainerImages returned 0xd00000e5
Failed to import docker image error 0xd00000e5.
EndSendRequest for image history Error 0xd00000e5
DownloadContainerImages returned 0xd00000e5
Failed to import docker image error 0xd00000e5.
EndSendRequest for image history Error 0xd00000e5
80b671e9b3c9184bbd86d2f150c58135:131419843158391101:131419843669317074 failed to send message AddInstance to node 5101db1125ead8d47d6f93321d3eb754:131419843160891129 with error FABRIC_E_TIMEOUT

ServiceManifest

<EntryPoint>
  <ContainerHost>
    <ImageName>myrepo.azurecr.io/sf/myapp</ImageName>
  </ContainerHost>
</EntryPoint>

ApplicationManifest

<Policies>
  <ContainerHostPolicies CodePackageRef="Code">
    <RepositoryCredentials AccountName="myrepo" Password="mysecret" PasswordEncrypted="false" />
  </ContainerHostPolicies>
</Policies>

Also, it appears that dockerd is not running though I saw in a previous debug log that the docker process manager started dockerd successfully, but then exited with error code 1 which the log said was ok. I haven't seen this happen again in the debug log as of yet.

One other thing to note, the image is rather large at about 9gb

Update: I installed docker on another host, logged into the private repo, pulled the image and then installed SF and was able to deploy the same manifests and run the container successfully. If the image does not exist in docker prior to SF trying to pull it then it fails. with the errors above.

dockerd process is not running successfully if the image doesn't exist

Update: I wiped the images docker rmi $(docker images -q) from my local development workstation running Windows Server 2016, using the local dev SF deployment and deploying the application causes docker to download the image from my private repo. I did the same process on the broken 2016 container host but the main difference is that server is running the Core OS so there is no GUI, the docker host that is working is my 2016 development workstation so it's local to Visual Studio.

Is my core container host broke, unsupported, or what? I am mostly at this point interested in how do I find out what's wrong with it so if it's something I did I don't do it again.

[RajeetN]

Server core host should be fine and work. Is this host deployed on Azure or local onebox? My suspicion is that image download is timing out, but I could be wrong and would need traces to diagnose the issue further.

[smokedlinq]

Local one box. If you can point me to docs on enabling tracing I'll gladly do that and provide it.

[smokedlinq]

Just for fun I deployed another Windows Server 2016 core server, had docker running on it, installed service fabric configuration same as my full GUI version and the dockerd process isn't getting started.

PS C:\> docker -H localhost:2375 images
error during connect: Get http://localhost:2375/v1.26/images/json: dial tcp [::1]:2375: connectex: No connection could be made because the target machine actively refused it.

[felschr]

I get the same "Failed to download container" error but on an Azure cluster (I tried Windows preview and Ubuntu).
I'm using the CI/CD integration to deploy my ASP.NET Core application with multiple services. Locally I am able to run the services using docker-compose from within Visual Studio.
When I try to run it in my local SF cluster I get a similar error:

Error event: SourceId='System.Hosting', Property='Download:1.0:1.0'.
There was an error during download.Container deployment is not supported on the node.

I have the latest preview installed though.

[RajeetN]

Are you using Server2016DataCenter-WithContainers image? The error you are seeing indicates we are not able to find docker on the node.

[felschr]

@RajeetN No, I only have the preview SF SDK & Docker for Windows installed on my local machine.
I didn't know I need a special OS. This explains why it doesn't work locally.
The problem on Azure remains, though., where I used the Server2016DataCenter-WithContainers.

[mani-ramaswamy]

Can you please retry with the 5.7 bits (non-preview) and tell us what you find? With Server2016DataCenter-WithContainers, it should work.

[felschr]

I've retried this with Azure Service Fabric 5.7 with Linux.
Can I run Linux containers on a Windows Service Fabric cluster? If so, I'll try that, too.

[RajeetN]

No you cannot run Linux containers on windows Service Fabric clusters today.

[felschr]

Ok, in that case I'm still blocked by this issue.

[prasadker]

Can you RDP on your machine and check if Docker is running?
Also, you mentioned Docker compose - can you paste the compose file here? Service Fabric supports only few compose directives for now.

[felschr]

Here are my compose files:

docker-compose.yml

services:
  activities:
    image: '[mydomain].azurecr.io/activities@[SHA]'
  calendar:
    image: '[mydomain].azurecr.io/calendar@[SHA]'
  usermanagement:
    image: '[mydomain].azurecr.io/usermanagement@[SHA]'
version: '2.0'

docker-compose.override.yml

version: '2'

services:
  usermanagement:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - "8001:80"

  activities:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - "8002:80"

  calendar:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - "8003:80"

Well, it's a Linux cluster, so I guess I cannot RDP into it, but I'll try if I can access it via SSH somehow.

[prasadker]

The directives look ok, but SF only supports docker compose version 3, so I don't know if it is rejecting checking the version - https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-docker-compose#supported-compose-directives

Yes please SSH into the node using the cluster FQDN and port 3389 for node 0, 3390 for node 1 and so on.

[felschr]

When I change the versions to 3 I get the same error.

@prasadker Thanks for the quick info on how to access the nodes via SSH.
Here are the results of docker ps on the node:

felix_schroeter@LinuxNode000000:~$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.30/containers/json: dial unix /var/run/docker.sock: connect: permission denied
felix_schroeter@LinuxNode000000:~$ sudo su
root@LinuxNode000000:/home/felix_schroeter# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@LinuxNode000000:/home/felix_schroeter#

And here is the result of ps -aux | grep "[d]ocker":

root@LinuxNode000000:/home/felix_schroeter# ps -aux | grep "[d]ocker"
root       6811  0.0  0.3 275856  7068 ?        Ssl  Sep19   0:42 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root     105012  0.1  1.7 699400 34704 pts/7    Ssl+ Sep18   3:02 /usr/bin/dockerd -H localhost:2375 -H unix:///var/run/docker.sock --pidfile /mnt/sfroot/sfdocker.pid

[felschr]

Is docker-compose supposed to be isntalled?:

root@LinuxNode000000:/home/felix_schroeter# docker-compose ps
The program 'docker-compose' is currently not installed. You can install it by typing:
apt install docker-compose

[mani-ramaswamy]

If you are operating against a SF cluster, you shouldn't need that. You will be using instructions @ https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-docker-compose.

Since many issues have been discussed, to clarify exactly where you are blocked - you are trying to run (Linux) containers on a (Linux) SF cluster in Azure, correct?

[felschr]

@mani-ramaswamy Yes, Linux containers on a Linux Service Fabric cluster.
I was using the official Service Fabric docker-compose VSTS release task to deploy.

It's an ASP.NET Core project created with Docker support via Visual Studio.
I already got it running on a Linux Docker swarm cluster on Azure.

[mani-ramaswamy]

Can you try connecting to the cluster via CLI, and then running the install script below?

sfctl cluster select --endpoint http://:19080
sfctl application upload --path --show-progress
sfctl application provision --application-type-build-path
sfctl application create --app-name fabric:/appName--app-type --app-version

And if you don't have a default service in the app manifest, then create the service as well.
sfctl service create --name fabric:/appName/serviceName --service-type --stateless --instance-count 1 --app-id --singleton-scheme

If you share your manifest files, I can try it on my cluster and see what's going on. Are you able to get it to work with a public docker hub image?

[mani-ramaswamy]

And to install sfctl, use the instructions @ https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cli.

[ericmaia]

I have an issue that may be related. I am getting the "failed to download" error when I try to deploy a SF Container project to my local cluster. I have noticed that when the local cluster starts up, the Docker engine dies. If Service Fabric is trying to use Docker to pull and install the image, then it would fail. I have already posted an issue in Docker For Windows (docker/for-win#1175) but would it be helpful for me to add more details here, or open a new issue for Service Fabric?

Edit: never mind, I just got a comment on that issue saying that SF does not support containers on Windows 10.

[Marusyk]

So, is it possible to run Linux container in Service Fabric on Windows? I'm getting:

There was an error during download.Failed to download container image
my image

 <ContainerHost>
        <ImageName>rabbitmq:3.6-management</ImageName>
 </ContainerHost>

[mani-ramaswamy]

No, to run Linux containers, you need a SF Linux cluster at this time.

[Marusyk]

@mani-ramaswamy ok, thanks. After switch Docker to the Windows container I'm also getting

There was an error during download.Failed to download container image
my image micdenny/rabbitmq-windows

https://hub.docker.com/r/micdenny/rabbitmq-windows/

What is the problem?

[mani-ramaswamy]

It could be timing out. Can you try the instructions @ https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-get-started-containers#configure-container-image-download-time and increase the time out and report back?

[mani-ramaswamy]

Please also ensure that you are using docker EE (and not the CE variant)

[Marusyk]

@mani-ramaswamy Thanks for response

By default, the Service Fabric runtime allocates a time of 20 minutes to download and extract container images

I tried very small containers and got error immediately after start.

Please also ensure that you are using Docker EE (and not the CE variant)

I'm using Docker CE on my dev machine.
Documentations says:

Prerequisites

• Install Docker CE for Windows so that you can run containers on Windows 10.

Docker EE for Windows requires Windows Server 2016 or later. I'm not developing on Windows Server

[mani-ramaswamy]

Service Fabric cannot run Windows containers on Windows 10 locally at present. This will be fixed in an upcoming release.

[muskanaul]

@mani-ramaswamy Any idea on how soon it will be fixed?

[mani-ramaswamy]

We're presently testing internal builds with this - the next minor version update of SF (6.2) will have this fixed.

[rn-3]

Getting the same error SF running on local cluster (6.1.467.9494).
Host is a VM running Windows Server 2016 & the guest Container is Windows Server Core.
The error is reported immediately so this is definitely not a time-out issue.

[mani-ramaswamy]

To re-confirm, the VM is running Windows Server 2016 Datacenter with containers, and the guest container is Windows Server 2016 Datacenter Server Core. The reason I ask is that Windows containers aren't compatible across releases - thus, Windows Server 2016 container aren't compatible with Windows Server 2016 version 1709 hosts and vice versa.

@harahma @RajeetN

[mani-ramaswamy]

BTW, Docker for Windows (CE) isn't supported, if you were using that. You need EE on the Server.

[rn-3]

Yes, the guest & the host are both - Microsoft Windows [Version 10.0.14393]. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: Mani Ramaswamy <notifications@github.com> Sent: Friday, March 30, 2018 9:24:14 AM To: Azure/service-fabric-issues Cc: rn-3; Comment Subject: Re: [Azure/service-fabric-issues] Unable to download container image from private repository (#315) To re-confirm, the VM is running Windows Server 2016 Datacenter with containers, and the guest container is Windows Server 2016 Datacenter Server Core. The reason I ask is that Windows containers aren't compatible across releases - thus, Windows Server 2016 container aren't compatible with Windows Server 2016 version 1709 hosts and vice versa. @harahma<https://github.com/harahma> @RajeetN<https://github.com/RajeetN> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://github.com/Azure/service-fabric-issues/issues/315#issuecomment-377562257>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ARnaj5bmuO8rKveuWyDFDNMPzotji65Xks5tjlwugaJpZM4N7UTD>.

[rn-3]

Using Docker version 17.06.2-ee-6, build e75fdb8. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: Mani Ramaswamy <notifications@github.com> Sent: Friday, March 30, 2018 9:36:04 AM To: Azure/service-fabric-issues Cc: rn-3; Comment Subject: Re: [Azure/service-fabric-issues] Unable to download container image from private repository (#315) BTW, Docker for Windows (CE) isn't supported, if you were using that. You need EE on the Server. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://github.com/Azure/service-fabric-issues/issues/315#issuecomment-377564939>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ARnajzMuRH5V-CcOe1BBXecJMcccVIS0ks5tjl70gaJpZM4N7UTD>.

[rn-3]

@mani-ramaswamy @RajeetN
Any update on this issue?

[RajeetN]

@rn-3, do you have version tag specified for your image?

[rn-3]

microsoft/windowsservercore:latest

[RajeetN]

alright, this is not about download from private repository that the original issue reported was about then? Could you please share the traces from your machine? If this is a cluster deployed on azure, please send us the cluster resource name and region.

[rn-3]

I'm getting the same error - 'Failed to download container image hub.docker.com/r//'
I'm using a local cluster on a VM. I'm seeing these errors in the Service Fabric Explorer, Where/how do I get traces?

[rn-3]

I tried deploying again using the docker-compose file and it seems like the Container's have created & started...
Initially, I tried using the Visual Studio Solution & Project template...

[ddobric]

I'm also getting same error when using locally (dev cluster) the local registry.
I.E.:

<CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
      <ContainerHost>
        <ImageName>mydockerapp:v1</ImageName>
      </ContainerHost>
    </EntryPoint>
    <!-- Pass environment variables to your container: -->
    <!--
    <EnvironmentVariables>
      <EnvironmentVariable Name="VariableName" Value="VariableValue"/>
    </EnvironmentVariables>
    -->
  </CodePackage>

[artisticcheese]

This shall be SF issue. I deploy this to 3 different SF servers and only one of them showing this issue. All are Windows 2016 LTSC and image is exactly the same, so it can not be mismatch etc.

[mikkelhegn]

@harahma

[harahma]

@aryamsft Can you take look at it?

[artisticcheese]

@harahma This is relevant to the issue in other thread with bunch of files created in temporary folder on that server. microsoft/service-fabric-issues#1122

[Shrirang97]

Now can we deploy containers in windows 10 1909 machine?