Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting "No digest available for image" when building Docker image #200

Closed
IgorFedchenko opened this issue Jul 15, 2020 · 19 comments
Closed
Assignees
Labels
kind/bug Some behavior is incorrect or out of spec
Milestone

Comments

@IgorFedchenko
Copy link

Hello guys.

From time to time, we are getting the following error during CI build:

Diagnostics:
  docker:image:Image (myimage):
    error: Pulumi.ResourceException: No digest available for image myimage:0.9.5
       at Pulumi.Docker.Docker.BuildImageAsync(String imageName, Union`2 pathOrBuild, Resource logResource, String[] cacheFrom)
       at Pulumi.Docker.Docker.BuildAndPushImageWorkerAsync(String baseImageName, Union`2 pathOrBuild, String repositoryUrl, Resource logResource, Boolean skipPush, ImageRegistryUnwrap registry)
       at Pulumi.Docker.Docker.<>c__DisplayClass0_0.<<BuildAndPushImageAsync>b__0>d.MoveNext()

Before that, we have image built successfully, but having there this error too:

docker:image:Image myimage  Successfully built 09ab25e0f4af
docker:image:Image myimage  Successfully tagged myimage:0.9.5
docker:image:Image myimage  Executing ' docker image inspect -f {{.Id}} myimage:0.9.5'
docker:image:Image myimage  error: Pulumi.ResourceException: No digest available for image myimage:0.9.5

What does this exception mean, and how can we fix it?
Thanks!

@leezen
Copy link
Contributor

leezen commented Jul 16, 2020

That error shows up because the execution of docker image inspect -f {{.Id}} myimage:0.9.5 failed. It'd be great if you're able to try debugging why the command doesn't succeed to make sure it's not a bug in the library vs. something local to your configuration.

@IgorFedchenko
Copy link
Author

Thanks, @leezen , I will check this out.

@Aaronontheweb
Copy link

I'm running into this too - it's intermittent. Disabling building of local images doesn't appear to affect this.

@Aaronontheweb
Copy link

We tried to see if this was a concurrency error with building multiple images in parallel - but it's not. Still occurs even if the images are forced to be built sequentially.

It looks like a race condition with the digest not being available immediately after the image is built - why is the digest needed in the first place?

@Aaronontheweb
Copy link

So I figured this out based on our debug logs (which I have anonymized somewhat here)

The problem is that as we're building multiple images in parallel, Pulumi attempts to compute the digest for one image before it finishes being built:

2020-09-24T00:07:50.5885535Z     pulumi:pulumi:Stack cluster-config-dev running 'dotnet build -nologo .' completed successfully
2020-09-24T00:07:50.6013433Z     pulumi:pulumi:Stack cluster-config-dev running debug: Fields to assign: ["urn","UserServiceDns","mediaServiceDns"]
2020-09-24T00:07:50.6035934Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=pulumi:pulumi:Stack, name=cluster-config-dev, custom=False
2020-09-24T00:07:50.6041539Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=pulumi:pulumi:Stack, name=cluster-config-dev, custom=False
2020-09-24T00:07:50.6047164Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=pulumi:pulumi:Stack, name=cluster-config-dev, custom=False
2020-09-24T00:07:50.6052236Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=pulumi:pulumi:Stack, name=cluster-config-dev, custom=False
2020-09-24T00:07:50.6057119Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=pulumi:pulumi:Stack, name=cluster-config-dev, custom=False
2020-09-24T00:07:50.6495605Z     pulumi:pulumi:Stack cluster-config-dev running read pulumi:pulumi:StackReference ourcompany/sampleapp-azure/dev
2020-09-24T00:07:50.7055548Z     kubernetes:yaml:ConfigFile seed-config-dev  
2020-09-24T00:07:50.8474966Z     docker:image:Image sampleapp.common.migrations  
2020-09-24T00:07:50.8476536Z     docker:image:Image sampleapp.mediaservice.webapi  
2020-09-24T00:07:50.8477445Z     docker:image:Image sampleapp.userservice.webapi  
2020-09-24T00:07:51.2898332Z     pulumi:pulumi:Stack cluster-config-dev running read pulumi:pulumi:StackReference ourcompany/sampleapp-azure/dev
2020-09-24T00:07:51.2920269Z     pulumi:pulumi:StackReference ourcompany/sampleapp-azure/dev  debug: Fields to assign: ["urn","name","id","secretOutputNames","outputs"]
2020-09-24T00:07:51.2964828Z     pulumi:pulumi:Stack cluster-config-dev running debug: Reading resource: id=ourcompany/sampleapp-azure/dev, t=$pulumi:pulumi:StackReference, name=$ourcompany/sampleapp-azure/dev
2020-09-24T00:07:51.2966743Z     pulumi:pulumi:Stack cluster-config-dev running debug: ReadResource RPC prepared: id=ourcompany/sampleapp-azure/dev, t=pulumi:pulumi:StackReference, name=ourcompany/sampleapp-azure/dev
2020-09-24T00:07:51.3780070Z     pulumi:providers:kubernetes k8s  
2020-09-24T00:07:53.3853273Z @ Previewing update....
2020-09-24T00:07:53.3854259Z     pulumi:providers:kubernetes k8s  debug: Fields to assign: ["urn","id"]
2020-09-24T00:07:53.3892196Z     kubernetes:core:ConfigMap aspnet-configs  
2020-09-24T00:07:53.4051689Z     kubernetes:core:ConfigMap seed-configs  
2020-09-24T00:07:53.4125215Z     kubernetes:core:ConfigMap email-configs  
2020-09-24T00:07:53.4168684Z     kubernetes:core:Secret sampleapp-sendgrid  
2020-09-24T00:07:53.4195699Z     kubernetes:core:ConfigMap environment-configs  
2020-09-24T00:07:54.3770788Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=pulumi:providers:kubernetes, name=k8s, custom=True
2020-09-24T00:07:54.7539612Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=pulumi:providers:kubernetes, name=k8s, custom=True
2020-09-24T00:07:54.7812349Z     kubernetes:core:ConfigMap aspnet-configs  debug: Fields to assign: ["apiVersion","immutable","urn","id","data","metadata","kind","binaryData"]
2020-09-24T00:07:57.3807918Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:ConfigMap, name=aspnet-configs, custom=True
2020-09-24T00:07:58.3787808Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:ConfigMap, name=aspnet-configs, custom=True
2020-09-24T00:07:58.3808296Z     kubernetes:yaml:ConfigFile seed-config-dev  debug: Fields to assign: ["urn"]
2020-09-24T00:07:58.3823199Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:yaml:ConfigFile, name=seed-config-dev, custom=False
2020-09-24T00:07:58.3856888Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:yaml:ConfigFile, name=seed-config-dev, custom=False
2020-09-24T00:07:58.3873493Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=kubernetes:yaml:ConfigFile, name=seed-config-dev, custom=False
2020-09-24T00:07:58.4007231Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=kubernetes:yaml:ConfigFile, name=seed-config-dev, custom=False
2020-09-24T00:07:58.4026832Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=kubernetes:yaml:ConfigFile, name=seed-config-dev, custom=False
2020-09-24T00:07:58.4034052Z     kubernetes:core:ConfigMap email-configs  debug: Fields to assign: ["apiVersion","immutable","urn","id","data","metadata","kind","binaryData"]
2020-09-24T00:07:58.4077742Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:ConfigMap, name=email-configs, custom=True
2020-09-24T00:07:58.4155023Z     pulumi:pulumi:Stack cluster-config-dev running debug: Invoking function: token=kubernetes:yaml:decode asynchronously
2020-09-24T00:07:58.4171965Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:ConfigMap, name=email-configs, custom=True
2020-09-24T00:07:58.4223295Z     pulumi:pulumi:Stack cluster-config-dev running debug: Invoke RPC prepared: token=kubernetes:yaml:decode
2020-09-24T00:07:58.4224633Z     kubernetes:core:ConfigMap environment-configs  debug: Fields to assign: ["apiVersion","immutable","urn","id","data","metadata","kind","binaryData"]
2020-09-24T00:07:58.4252896Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:ConfigMap, name=environment-configs, custom=True
2020-09-24T00:07:58.4254610Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:ConfigMap, name=environment-configs, custom=True
2020-09-24T00:07:58.4256171Z     kubernetes:core:Secret sampleapp-sendgrid  debug: Fields to assign: ["apiVersion","immutable","urn","id","data","stringData","metadata","type","kind"]
2020-09-24T00:07:58.4257745Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:Secret, name=sampleapp-sendgrid, custom=True
2020-09-24T00:07:58.4259155Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:Secret, name=sampleapp-sendgrid, custom=True
2020-09-24T00:07:58.4380489Z     docker:image:Image sampleapp.common.migrations  debug: Fields to assign: ["urn","baseImageName","registryServer","imageName"]
2020-09-24T00:07:58.4382410Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=docker:image:Image, name=sampleapp.common.migrations, custom=False
2020-09-24T00:07:58.4385410Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=docker:image:Image, name=sampleapp.common.migrations, custom=False
2020-09-24T00:07:58.4389967Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=docker:image:Image, name=sampleapp.common.migrations, custom=False
2020-09-24T00:07:58.4395110Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=docker:image:Image, name=sampleapp.common.migrations, custom=False
2020-09-24T00:07:58.4400178Z     docker:image:Image sampleapp.mediaservice.webapi  debug: Fields to assign: ["urn","baseImageName","registryServer","imageName"]
2020-09-24T00:07:58.4404080Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=docker:image:Image, name=sampleapp.mediaservice.webapi, custom=False
2020-09-24T00:07:58.4410800Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=docker:image:Image, name=sampleapp.mediaservice.webapi, custom=False
2020-09-24T00:07:58.4415817Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=docker:image:Image, name=sampleapp.mediaservice.webapi, custom=False
2020-09-24T00:07:58.4438743Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=docker:image:Image, name=sampleapp.mediaservice.webapi, custom=False
2020-09-24T00:07:58.4447702Z     docker:image:Image sampleapp.userservice.webapi  debug: Fields to assign: ["urn","baseImageName","registryServer","imageName"]
2020-09-24T00:07:58.4453364Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=docker:image:Image, name=sampleapp.userservice.webapi, custom=False
2020-09-24T00:07:58.4459145Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=docker:image:Image, name=sampleapp.userservice.webapi, custom=False
2020-09-24T00:07:58.4460514Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=docker:image:Image, name=sampleapp.userservice.webapi, custom=False
2020-09-24T00:07:58.4461764Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=docker:image:Image, name=sampleapp.userservice.webapi, custom=False
2020-09-24T00:08:23.8520616Z @ Previewing update....
2020-09-24T00:08:23.8524345Z     pulumi:pulumi:Stack cluster-config-dev running debug: Fields to assign: ["apiVersion","urn","spec","status","id","metadata","kind"]
2020-09-24T00:08:23.8558512Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:apps/v1:Deployment, name=user-service, custom=True
2020-09-24T00:08:23.8565173Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:apps/v1:Deployment, name=user-service, custom=True
2020-09-24T00:08:23.8571536Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=docker:image:Image, name=sampleapp.common.migrations, custom=False
2020-09-24T00:08:23.8577720Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=docker:image:Image, name=sampleapp.mediaservice.webapi, custom=False
2020-09-24T00:08:23.8584371Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=docker:image:Image, name=sampleapp.userservice.webapi, custom=False
2020-09-24T00:08:23.8592754Z     pulumi:pulumi:Stack cluster-config-dev running debug: Fields to assign: ["apiVersion","urn","spec","status","id","metadata","kind"]
2020-09-24T00:08:23.8598726Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:Service, name=user-service-private, custom=True
2020-09-24T00:08:23.8604812Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:Service, name=user-service-private, custom=True
2020-09-24T00:08:23.8615565Z     pulumi:pulumi:Stack cluster-config-dev running debug: Fields to assign: ["apiVersion","urn","spec","status","id","metadata","kind"]
2020-09-24T00:08:23.8628378Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:apps/v1:Deployment, name=media-service, custom=True
2020-09-24T00:08:23.8631207Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:apps/v1:Deployment, name=media-service, custom=True
2020-09-24T00:08:23.8639637Z     pulumi:pulumi:Stack cluster-config-dev running debug: Fields to assign: ["apiVersion","urn","spec","status","id","metadata","kind"]
2020-09-24T00:08:23.8644069Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:Service, name=media-service-private, custom=True
2020-09-24T00:08:23.8649928Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:Service, name=media-service-private, custom=True
2020-09-24T00:08:23.8655627Z     pulumi:pulumi:Stack cluster-config-dev running debug: Fields to assign: ["apiVersion","urn","spec","status","id","metadata","kind"]
2020-09-24T00:08:23.8660048Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:networking.k8s.io/v1beta1:Ingress, name=appgateway-ingress, custom=True
2020-09-24T00:08:23.8664725Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:networking.k8s.io/v1beta1:Ingress, name=appgateway-ingress, custom=True
2020-09-24T00:08:23.8669210Z     kubernetes:core:ConfigMap seed-configs  debug: Fields to assign: ["apiVersion","immutable","urn","id","data","metadata","kind","binaryData"]
2020-09-24T00:08:23.8674040Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource start: t=kubernetes:core/v1:ConfigMap, name=seed-configs, custom=True
2020-09-24T00:08:23.8677707Z     pulumi:pulumi:Stack cluster-config-dev running debug: Preparing resource: t=kubernetes:core/v1:ConfigMap, name=seed-configs, custom=True
2020-09-24T00:08:23.8684983Z     pulumi:pulumi:Stack cluster-config-dev running debug: RegisterResourceOutputs RPC prepared: urn=urn:pulumi:dev::cluster-config::pulumi:pulumi:Stack::cluster-config-dev
2020-09-24T00:08:23.8686674Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=pulumi:providers:kubernetes, name=k8s, custom=True
2020-09-24T00:08:23.8690744Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=pulumi:providers:kubernetes, name=k8s, custom=True
2020-09-24T00:08:23.8695014Z     docker:image:Image sampleapp.common.migrations  Starting docker build and push...
2020-09-24T00:08:23.8701302Z     docker:image:Image sampleapp.userservice.webapi  Starting docker build and push...
2020-09-24T00:08:23.8705890Z     docker:image:Image sampleapp.mediaservice.webapi  Starting docker build and push...
2020-09-24T00:08:23.8707702Z     docker:image:Image sampleapp.common.migrations  Building container image 'sampleappdevimagese754a4e1.azurecr.io/sampleapp.common.migrations:0.9.6': context=/home/vsts/work/1/s/src/sampleapp.Common.Migrations
2020-09-24T00:08:23.8714642Z     docker:image:Image sampleapp.mediaservice.webapi  Building container image 'sampleappdevimagese754a4e1.azurecr.io/sampleapp.mediaservice.webapi:0.9.6': context=/home/vsts/work/1/s/src/mediaService/sampleapp.mediaService.WebApi
2020-09-24T00:08:23.8719642Z     docker:image:Image sampleapp.userservice.webapi  Building container image 'sampleappdevimagese754a4e1.azurecr.io/sampleapp.userservice.webapi:0.9.6': context=/home/vsts/work/1/s/src/UserService/sampleapp.UserService.WebApi
2020-09-24T00:08:23.8727144Z     docker:image:Image sampleapp.userservice.webapi  Executing ' docker build /home/vsts/work/1/s/src/UserService/sampleapp.UserService.WebApi -t sampleappdevimagese754a4e1.azurecr.io/sampleapp.userservice.webapi:0.9.6'
2020-09-24T00:08:23.8731258Z     docker:image:Image sampleapp.common.migrations  Executing ' docker build /home/vsts/work/1/s/src/sampleapp.Common.Migrations -t sampleappdevimagese754a4e1.azurecr.io/sampleapp.common.migrations:0.9.6'
2020-09-24T00:08:23.8735682Z     docker:image:Image sampleapp.mediaservice.webapi  Executing ' docker build /home/vsts/work/1/s/src/mediaService/sampleapp.mediaService.WebApi -t sampleappdevimagese754a4e1.azurecr.io/sampleapp.mediaservice.webapi:0.9.6'
2020-09-24T00:08:23.8740432Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=pulumi:providers:kubernetes, name=k8s, custom=True
2020-09-24T00:08:23.8749956Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=kubernetes:core/v1:ConfigMap, name=aspnet-configs, custom=True
2020-09-24T00:08:23.8757951Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=kubernetes:core/v1:ConfigMap, name=aspnet-configs, custom=True
2020-09-24T00:08:23.8769373Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=kubernetes:core/v1:ConfigMap, name=seed-configs, custom=True
2020-09-24T00:08:23.8772642Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=kubernetes:core/v1:ConfigMap, name=seed-configs, custom=True
2020-09-24T00:08:23.8776452Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=kubernetes:core/v1:Secret, name=sampleapp-sendgrid, custom=True
2020-09-24T00:08:23.8777946Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=kubernetes:core/v1:Secret, name=sampleapp-sendgrid, custom=True
2020-09-24T00:08:23.8779461Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=kubernetes:core/v1:ConfigMap, name=environment-configs, custom=True
2020-09-24T00:08:23.8780831Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=kubernetes:core/v1:ConfigMap, name=environment-configs, custom=True
2020-09-24T00:08:23.8794001Z     pulumi:pulumi:Stack cluster-config-dev running debug: Prepared resource: t=kubernetes:core/v1:ConfigMap, name=email-configs, custom=True
2020-09-24T00:08:23.8795931Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor start: t=kubernetes:core/v1:ConfigMap, name=email-configs, custom=True
2020-09-24T00:08:23.8797074Z     docker:image:Image sampleapp.common.migrations  Sending build context to Docker daemon  32.64MB
2020-09-24T00:08:23.8797925Z     docker:image:Image sampleapp.common.migrations  
2020-09-24T00:08:23.8798863Z     docker:image:Image sampleapp.common.migrations  Step 1/4 : FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS base
2020-09-24T00:08:23.8800526Z     docker:image:Image sampleapp.common.migrations  3.1: Pulling from dotnet/core/sdk
2020-09-24T00:08:23.8816356Z     docker:image:Image sampleapp.common.migrations  57df1a1f1ad8: Already exists
2020-09-24T00:08:23.8819851Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=kubernetes:core/v1:ConfigMap, name=aspnet-configs, custom=True
2020-09-24T00:08:23.8821597Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=kubernetes:core/v1:ConfigMap, name=seed-configs, custom=True
2020-09-24T00:08:23.8827549Z     pulumi:pulumi:Stack cluster-config-dev running debug: RegisterResourceOutputs RPC prepared: urn=urn:pulumi:dev::cluster-config::kubernetes:yaml:ConfigFile::seed-config-dev
2020-09-24T00:08:23.8831018Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=kubernetes:core/v1:ConfigMap, name=email-configs, custom=True
2020-09-24T00:08:23.8834691Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=kubernetes:core/v1:Secret, name=sampleapp-sendgrid, custom=True
2020-09-24T00:08:23.8838106Z     pulumi:pulumi:Stack cluster-config-dev running debug: Registering resource monitor end: t=kubernetes:core/v1:ConfigMap, name=environment-configs, custom=True
2020-09-24T00:08:23.8839222Z     docker:image:Image sampleapp.mediaservice.webapi  Sending build context to Docker daemon  129.6MB
2020-09-24T00:08:23.8844890Z     docker:image:Image sampleapp.mediaservice.webapi  
2020-09-24T00:08:23.8845714Z     docker:image:Image sampleapp.mediaservice.webapi  Step 1/5 : FROM mcr.microsoft.com/dotnet/core/aspnet:3.1 AS base
2020-09-24T00:08:23.8846682Z     docker:image:Image sampleapp.mediaservice.webapi  3.1: Pulling from dotnet/core/aspnet
2020-09-24T00:08:23.8850154Z     docker:image:Image sampleapp.mediaservice.webapi  d121f8d1c412: Pulling fs layer
2020-09-24T00:08:23.8853714Z     docker:image:Image sampleapp.mediaservice.webapi  7b2d5a029506: Pulling fs layer
2020-09-24T00:08:23.8861532Z     docker:image:Image sampleapp.mediaservice.webapi  778ad5bbc2d1: Pulling fs layer
2020-09-24T00:08:23.8887034Z     docker:image:Image sampleapp.mediaservice.webapi  6de7a0eb2598: Pulling fs layer
2020-09-24T00:08:23.8894699Z     docker:image:Image sampleapp.mediaservice.webapi  6d9bf7184ec4: Pulling fs layer
2020-09-24T00:08:23.8903173Z     docker:image:Image sampleapp.mediaservice.webapi  6d9bf7184ec4: Waiting
2020-09-24T00:08:23.8913921Z     docker:image:Image sampleapp.mediaservice.webapi  6de7a0eb2598: Waiting
2020-09-24T00:08:23.8924266Z     docker:image:Image sampleapp.mediaservice.webapi  778ad5bbc2d1: Verifying Checksum
2020-09-24T00:08:23.8929544Z     docker:image:Image sampleapp.mediaservice.webapi  778ad5bbc2d1: Download complete
2020-09-24T00:08:23.8932627Z     docker:image:Image sampleapp.common.migrations  71e126169501: Already exists
2020-09-24T00:08:23.8935738Z     docker:image:Image sampleapp.mediaservice.webapi  7b2d5a029506: Verifying Checksum
2020-09-24T00:08:23.8936765Z     docker:image:Image sampleapp.mediaservice.webapi  7b2d5a029506: Download complete
2020-09-24T00:08:23.8937652Z     docker:image:Image sampleapp.mediaservice.webapi  d121f8d1c412: Verifying Checksum
2020-09-24T00:08:23.8940405Z     docker:image:Image sampleapp.mediaservice.webapi  d121f8d1c412: Download complete
2020-09-24T00:08:23.8941314Z     docker:image:Image sampleapp.mediaservice.webapi  6de7a0eb2598: Verifying Checksum
2020-09-24T00:08:23.8942226Z     docker:image:Image sampleapp.mediaservice.webapi  6de7a0eb2598: Download complete
2020-09-24T00:08:23.8945181Z     docker:image:Image sampleapp.mediaservice.webapi  6d9bf7184ec4: Verifying Checksum
2020-09-24T00:08:23.8946074Z     docker:image:Image sampleapp.mediaservice.webapi  6d9bf7184ec4: Download complete
2020-09-24T00:08:23.8946931Z     docker:image:Image sampleapp.common.migrations  1af28a55c3f3: Already exists
2020-09-24T00:08:23.8947786Z     docker:image:Image sampleapp.common.migrations  03f1c9932170: Already exists
2020-09-24T00:08:23.8948657Z     docker:image:Image sampleapp.common.migrations  1e9f61add744: Pulling fs layer
2020-09-24T00:08:23.8951671Z     docker:image:Image sampleapp.common.migrations  8bc534dd6017: Pulling fs layer
2020-09-24T00:08:23.8954412Z     docker:image:Image sampleapp.common.migrations  7b0d6e95dc2c: Pulling fs layer
2020-09-24T00:08:23.8955440Z     docker:image:Image sampleapp.common.migrations  1e9f61add744: Verifying Checksum
2020-09-24T00:08:23.8956311Z     docker:image:Image sampleapp.common.migrations  1e9f61add744: Download complete
2020-09-24T00:08:23.8959033Z     docker:image:Image sampleapp.common.migrations  7b0d6e95dc2c: Verifying Checksum
2020-09-24T00:08:23.8959903Z     docker:image:Image sampleapp.common.migrations  7b0d6e95dc2c: Download complete
2020-09-24T00:08:23.8962877Z     docker:image:Image sampleapp.userservice.webapi  Sending build context to Docker daemon  155.1MB
2020-09-24T00:08:23.8963790Z     docker:image:Image sampleapp.userservice.webapi  
2020-09-24T00:08:23.8964769Z     docker:image:Image sampleapp.userservice.webapi  Step 1/5 : FROM mcr.microsoft.com/dotnet/core/aspnet:3.1 AS base
2020-09-24T00:08:23.8965733Z     docker:image:Image sampleapp.userservice.webapi  3.1: Pulling from dotnet/core/aspnet
2020-09-24T00:08:23.8968769Z     docker:image:Image sampleapp.userservice.webapi  d121f8d1c412: Pulling fs layer
2020-09-24T00:08:23.8969639Z     docker:image:Image sampleapp.userservice.webapi  7b2d5a029506: Pulling fs layer
2020-09-24T00:08:23.8970507Z     docker:image:Image sampleapp.userservice.webapi  778ad5bbc2d1: Pulling fs layer
2020-09-24T00:08:23.8973847Z     docker:image:Image sampleapp.userservice.webapi  6de7a0eb2598: Pulling fs layer
2020-09-24T00:08:23.8976602Z     docker:image:Image sampleapp.userservice.webapi  6d9bf7184ec4: Pulling fs layer
2020-09-24T00:08:23.8977530Z     docker:image:Image sampleapp.userservice.webapi  6de7a0eb2598: Waiting
2020-09-24T00:08:23.8981169Z     docker:image:Image sampleapp.userservice.webapi  6d9bf7184ec4: Waiting
2020-09-24T00:08:23.8992341Z     docker:image:Image sampleapp.userservice.webapi  778ad5bbc2d1: Verifying Checksum
2020-09-24T00:08:23.9000608Z     docker:image:Image sampleapp.userservice.webapi  778ad5bbc2d1: Download complete
2020-09-24T00:08:23.9004149Z     docker:image:Image sampleapp.userservice.webapi  7b2d5a029506: Verifying Checksum
2020-09-24T00:08:23.9024059Z     docker:image:Image sampleapp.userservice.webapi  7b2d5a029506: Download complete
2020-09-24T00:08:23.9051343Z     docker:image:Image sampleapp.userservice.webapi  d121f8d1c412: Verifying Checksum
2020-09-24T00:08:23.9056364Z     docker:image:Image sampleapp.userservice.webapi  d121f8d1c412: Download complete
2020-09-24T00:08:23.9078701Z     docker:image:Image sampleapp.userservice.webapi  6de7a0eb2598: Verifying Checksum
2020-09-24T00:08:23.9121966Z     docker:image:Image sampleapp.userservice.webapi  6de7a0eb2598: Download complete
2020-09-24T00:08:23.9125277Z     docker:image:Image sampleapp.userservice.webapi  6d9bf7184ec4: Verifying Checksum
2020-09-24T00:08:23.9131290Z     docker:image:Image sampleapp.userservice.webapi  6d9bf7184ec4: Download complete
2020-09-24T00:08:23.9132459Z     docker:image:Image sampleapp.common.migrations  8bc534dd6017: Verifying Checksum
2020-09-24T00:08:23.9135048Z     docker:image:Image sampleapp.common.migrations  8bc534dd6017: Download complete
2020-09-24T00:08:23.9135956Z     docker:image:Image sampleapp.common.migrations  1e9f61add744: Pull complete
2020-09-24T00:08:23.9138335Z     docker:image:Image sampleapp.userservice.webapi  d121f8d1c412: Pull complete
2020-09-24T00:08:23.9139276Z     docker:image:Image sampleapp.mediaservice.webapi  d121f8d1c412: Pull complete
2020-09-24T00:08:23.9141769Z     docker:image:Image sampleapp.userservice.webapi  7b2d5a029506: Pull complete
2020-09-24T00:08:23.9144473Z     docker:image:Image sampleapp.mediaservice.webapi  7b2d5a029506: Pull complete
2020-09-24T00:08:23.9147025Z     docker:image:Image sampleapp.mediaservice.webapi  778ad5bbc2d1: Pull complete
2020-09-24T00:08:23.9149403Z     docker:image:Image sampleapp.userservice.webapi  778ad5bbc2d1: Pull complete
2020-09-24T00:08:23.9151763Z     docker:image:Image sampleapp.common.migrations  8bc534dd6017: Pull complete
2020-09-24T00:08:23.9155592Z     docker:image:Image sampleapp.userservice.webapi  6de7a0eb2598: Pull complete
2020-09-24T00:08:23.9157953Z     docker:image:Image sampleapp.mediaservice.webapi  6de7a0eb2598: Pull complete
2020-09-24T00:08:23.9161731Z     docker:image:Image sampleapp.common.migrations  7b0d6e95dc2c: Pull complete
2020-09-24T00:08:23.9164632Z     docker:image:Image sampleapp.mediaservice.webapi  6d9bf7184ec4: Pull complete
2020-09-24T00:08:23.9165691Z     docker:image:Image sampleapp.userservice.webapi  6d9bf7184ec4: Pull complete
2020-09-24T00:08:23.9168302Z     docker:image:Image sampleapp.common.migrations  Digest: sha256:95f46b6614f7fb759cdad3b38286e6d9a25422113a07b5862dec1400379e796b
2020-09-24T00:08:23.9172727Z     docker:image:Image sampleapp.userservice.webapi  Digest: sha256:adeb306016cb11425faad95b59f138c58910023f5754caeecd815b403330c7a4
2020-09-24T00:08:23.9177126Z     docker:image:Image sampleapp.common.migrations  Status: Downloaded newer image for mcr.microsoft.com/dotnet/core/sdk:3.1
2020-09-24T00:08:23.9181126Z     docker:image:Image sampleapp.mediaservice.webapi  Digest: sha256:adeb306016cb11425faad95b59f138c58910023f5754caeecd815b403330c7a4
2020-09-24T00:08:23.9184159Z     docker:image:Image sampleapp.common.migrations   ---> c4155a9104a8
2020-09-24T00:08:23.9186411Z     docker:image:Image sampleapp.common.migrations  Step 2/4 : WORKDIR /app
2020-09-24T00:08:23.9188506Z     docker:image:Image sampleapp.mediaservice.webapi  Status: Downloaded newer image for mcr.microsoft.com/dotnet/core/aspnet:3.1
2020-09-24T00:08:23.9191024Z     docker:image:Image sampleapp.mediaservice.webapi   ---> 28de0d96c539
2020-09-24T00:08:23.9191836Z     docker:image:Image sampleapp.userservice.webapi  Status: Downloaded newer image for mcr.microsoft.com/dotnet/core/aspnet:3.1
2020-09-24T00:08:23.9194191Z     docker:image:Image sampleapp.userservice.webapi   ---> 28de0d96c539
2020-09-24T00:08:23.9196337Z     docker:image:Image sampleapp.userservice.webapi  Step 2/5 : WORKDIR /app
2020-09-24T00:08:23.9197300Z     docker:image:Image sampleapp.mediaservice.webapi  Step 2/5 : WORKDIR /app
2020-09-24T00:08:23.9198280Z     docker:image:Image sampleapp.common.migrations   ---> Running in dab4f3591cb6
2020-09-24T00:08:23.9200146Z     docker:image:Image sampleapp.userservice.webapi   ---> Running in 828de1971710
2020-09-24T00:08:23.9201876Z     docker:image:Image sampleapp.mediaservice.webapi   ---> Running in 4ec2e720e6a5
2020-09-24T00:08:23.9202805Z     docker:image:Image sampleapp.mediaservice.webapi  Removing intermediate container 4ec2e720e6a5
2020-09-24T00:08:23.9204036Z     docker:image:Image sampleapp.mediaservice.webapi   ---> 340bccda30f7
2020-09-24T00:08:23.9216295Z     docker:image:Image sampleapp.mediaservice.webapi  Step 3/5 : EXPOSE 15550
2020-09-24T00:08:23.9217009Z     docker:image:Image sampleapp.userservice.webapi  Removing intermediate container 828de1971710
2020-09-24T00:08:23.9218000Z     docker:image:Image sampleapp.userservice.webapi   ---> bd53ff10f041
2020-09-24T00:08:23.9218747Z     docker:image:Image sampleapp.userservice.webapi  Step 3/5 : EXPOSE 15550
2020-09-24T00:08:23.9219579Z     docker:image:Image sampleapp.mediaservice.webapi   ---> Running in cef329f9307b
2020-09-24T00:08:23.9220419Z     docker:image:Image sampleapp.userservice.webapi   ---> Running in 4b41822c5263
2020-09-24T00:08:23.9221062Z     docker:image:Image sampleapp.common.migrations  Removing intermediate container dab4f3591cb6
2020-09-24T00:08:23.9221946Z     docker:image:Image sampleapp.common.migrations   ---> 03d1a932da1e
2020-09-24T00:08:23.9222649Z     docker:image:Image sampleapp.common.migrations  Step 3/4 : COPY ./bin/Release/netcoreapp3.1/publish /app
2020-09-24T00:08:23.9223338Z     docker:image:Image sampleapp.userservice.webapi  Removing intermediate container 4b41822c5263
2020-09-24T00:08:23.9224162Z     docker:image:Image sampleapp.userservice.webapi   ---> 362cdc88929b
2020-09-24T00:08:23.9224820Z     docker:image:Image sampleapp.userservice.webapi  Step 4/5 : COPY ./bin/Release/netcoreapp3.1/publish /app
2020-09-24T00:08:23.9225765Z     docker:image:Image sampleapp.mediaservice.webapi  Removing intermediate container cef329f9307b
2020-09-24T00:08:23.9226646Z     docker:image:Image sampleapp.mediaservice.webapi   ---> 0fcb27e2bcd4
2020-09-24T00:08:23.9227356Z     docker:image:Image sampleapp.mediaservice.webapi  Step 4/5 : COPY ./bin/Release/netcoreapp3.1/publish /app
2020-09-24T00:08:23.9228287Z     docker:image:Image sampleapp.common.migrations   ---> eca44a89d66a
2020-09-24T00:08:23.9229127Z     docker:image:Image sampleapp.mediaservice.webapi   ---> 2dd8184281ef
2020-09-24T00:08:23.9229847Z     docker:image:Image sampleapp.mediaservice.webapi  Step 5/5 : ENTRYPOINT ["dotnet", "sampleapp.mediaService.WebApi.dll"]
2020-09-24T00:08:23.9230998Z     docker:image:Image sampleapp.userservice.webapi   ---> ac9ad22f3c93
2020-09-24T00:08:23.9231934Z     docker:image:Image sampleapp.userservice.webapi  Step 5/5 : ENTRYPOINT ["dotnet", "sampleapp.UserService.WebApi.dll"]
2020-09-24T00:08:23.9232929Z     docker:image:Image sampleapp.mediaservice.webapi   ---> Running in f20d295184c4
2020-09-24T00:08:23.9233641Z     docker:image:Image sampleapp.mediaservice.webapi  Removing intermediate container f20d295184c4
2020-09-24T00:08:23.9234559Z     docker:image:Image sampleapp.mediaservice.webapi   ---> 9f146a9ed8e3
2020-09-24T00:08:23.9235216Z     docker:image:Image sampleapp.mediaservice.webapi  Successfully built 9f146a9ed8e3
2020-09-24T00:08:23.9236002Z     docker:image:Image sampleapp.mediaservice.webapi  Successfully tagged sampleappdevimagese754a4e1.azurecr.io/sampleapp.mediaservice.webapi:0.9.6
2020-09-24T00:08:23.9236845Z     docker:image:Image sampleapp.common.migrations  Step 4/4 : ENTRYPOINT ["dotnet", "sampleapp.Common.Migrations.dll"]
2020-09-24T00:08:23.9237794Z     docker:image:Image sampleapp.common.migrations   ---> Running in e5bd0f0a3525
2020-09-24T00:08:23.9238967Z     docker:image:Image sampleapp.mediaservice.webapi  Executing ' docker image inspect -f {{.Id}} sampleappdevimagese754a4e1.azurecr.io/sampleapp.mediaservice.webapi:0.9.6'
2020-09-24T00:08:23.9240338Z     docker:image:Image sampleapp.userservice.webapi  Executing ' docker image inspect -f {{.Id}} sampleappdevimagese754a4e1.azurecr.io/sampleapp.userservice.webapi:0.9.6'
2020-09-24T00:08:23.9241316Z     docker:image:Image sampleapp.common.migrations  Removing intermediate container e5bd0f0a3525
2020-09-24T00:08:23.9242207Z     docker:image:Image sampleapp.common.migrations   ---> db90f312709e
2020-09-24T00:08:23.9243044Z     docker:image:Image sampleapp.common.migrations  Successfully built db90f312709e
2020-09-24T00:08:23.9244113Z     docker:image:Image sampleapp.common.migrations  Successfully tagged sampleappdevimagese754a4e1.azurecr.io/sampleapp.common.migrations:0.9.6
2020-09-24T00:08:23.9245316Z     docker:image:Image sampleapp.mediaservice.webapi  sha256:9f146a9ed8e3f090f2737ac5c1eb9776a0677a6ac8b772b987766a7d52478958
2020-09-24T00:08:23.9246692Z     docker:image:Image sampleapp.mediaservice.webapi  successfully pushed to docker
2020-09-24T00:08:23.9249592Z     pulumi:pulumi:Stack cluster-config-dev running debug: RegisterResourceOutputs RPC prepared: urn=urn:pulumi:dev::cluster-config::docker:image:Image::sampleapp.mediaservice.webapi
2020-09-24T00:08:23.9251319Z     docker:image:Image sampleapp.common.migrations  Executing ' docker image inspect -f {{.Id}} sampleappdevimagese754a4e1.azurecr.io/sampleapp.common.migrations:0.9.6'
2020-09-24T00:08:23.9252826Z     docker:image:Image sampleapp.userservice.webapi  error: Pulumi.ResourceException: No digest available for image sampleappdevimagese754a4e1.azurecr.io/sampleapp.userservice.webapi:0.9.6
2020-09-24T00:08:23.9253898Z     docker:image:Image sampleapp.common.migrations  sha256:db90f312709e6c028b410a4be7d3e8f48b53b50bbb1be550aef66a188d94f7c0
2020-09-24T00:08:23.9254658Z     docker:image:Image sampleapp.common.migrations  successfully pushed to docker
2020-09-24T00:08:23.9434581Z     pulumi:pulumi:Stack cluster-config-dev  78 debugs
2020-09-24T00:08:23.9435589Z     pulumi:pulumi:StackReference ourcompany/sampleapp-azure/dev  1 debug

sampleapp.mediaservice.webapi and sampleapp.common.migrations are both successfully tagged by Pulumi before docker image inspect is called on the other.

This is not the case for sampleapp.userservice.webapi - this image isn't finished being built; there is no Successfully tagged or Successfully built message from Docker prior to the docker image inspect call is made on it.

Are there any work-arounds for this in lieu of a fix?

@Aaronontheweb
Copy link

cc @leezen on the above

@leezen
Copy link
Contributor

leezen commented Sep 24, 2020

@Aaronontheweb Thanks for the extensive debugging output. We'll have to take a deeper look at this. At a cursory glance, I wouldn't expect the inspect invocation to happen until after the build completes (see https://github.com/pulumi/pulumi-docker/blob/master/sdk/dotnet/Docker.cs#L365).

@leezen leezen added the bug label Sep 24, 2020
@Aaronontheweb
Copy link

could it be an issue with building multiple images in parallel?

@jkrusic19
Copy link

Hello,

We are also experiencing this issue. We are trying to build 3 docker images for use in AWS Fargate. We are running pulumi 2.10.1, pulumi docker 2.2.3, pulumi aws 2.11.0, pulumi 2.5.0. Language C#.

Frequency: Once we start getting the error, it happens every single execution for a bit after. Different Images however, produce the error

if (fargateargs.DockerEnvironmentVariables != null)
            {
                Console.WriteLine("[INFO] Building DockerImage with Build Args on Dockerbuild cmd");
                image = new Image(
                    $"{fargateargs.environment}-{fargateargs.service}",
                    new ImageArgs {
                        Build = new DockerBuild {
                            Dockerfile = fargateargs.DockerFile,
                            Context = fargateargs.DockerFileContext,
                            Args = fargateargs.DockerEnvironmentVariables
                        },
                        ImageName = repo.RepositoryUrl,
                        Registry = new ImageRegistry {
                            Server = repo.RepositoryUrl,
                            Username = credentials.GetAt( 0 ),
                            Password = credentials.GetAt( 1 )
                        }
                    }
                );
            }
            else
            {
                image = new Image(
                    $"{fargateargs.environment}-{fargateargs.service}",
                    new ImageArgs {
                        Build = new DockerBuild {
                            Dockerfile = fargateargs.DockerFile,
                            Context = fargateargs.DockerFileContext
                        },
                        ImageName = repo.RepositoryUrl,
                        Registry = new ImageRegistry {
                            Server = repo.RepositoryUrl,
                            Username = credentials.GetAt( 0 ),
                            Password = credentials.GetAt( 1 )
                        }
                    }
                );
            }

@dbeattie71
Copy link

I noticed Process.Start() is called before the Process.Exit event is registered in the RunCommandThatCanFail method. The event is registered in WaitForExitAsync. Could that potentially cause problems?

@mikhailshilkov
Copy link
Member

It took a while to reproduce this issue locally, but I got it now. However, based on the logs, the build process has successfully exited before the inspect process started. Here are the relevant lines (I added some extra logging locally):

I1001 14:17:41.909706   33275 eventsink.go:70] eventSink::Info(<{%reset%}>Successfully built 709d4eda98f7<{%reset%}>)
    docker:image:Image test0  Successfully built 709d4eda98f7
...
I1001 14:17:41.919887   33275 eventsink.go:70] eventSink::Info(<{%reset%}>Successfully tagged registryboosa.azurecr.io/test0:v1.0.5<{%reset%}>)
    docker:image:Image test0  Successfully tagged registryboosa.azurecr.io/test0:v1.0.5
...
I1001 14:17:41.930749   33275 eventsink.go:70] eventSink::Info(<{%reset%}>Process 33318 build ./node-app -t registryboosa.azurecr.io/test0:v1.0.5 exited<{%reset%}>)
    pulumi:pulumi:Stack azure-nextgen-cs-dev running Process 33318 build ./node-app -t registryboosa.azurecr.io/test0:v1.0.5 exited
I1001 14:17:41.931767   33275 eventsink.go:70] eventSink::Info(<{%reset%}>Executing ' docker image inspect -f {{.Id}} registryboosa.azurecr.io/test0:v1.0.5'<{%reset%}>)
    docker:image:Image test0  Executing ' docker image inspect -f {{.Id}} registryboosa.azurecr.io/test0:v1.0.5'
...
I1001 14:17:42.049298   33275 eventsink.go:70] eventSink::Info(<{%reset%}>Process! 35679 image inspect -f {{.Id}} registryboosa.azurecr.io/test0:v1.0.5 exited<{%reset%}>)
    pulumi:pulumi:Stack azure-nextgen-cs-dev running Process 35679 image inspect -f {{.Id}} registryboosa.azurecr.io/test0:v1.0.5 exited
...
I1001 14:17:42.057613   33275 eventsink.go:86] eventSink::Error(<{%reset%}>Pulumi.ResourceException: No digest available for image registryboosa.azurecr.io/test0:v1.0.5
   at Pulumi.Docker.Docker.BuildImageAsync(String imageName, Union`2 pathOrBuild, Resource logResource, String[] cacheFrom) in /Users/mikhailshilkov/go/src/github.com/pulumi/pulumi-docker/sdk/dotnet/Docker.cs:line 383
   at Pulumi.Docker.Docker.BuildAndPushImageWorkerAsync(String baseImageName, Union`2 pathOrBuild, String repositoryUrl, Resource logResource, Boolean skipPush, ImageRegistryUnwrap registry) in /Users/mikhailshilkov/go/src/github.com/pulumi/pulumi-docker/sdk/dotnet/Docker.cs:line 233
   at Pulumi.Docker.Docker.<>c__DisplayClass0_0.<<BuildAndPushImageAsync>b__0>d.MoveNext() in /Users/mikhailshilkov/go/src/github.com/pulumi/pulumi-docker/sdk/dotnet/Docker.cs:line 146
--- End of stack trace from previous location where exception was thrown ---
   at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
   at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
   at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
   at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop)
   at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey)
   at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args)
   at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
   at Pulumi.Deployment.Runner.<>c__DisplayClass7_0.<<WhileRunningAsync>g__HandleCompletion|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Pulumi.Deployment.Runner.WhileRunningAsync()<{%reset%}>)
    docker:image:Image test0  error: Pulumi.ResourceException: No digest available for image registryboosa.azurecr.io/test0:v1.0.5

I'm deploying 30 images at the same time, so there is a bunch of chatter in between where ... are placed.

The sequence of log statements makes sense and yet the error happens. I'll keep digging.

@Aaronontheweb
Copy link

Thanks for looking into this @mikhailshilkov - really appreciate it

@mikhailshilkov
Copy link
Member

It turns out that we are missing stdout content due to a race condition, which leads to inspect not reporting the hash. This repro yielded the error reliably on my machine and this PR fixed it for me.

Reviews are welcome!

@mikhailshilkov
Copy link
Member

I noticed Process.Start() is called before the Process.Exit event is registered in the RunCommandThatCanFail method. The event is registered in WaitForExitAsync. Could that potentially cause problems?

That's a good point, but probably not related to this particular issue. I think I'll fix that in the same PR while I'm making a change.

@IgorFedchenko
Copy link
Author

@mikhailshilkov Thanks! Is it going to be released anytime soon?

@mikhailshilkov
Copy link
Member

You could give it a try right now if you are okay with using an alpha version for a little while: https://www.nuget.org/packages/Pulumi.Docker/2.4.0-alpha.1601877083

@jkrusic19
Copy link

@mikhailshilkov Been using it for majority of the day today and works as expected. Thanks for the quick turn-around look forward to the next prod release to update the dependancies.

@Aaronontheweb
Copy link

We're using it as well and I believe the issue has been resolved for us

@stack72
Copy link
Contributor

stack72 commented Oct 7, 2020

Hi all

I have just made v2.3.1 release that includes this bug fix

Paul

@leezen leezen modified the milestones: current, 0.46 Nov 2, 2020
@infin8x infin8x added the kind/bug Some behavior is incorrect or out of spec label Jul 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Development

No branches or pull requests

8 participants