Linux App Service: ERROR - failed to register layer: Error processing tar file(exit status 1): Container ID 1210765499 cannot be mapped to a host ID

[GuyHarwood]

Repro steps.

1. Publish docker image to azure container registry
2. Create Linux App Service
3. Set docker image to private registry
4. insert registry and image details
5. Browse to app URL

eventually receive the following webpage after a long wait...

503 Service Temporarily Unavailable

nginx

Image builds and runs fine locally (either in a swarm or standalone).

If i use az webapp log tail -g myresgroup -n myappsvc i get the following output...

2017-08-02 16:40:55.815 INFO  - latest: Pulling from admin
be666fb3dd35: Already exists
83be8b767650: Already exists
881a76cd4924: Already exists
984e4aef83ee: Already exists
9ac4c8a58ddd: Already exists
252f1f7248eb: Already exists
30a9f1ff73ff: Already exists
cda5a28e5c04: Already exists
d396e6e8e047: Pulling fs layer
2edae4c6ea2e: Pulling fs layer
46812c39b370: Pulling fs layer
8cac5df2cc39: Pulling fs layer
e95b2ad0bdc1: Pulling fs layer
74847f6a31cf: Pulling fs layer
6649110e4087: Pulling fs layer
e95b2ad0bdc1: Waiting
74847f6a31cf: Waiting
6649110e4087: Waiting
8cac5df2cc39: Waiting
d396e6e8e047: Download complete
d396e6e8e047: Pull complete
2edae4c6ea2e: Download complete
2edae4c6ea2e: Pull complete
8cac5df2cc39: Download complete
e95b2ad0bdc1: Verifying Checksum
e95b2ad0bdc1: Download complete
6649110e4087: Verifying Checksum
6649110e4087: Download complete
74847f6a31cf: Verifying Checksum
74847f6a31cf: Download complete
46812c39b370: Verifying Checksum
46812c39b370: Download complete

2017-08-02 16:40:55.816 ERROR - failed to register layer: Error processing tar file(exit status 1): Container ID XXXXXXX cannot be mapped to a host ID

I have tried standing up a completely new Linux App Service Plan but this makes no difference.
Have also checked disk space, and there is plenty.

If i use a sample image from docker hub, such as training/webapp it works fine.

How can i debug this further without any more detailed diagnostics info? and what could be causing this to fail?

[naziml]

This happens when the docker image uses a high-UID because we have userns mapping enabled for docker. You can repro this issue even locally if you enable userns mapping. What kind of image do you hit this for? We have seen this for node 8.x images that use npm 5 that hit this issue. I believe their latest node 8x image that uses the latest npm5x version doesn't, but every previous npm 5x version did.

[GuyHarwood]

interesting. Yes, we use node 8.1.2 and the base image is node:latest

I will repro locally, and look into a workaround.

Thank you for the info.

[GuyHarwood]

So, using node:alpine instead of node:latest worked. 🥇

One final thing - I was hoping that az webapp log tail? would capture stdout to help debug any further issues we come across.

The node:alpine image sets INFO as the default log level, so why does it not capture this? Is this persisted to a separate file that I need to view via the scm app?

[richtong]

We just had this happen with jupyter/scipy-notebook. Any ideas about how to fix. I'm looking at it and I can't see any high uid or gids. What's the best way to debug

[PedroGuerraPT]

For anyone facing this issue on NPM 9.1.1/9.1.2, we have created the following blog post detailing how to further troubleshoot and resolve the high UID constraint:

https://azureossd.github.io/2022/06/30/Docker-User-Namespace-remapping-issues/index.html

[PedroGuerraPT]

Current symptom has been exposed to NPM maintainers, breaking change discussion can be tracked here: npm/rfcs#546

[nazario29]

In my case I could experience this issue with any of npm 9.x versions, actually tried all of them below 9.1.1.
For anyone that is struggling with this issue please follow Qusic workaround 1b7c1f8 🥇

[PedroGuerraPT]

In my case I could experience this issue with any of npm 9.x versions, actually tried all of them below 9.1.1. For anyone that is struggling with this issue please follow Qusic workaround 1b7c1f8 🥇

I want to mention that the chown needs to be included on the same instruction/layer where npm install is ran, otherwise the same error will persist.

I may also share that the issue started to occur specifically on npm 9.0.0-pre.6 (we were unable to reproduce the symptom on 9.0.0-pre.5):

Reasons for the change and further discussion can be seen on the aforementioned RFC, unfortunately it seems the behavior is here to stay on release 9.

[melroy89]

Fix npm/pacote#261 (review)

[PedroGuerraPT]

Fix npm/pacote#261 (review)

Thank you for keeping us posted Melroy, this has been impacting a good portion of our customers for quite a while 💪

Tagging @Ajsalemo for visibility

[melroy89]

You can force update the npm dependency: npm install -g pacote@latest

[NoNameProvided]

The above-mentioned PR has been merged and included in the 9.5.1 release of NPM.

[neckell]

npm install -g pacote@latest on dockerfile fixed this issue for me in an app service deployment

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano
Azure App Service