hcsshim::Im portLayer failed in Win32: Access is denied. (0x5)

[gwellington]

Description

Steps to reproduce the issue:

1. Unable to reproduce this issue reliably - it seems to get into this state randomly.

Describe the results you received:
When pulling a Docker container, sometimes we are unable to extract the image. Once the error occurs, the system is unrecoverable and we need to trash the instance. The error that we receive is:

failed to register layer: re-exec error: exit status 1: output: time="2018-01-11T18:16:04Z" level=error msg="hcsshim::Im portLayer failed in Win32: Access is denied. (0x5) layerId=\\\\?\\E:\\docker\\windowsfilter\\b6888696dee4442c3d5709709bd a7fd57192aca264e3f58ccd0fd87c1e5d228b flavour=1 folder=C:\\Users\\myhandle\\AppData\\Local\\Temp\\hcs873555083" hcsshim::ImportLayer failed in Win32: Access is denied. (0x5) layerId=\\?\E:\docker\windowsfilter\b6888696dee4442c3d5709 709bda7fd57192aca264e3f58ccd0fd87c1e5d228b flavour=1 folder=C:\Users\myhandle\AppData\Local\Temp\hcs873555083

Describe the results you expected:
Expect the container image to be reliably pulled and extracted

Additional information you deem important (e.g. issue happens only occasionally):
Issue only happens occasionally and we're unable to track down additional information when debugging. Running dockerd -D doesn't produce any other additional output than the above.

We've seen this happen for both the micrososft/windowsservercore image and other images built off of it. What is weird is that we've seen this issue with the RTM version Docker version 1.12.2-cs2-ws-beta, build 050b611 and the latest: Docker version 17.06.2-ee-6, build e75fdb8

Output of docker version:

Docker version 17.06.2-ee-6, build e75fdb8

Output of docker info:

Containers: 76
 Running: 0
 Paused: 0
 Stopped: 76
Images: 3
Server Version: 17.06.2-ee-6
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2007.amd64fre.rs1_release.171231-1800)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 32GiB
Name: host
ID: VZIV:QABT:63VB:KUOW:4FOR:BTTB:QULZ:U7KS:CIZK:QWTU:63SI:65EG
Docker Root Dir: E:\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS, 2016 with Containers with latest Windows updates

Sample Dockerfile as follows

FROM windowsservercore:0.2.6

COPY ["modules/Quest",      "C:/Windows/System32/WindowsPowerShell/v1.0/Modules/Quest"]
COPY ["modules/JiraPS",     "C:/Windows/System32/WindowsPowerShell/v1.0/Modules/JiraPS"]
COPY ["modules/ADHelpers",  "C:/Windows/System32/WindowsPowerShell/v1.0/Modules/ADHelpers"]
COPY ["modules/Helpers",    "C:/Windows/System32/WindowsPowerShell/v1.0/Modules/Helpers"]
COPY ["scripts",            "C:/company/scripts"]

RUN powershell C:\\company\\entrypoint.ps1; \
    Install-PackageProvider -Name Nuget -Force | Out-Null; \
    Install-PackageProvider -Name NuGet -Force | Out-Null; \
    Register-PSRepository -Name Artifactory -SourceLocation 'https://publish.artifactory.company.com/artifactory/api/nuget/internal-nuget-sandbox' -InstallationPolicy Trusted; \
    Install-Module -Name Auth -Scope AllUsers -Repository Artifactory -Force; \
    C:/Windows/System32/WindowsPowerShell/v1.0/Modules/Quest/ADDQADSnapIn.ps1;

COPY automationentrypoint.ps1 C:/company/automationentrypoint.ps1

SHELL ["powershell", "-Command"]
ENTRYPOINT C:/company/automationentrypoint.ps1;
CMD ["-?"]```

[darstahl]

When you encounter this, are there any other containers running on the host? Either via Docker or in powershell get-computeprocess?

Is there an anti-virus software installed which might be scanning the file in question (C:\Users\myhandle\AppData\Local\Temp\hcs873555083)?

[gwellington]

Hey @darrenstahlmsft I think we can kind of reproduce this. When I run something that purposefully crashes our server, it reboots and then we run into the error where we can't extract images.

The code that I'm running to forcibly reproduce this is:

(0..30) | % { docker run -d microsoft/windowsservercore:latest powershell "$a = (0..60); while ($true) { $a += $a }"

Afterwards when the instance comes back up, we seem to run into the "Access is denied issue" while extract layers. Granted forcibly causing the machine to crash isn't ideal, but it's the best way I've been able to try and reproduce this issue.

We are running an AV software and we don't have that folder whitelisted, however, we aren't seeing anything come up in the logs where it's flagging that folder, process, or related files.