Windows SMB Mapping Failures When VM and Container Hotfixes are Mismatched

[micahyoung]

Description

New-SmbMapping, net use and related commands fail frequently when run in containers on Windows Server Datacenter with Containers 2019 (1809, 1803, and 1709) Hosts when they contain HotFixes from February 2019 and the Docker image used contains HotFixes from March 2019.

We see that the presence of HotFix KB4489899 on the Docker image but not on the Host, is correlated to the failure but have not confirmed whether it is the specific cause.

Steps to reproduce the issue:

1. Create a Windows Server 2019 Datacenter Host VM (which contains the February HotFixes)

   $ az vm create -n febVM -g REDACTED --image MicrosoftWindowsServer:WindowsServer:2019-Datacenter-with-Containers-smalldisk:2019.0.20190214
   

2. From an RDP connection to the VM, open an Adminstrator Powershell terminal for the following steps

3. Confirm February HotFixes on Host:

   > Get-Hotfix
   
   Source        Description      HotFixID      InstalledBy          InstalledOn
   ------        -----------      --------      -----------          -----------
   febVM         Update           KB4483452                          2/13/2019 12:00:00 AM
   febVM         Security Update  KB4470788                          2/13/2019 12:00:00 AM
   febVM         Security Update  KB4487044                          2/13/2019 12:00:00 AM
   

4. Install docker

   > Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
   > Install-Package -Name docker -ProviderName DockerMsftProvider -Force
   

5. Confirm March HotFixes on the Docker image mcr.microsoft.com/windows/servercore:1809-KB4489899:

   > docker run mcr.microsoft.com/windows/servercore:1809-KB4489899 powershell Get-Hotfix
   
   Source        Description      HotFixID      InstalledBy          InstalledOn
   ------        -----------      --------      -----------          -----------
   0BC43963DE9C  Security Update  KB4470788                          3/9/2019 12:00:00 AM
   0BC43963DE9C  Security Update  KB4489899                          3/9/2019 12:00:00 AM
   

6. Repeatedly attempt to start a Docker container (with March HotFixes) and run net use.

   > while($true){ docker run mcr.microsoft.com/windows/servercore:1809-KB4489899 powershell "Start-Sleep 1; net use"; if ($LASTEXITCODE -ne 0) {break} }
   

   • Start-sleep 1 is used to rule out unrelated container startup race conditions.

Describe the results you received:

• net use frequently (but not always) displays an error:

  System error 5 has occurred.
  
  Access is denied.
  

• Any net use ... or New-SmbMapping ... commands that attempt to make a connection fail similarly.

Describe the results you expected:

• We expect net use to always display the default output:

  New connections will be remembered.
  
  There are no entries in the list.
  

• We expect net use and New-SmbMapping to successfully establish SMB connections

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Enterprise
Version:           18.09.5
API version:       1.39
Go version:        go1.10.8
Git commit:        be4553c277
Built:             04/11/2019 06:44:52
OS/Arch:           windows/amd64
Experimental:      false

Server: Docker Engine - Enterprise
Engine:
 Version:          18.09.5
 API version:      1.39 (minimum version 1.24)
 Go version:       go1.10.8
 Git commit:       be4553c277
 Built:            04/11/2019 06:43:04
 OS/Arch:          windows/amd64
 Experimental:     false

Output of docker info:

(paste your output here)Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 4
Server Version: 18.09.5
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.316)
OSType: windows
Architecture: x86_64
CPUs: 1
Total Memory: 3.5GiB
Name: febVM
ID: 2IMG:UDG5:YM6Z:NM4X:SAAG:5QTO:Y63A:CLMA:NQRQ:I2HP:MDJC:DWPI
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

• Windows Version: Microsoft Windows Server 2019 Datacenter / 10.0.17763 N/A Build 17763
• Docker for Windows Version: 18.09.5
• Windows VM Azure URN: MicrosoftWindowsServer:WindowsServer:2019-Datacenter-with-Containers-smalldisk:2019.0.20190214
• Docker image Version: mcr.microsoft.com/windows/servercore:1809-KB4489899
• Also confirmed in several other IaaSes and Docker Enterprise versions.

[olljanat]

@micahyoung that is how process isolation mode on Windows containers works. They call host machine kernel directly so any kernel related hotfixes must be installed to host machine.

If I recall correctly on early preview versions of Windows container there was limitation that host and container image version needed to share even patch level but that would make upgrade process very tricky which why it is removed.