crictl pull fails on windows because of unicode character in path

[franznemeth]

Description

We are using containerd on Windows Server 2019 (1809) for our CI as a CRI for rke2 kubernetes. We have images which have Visual Studio 2019 installed.

The installation of VS2019 has added some files with unicode characters in it's name.

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TextMate\Starterkit\Extensions\cmake\Snippets>dir 
 Volume in drive C has no label. 
 Volume Serial Number is D231-E700

 Directory of C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TextMate\Starterkit\Extensions\cmake\Snippets

02/11/2022  09:44 AM    <DIR>          .
02/11/2022  09:44 AM    <DIR>          ..
02/11/2022  09:44 AM               506 File(Glob …).tmSnippet
02/11/2022  09:44 AM               514 Foreach … EndForeach.tmSnippet
02/11/2022  09:44 AM               488 If … Endif.tmSnippet
02/11/2022  09:44 AM               512 List(Append …).tmSnippet
02/11/2022  09:44 AM               472 Message(…).tmSnippet
02/11/2022  09:44 AM               486 Set(Variable …).tmSnippet
               6 File(s)          2,978 bytes
               2 Dir(s)  105,865,719,808 bytes free

These characters prevent containerd from successfully pulling the container from a registry.

time="2022-03-03T07:34:26+01:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"<removed>\": failed to extract layer sha256:<removed>: open \\\\?\\C:\\Windows\\TEMP\\hcs071695283\\Files\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\Common7\\IDE\\CommonExtensions\\Microsoft\\TextMate\\Starterkit\\Extensions\\cmake\\Snippets\\File(Glob �).tmSnippet: The filename, directory name, or volume label syntax is incorrect.: unknown"

Pulling the same image using another CRI like docker succeeds

Steps to reproduce the issue

1. Create container image with filenames using unicode characters
   Build a container using the following command: docker build -m 8G -t foo . (it's important to use the -m parameter otherwise the installer fails):

# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019
# restore the default Windows shell for correct batch processing.
SHELL ["cmd", "/S", "/C"]
# https://docs.microsoft.com/en-us/visualstudio/install/advanced-build-tools-container?view=vs-2019
# copy install script
COPY Install.cmd C:\TEMP\
# download collect.exe in case of an install failure.
ADD https://aka.ms/vscollect.exe C:\TEMP\collect.exe
# download the BuildTools bootstrapper.
# latest: http://aka.ms/vs/16/release/vs_buildtools.exe
# or specific: https://docs.microsoft.com/en-us/visualstudio/releases/2019/history
# 16.11.10
ADD https://download.visualstudio.microsoft.com/download/pr/791f3d28-7e20-45d9-9373-5dcfbdd1f6db/d5eabc3f4472d5ab18662648c8b6a08ea0553699819b88f89d84ec42d12f6ad7/vs_BuildTools.exe C:\TEMP\vs_buildtools.exe
# install BuildTools.
# https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2019
RUN C:\TEMP\Install.cmd C:\TEMP\vs_buildtools.exe `
  --quiet --wait --norestart `
  --add Microsoft.VisualStudio.Workload.VCTools `
  --includeRecommended

Install.cmd:

@if not defined _echo echo off
setlocal enabledelayedexpansion
call %*
if "%ERRORLEVEL%"=="3010" (
    exit /b 0
) else (
    if not "%ERRORLEVEL%"=="0" (
        set ERR=%ERRORLEVEL%
        call C:\TEMP\collect.exe -zip:C:\TEMP\vslogs.zip
        exit /b !ERR!
    )
)

2. Upload this image to a registry
3. Try to pull image using crictl

Describe the results you received and expected

The image to be pulled normally

What version of containerd are you using?

containerd github.com/containerd/containerd v1.5.7 8686ede

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[kzys]

While the Unicode character is suspicious, the beginning (\\?\C:\) looks also wrong. The syntax is \\[host name]\\[drive letter]:\[path] I believe.

\\?\C:\Windows\TEMP\hcs071695283\Files\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TextMate\Starterkit\Extensions\cmake\Snippets\File(Glob �).tmSnippet

[franznemeth]

The \\?\C:\path is the way windows parses a path that exceeds its max length so it is to be expected in our usecase.
Docs: https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd

[jterry75]

Hey this is a known issue. Can you upgrade to containerd 1.6.0+?

[jterry75]

@dcantah - FYI

[dcantah]

Yep, unfortunately in 1.5.x this was broken. This change describes and fixes the issue microsoft/hcsshim#1138 and I planned to remedy it in the 1.5.x line here #6588 (whenever a new release would get cut, need to figure out why the linter could be complaining first but haven't got the time..). 1.6.0 and any patch releases to come should have this solved.

[franznemeth]

Since containerd comes with rke2 and I cannot pick the version I will manually install the newer version and get back to you. Thanks for the quick replies!

[franznemeth]

Tested on 1.6.1 on ws19 and the issue is fixed. Thanks for your assistance. I have opened an issue with rke2 to upgrade to a newer containerd version. rancher/rke2#2541

PS C:\> ctr
NAME:
   ctr -
        __
  _____/ /______
 / ___/ __/ ___/
/ /__/ /_/ /
\___/\__/_/

containerd CLI


USAGE:
   ctr.exe [global options] command [command options] [arguments...]

VERSION:
   v1.6.1

DESCRIPTION:
...
PS C:\tools> ctr images ls
PS C:\> ctr images pull <removed>
<removed>
elapsed: 173.6s                                                                   total:  5.6 Gi (32.9 MiB/s)
unpacking windows/amd64 sha256:<removed>...
done: 11m28.1949406s