Container Feature installation broken due to self signed certificates (since 0.251.0)

[pF-luis]

• VSCode Version: 1.71.0
• Local OS Version: WSL2 Ubuntu 22.04 LTS on Windows 10 Pro(Build 19044.1889)
• Remote OS Version: debian bullseye
• Remote Extension/Connection Type: Containers
• Logs:

[41 ms] Remote-Containers 0.251.0 in VS Code 1.71.0 (784b0177c56c607789f9638da7b6bf3230d47a8c).
[40 ms] Start: Run: wsl -d Ubuntu-20.04 -e wslpath -u \\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ
[423 ms] Start: Resolving Remote
[450 ms] Start: Run: wsl -d Ubuntu-20.04 -e wslpath -u \\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ
[683 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/userXYZ/projects/projectXYZ' && /bin/sh
[697 ms] Start: Run in host: id -un
[797 ms] userXYZ
[798 ms] 
[798 ms] Start: Run in host: cat /etc/passwd
[801 ms] Start: Run in host: echo ~
[802 ms] /home/userXYZ
[802 ms] 
[803 ms] Start: Run in host: test -x '/home/userXYZ/.vscode-remote-containers/bin/784b0177c56c607789f9638da7b6bf3230d47a8c/node'
[805 ms] 
[805 ms] 
[806 ms] Start: Run in host: test -f '/home/userXYZ/.vscode-remote-containers/dist/vscode-remote-containers-server-0.251.0.js'
[807 ms] 
[808 ms] 
[809 ms] userEnvProbe: loginInteractiveShell (default)
[810 ms] userEnvProbe shell: /bin/bash
[1355 ms] userEnvProbe PATHs:
Probe:     '/home/userXYZ/.local/bin:/home/userXYZ/bin:/home/userXYZ/.nvm/versions/node/v16.13.1/bin:/home/userXYZ/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Amazon Corretto/jdk1.8.0_332/bin:/mnt/c/Program Files/Amazon Corretto/jdk11.0.15_9/bin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/Program Files/Amazon Corretto/jdk1.8.0_332/bin:/mnt/c/Program Files/SafeNet/Authentication/SAC/x64:/mnt/c/Program Files/SafeNet/Authentication/SAC/x32:/mnt/c/Program Files/nodejs/":/mnt/c/Program Files/Amazon Corretto/jdk1.8.0_192/bin:/mnt/c/Program Files/Microsoft VS Code/bin":/mnt/c/Program Files/Amazon Corretto/jdk1.8.0_192:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/Java/jdk1.8.0_192/bin:/mnt/c/apache-maven-3.6.3/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/userXYZ/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/userXYZ/cygwin/bin:/mnt/c/php7.2:/mnt/c/Program Files/JetBrains/IntelliJ IDEA 2020.3.2/bin:/mnt/c/Users/userXYZ/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin:/home/userXYZ/.dotnet/tools'
Container: None
[1358 ms] Setting up container for folder or workspace: /home/userXYZ/projects/projectXYZ
[1360 ms] Start: Check Docker is running
[1360 ms] Start: Run in Host: docker version --format {{.Server.APIVersion}}
[2096 ms] Server API version: 4.1.0
[2096 ms] Start: Run in Host: docker volume ls -q
[2538 ms] Start: Run in Host: docker ps -q -a --filter label=vsch.local.folder=\\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ --filter label=vsch.quality=stable
[2955 ms] Start: Run in Host: docker ps -q -a --filter label=devcontainer.local_folder=\\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ
[3407 ms] Start: Run in Host: /home/userXYZ/.vscode-remote-containers/bin/784b0177c56c607789f9638da7b6bf3230d47a8c/node /home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js up --workspace-folder /home/userXYZ/projects/projectXYZ --workspace-mount-consistency cached --id-label devcontainer.local_folder=\\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ --log-level debug --log-format json --config /home/userXYZ/projects/projectXYZ/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[3793 ms] remote-containers 0.251.0.
[3793 ms] Start: Run: docker buildx version
[3840 ms] Start: Resolving Remote
[3845 ms] Start: Run: git rev-parse --show-cdup
[3853 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=\\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ
[4270 ms] local container features stored at: /home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/node_modules/vscode-dev-containers/container-features
[4272 ms] Start: Run: tar --no-same-owner -x -f -
[4340 ms] * Processing feature: git
[4480 ms] Failed to get registry auth token with error: Error: self signed certificate in certificate chain
[4575 ms] (!) WARNING: Falling back to deprecated GitHub Release syntax. See https://github.com/devcontainers/spec/blob/main/proposals/devcontainer-features.md#referencing-a-feature for updated specification.
[4575 ms] Github feature.
[4575 ms] Invalid parse for GitHub Release feature: Follow format '<publisher>/<feature-set>/<feature>, or republish feature to OCI registry.'
[4446 ms] Error: Failed to process feature ghcr.io/devcontainers/features/git:1
[4446 ms]     at QR (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:188:2956)
[4447 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[4447 ms]     at async uc (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:188:2279)
[4447 ms]     at async Zs (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:188:12771)
[4447 ms]     at async OF (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:3261)
[4447 ms]     at async Wf (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:2801)
[4447 ms]     at async pb (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:1779)
[4447 ms]     at async DF (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:263:2006)
[4448 ms]     at async to (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:263:3110)
[4448 ms]     at async Ak (/home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:383:8108)
[4458 ms] Exit code 1
[4460 ms] Command failed: /home/userXYZ/.vscode-remote-containers/bin/784b0177c56c607789f9638da7b6bf3230d47a8c/node /home/userXYZ/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js up --workspace-folder /home/userXYZ/projects/projectXYZ --workspace-mount-consistency cached --id-label devcontainer.local_folder=\\wsl$\Ubuntu-20.04\home\userXYZ\projects\projectXYZ --log-level debug --log-format json --config /home/userXYZ/projects/projectXYZ/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[4460 ms] Exit code 1

Steps to Reproduce:

1. Add devcontainer.json and Dockerfile
2. Rebuild container (without cache)
3. Error as seen in logs appears

• I also tested the pre-release version which produced the same issue
• the 0.245.2 version works just fine
• We normally download the company certificates within the Dockerfile and then execute update-ca-certificates. Now since the new version it "seems" to me that all the feature installations like git happen before the dockerfile is executed?

Does this issue occur when you try this locally?: Yes
Does this issue occur when you try this locally and all extensions are disabled?: Yes

Thanks for you help

[fatelei]

has same problem

[ray-kaminski]

^^this

[OneCyrus]

Features as OCI artifacts should resolve this. Any ETA for support in VSCode?

https://code.visualstudio.com/blogs/2022/09/15/dev-container-features

[idwessough]

Features as OCI artifacts should resolve this. Any ETA for support in VSCode?

https://code.visualstudio.com/blogs/2022/09/15/dev-container-features

Sadly for me it did not resolve the problem

No one feature is working in this version..... :
#7060

[jeteve]

same problem

[jeteve]

Features as OCI artifacts should resolve this. Any ETA for support in VSCode?

https://code.visualstudio.com/blogs/2022/09/15/dev-container-features

That solved it for me. Replace old style feature with this new way, and it works.

[OneCyrus]

Features as OCI artifacts should resolve this. Any ETA for support in VSCode?
https://code.visualstudio.com/blogs/2022/09/15/dev-container-features

That solved it for me. Replace old style feature with this new way, and it works.

how does your devcontainer definition look? my vscode is not happy with every way i tried. so i guessed it‘s not supported yet.

[ilaner]

For us, the workaround we found is to run the install.sh scripts in the dockerfile itself:
https://github.com/demisto/content/blob/master/.devcontainer/Dockerfile

[lioncubs]

Sorry for bringing this back up, but we are also stuck in the same scenario - in vscode as well as when running devcontainer/cli@0.20.0 (I'm not sure where the functionality was changed.

We need to ensure that our "self-signed" cert is installed PRIOR to any feature additions - If it is not, the first feature install fails due to invalid signatures.

I understand from the thread in #6995, it's not a vscode bug - so how do we solve the issue locally?

Using devcontainer/cli@0.6.0 or an older vscode I can build our devcontainer with a Dockerfile that looks something like this

#Sample devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/cpp
{
	"name": "C++",
	"build": {
		"dockerfile": "Dockerfile",
		// Update 'VARIANT' to pick an Debian / Ubuntu OS version: debian-11, debian-10, ubuntu-22.04, ubuntu-20.04, ubuntu-18.04
		// Use Debian 11, Ubuntu 18.04 or Ubuntu 22.04 on local arm64/Apple Silicon
		"args": { 
                      "VARIANT": "ubuntu-22.04" ,
                      "ADD_LOCAL_CERTS": "true"
                }
	},
	"runArgs": ["--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],

	// Configure tool-specific properties.
	"customizations": {
		// Configure properties specific to VS Code.
		"vscode": {
			// Add the IDs of extensions you want installed when the container is created.
			"extensions": [
				"ms-vscode.cpptools",
				"ms-vscode.cmake-tools"
			]
		}
	},

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "gcc -v",

	// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
	"remoteUser": "vscode",
	"features": {
		"docker-in-docker": "latest",
		"git": "latest",
		"golang": "1.18.4"
	}
}

#Sample Dockerfile

# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.236.0/containers/cpp/.devcontainer/base.Dockerfile

# [Choice] Debian / Ubuntu version (use Debian 11, Ubuntu 18.04/22.04 on local arm64/Apple Silicon): debian-11, debian-10, ubuntu-22.04, ubuntu-20.04, ubuntu-18.04
ARG VARIANT="bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/cpp:0-${VARIANT}

#################################
# Fixup the Certs if needed
# Need for Certificate on
ARG ADD_LOCAL_CERTS="none"
COPY install-local-certs.sh /tmp/
COPY certs.zip /tmp/
RUN if [ "${ADD_LOCAL_CERTS}" = "true" ]; then \
        chmod +x /tmp/install-local-certs.sh && \
        /tmp/install-local-certs.sh; \
    fi && \
    rm /tmp/install-local-certs.sh && \
    rm /tmp/certs.zip
#################################

# [Optional] Install CMake version different from what base image has already installed. 
# CMake reinstall choices: none, 3.21.5, 3.22.2, or versions from https://cmake.org/download/
ARG REINSTALL_CMAKE_VERSION_FROM_SOURCE="none"

# Optionally install the cmake for vcpkg
COPY ./reinstall-cmake.sh /tmp/
RUN if [ "${REINSTALL_CMAKE_VERSION_FROM_SOURCE}" != "none" ]; then \
        chmod +x /tmp/reinstall-cmake.sh && /tmp/reinstall-cmake.sh ${REINSTALL_CMAKE_VERSION_FROM_SOURCE}; \
    fi \
    && rm -f /tmp/reinstall-cmake.sh

# [Optional] Uncomment this section to install additional vcpkg ports.
# RUN su vscode -c "${VCPKG_ROOT}/vcpkg install <your-port-name-here>"

# [Optional] Uncomment this section to install additional packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
#     && apt-get -y install --no-install-recommends <your-package-list-here>

With the latest devcontainer/cli and vscode we are failing before we fix the certs and of course, if I have no features added the devcontainer build succeeds

BTW: I was also hoping to create a local "feature" to install the certs in specified order - and there is still some preprocessing I don't YET understand that is blocked - I got excited that the "feature" trick might work

[ambar-qnx]

I am also facing the same issue. I am getting following error when I try to create a dev container configuration file. My system uses netskope mitm proxy, and the netskope root ca is a part of the certificate store on my windows sytstem.

I try to run Dev Contgainers: Add Dev Containers Configuration Files i get the following error:

The logs from dev container log are below:

[112595 ms] Dev Containers 0.266.1 in VS Code 1.74.3 (97dec172d3256f8ca4bfb2143f3f76b503ca0534).
[112594 ms]  -- CREATEDEVCONTAINER v2
[117362 ms] Start: Run: C:\Users\xxxxxx\AppData\Local\Programs\Microsoft VS Code\Code.exe --ms-enable-electron-run-as-node c:\Users\xxxxxx\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js templates apply --workspace-folder C:\Users\xxxxxx\AppData\Local\Temp\tmp-output-dir-1674139471111 --template-id ghcr.io/devcontainers/templates/docker-existing-docker-compose:latest --template-args {} --features [] --tmp-dir C:\Users\xxxxxx\AppData\Local\Temp\tmp-dir-1674139471111
[117653 ms] [2023-01-19T14:44:31.461Z] @devcontainers/cli 0.25.2. Node.js v16.14.2. win32 10.0.22621 x64.
[117669 ms] (node:39420) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[117669 ms] (Use `Code --trace-deprecation ...` to show where the warning was created)
[117722 ms] [2023-01-19T14:44:31.530Z] Failed to get registry auth token with error: Error: self signed certificate in certificate chain
[117752 ms] [2023-01-19T14:44:31.560Z] Failed to fetch template manifest for ghcr.io/devcontainers/templates/docker-existing-docker-compose:latest
[117753 ms] [2023-01-19T14:44:31.560Z] Failed to fetch template 'ghcr.io/devcontainers/templates/docker-existing-docker-compose:latest'.
[117765 ms] Exit code 1

The only way to get this to work is by disabling netskope

PS: Removed my windows user name from the output above.

[OneCyrus]

looks like the feature installation changed again. it fetches the feature from the host instead of a bootstrapping image and the actual installation happens in the defined image of the devcontainer.

we got it working with adding certificate ripper to the dockerfile which we have defined in the devcontainer.json

# Export certs with crip and add them to the certificate store
RUN wget https://github.com/Hakky54/certificate-ripper/releases/download/2.0.1/crip-linux-amd64.tar.gz --no-check-certificate -qO- | tar xvz -C /tmp/
RUN sudo /tmp/crip export pem --url=https://www.google.com --destination /usr/local/share/ca-certificates/
RUN sudo update-ca-certificates
ENV NODE_OPTIONS=--use-openssl-ca

still have the self signed certificate issue with clone in volume though (#3713) as that happens still in the bootstrap container.

[kennethredler]

@chrmarti any chance this can be addressed soon?

[mholttech]

I know that everyone has mixed results in trying to figure this out, including myself, but I finally found a way that works in my corporate environment where all systems need custom Root CA certificates loaded.

I published my findings at mholttech/devcontainer-features. I tested this with the repo residing inside of Windows and inside of an Ubuntu WSL2 instance.

[yogeshdhawale]

This does not work for home environment as well. However, if I start the devcontainer directly from windows it work. Fails when started from wsl. I tried lot many solutions but doesn't seems to be working.
If devcontainer file does not contain any featuers, then it works with wsl as well.
No proxy involved. I tried various ways, but nothing seems to be working.

[2023-09-06T11:17:16.316Z] Start: Run: tar --no-same-owner -x -f -
[2023-09-06T11:17:16.354Z] Stop (38 ms): Run: tar --no-same-owner -x -f -
[2023-09-06T11:17:16.393Z] Resolving Feature dependencies for 'ghcr.io/devcontainers/features/java:1'...
[2023-09-06T11:17:16.394Z] * Processing feature: ghcr.io/devcontainers/features/java:1
[2023-09-06T11:17:16.774Z] Start: Run: docker-credential-desktop.exe get
[2023-09-06T11:17:17.414Z] Stop (640 ms): Run: docker-credential-desktop.exe get
[2023-09-06T11:19:31.927Z] Error getting blob: Error: connect ETIMEDOUT 198.51.44.1:443
[2023-09-06T11:19:31.929Z] Error: Failed to download package for ghcr.io/devcontainers/features/java

[yogeshdhawale]

This, btw, worked for me on fedora wsl image seamlessly. However, Ubuntu wsl continues to fail with timeout error

[jorgecuevas1]

same error here