Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Toolbox (a.k.a. Fedora Toolbox) #3345

Open
lbssousa opened this issue Jul 10, 2020 · 41 comments
Open

Support Toolbox (a.k.a. Fedora Toolbox) #3345

lbssousa opened this issue Jul 10, 2020 · 41 comments
Assignees
Labels
containers Issue in vscode-remote containers feature-request Request for new features or functionality

Comments

@lbssousa
Copy link

Since Remote Containers extension actually supports Podman container runtime, I would like to go a step further and support container environments provided by Toolbox (do not confuse with Docker Toolbox).

Toolbox is a wrapper tool for Podman that allows creating and entering mutable containerized environments. I seems something like WSL approach, but for OCI containers via Podman.

There's already a way to connect VSCode to a Toolbox environment via Remote SSH extension, using this workaround, but I want to do the same directly via Remote Containers extension (or maybe a new Remote Toolbox one)

Maybe @debarshiray could provide more info about what's needed to add Toolbox support to VSCode Remote Development extensions.

@github-actions github-actions bot added the containers Issue in vscode-remote containers label Jul 10, 2020
@chrmarti
Copy link
Contributor

chrmarti commented Aug 6, 2020

Since it is a wrapper: Can the podman CLI be used to access it? If so, it should already be possible to connect to it with Remote-Containers.

@chrmarti chrmarti self-assigned this Aug 6, 2020
@chrmarti chrmarti added the feature-request Request for new features or functionality label Aug 6, 2020
@DMouse10462
Copy link

DMouse10462 commented Aug 12, 2020

Can the podman CLI be used to access it?

Yes, you can access it just like any regular container. I'm not the issue submitter, but I can share some of what I encounter when trying to connect to an existing toolbox.

Some basic info for my setup:
OS: Fedora Workstation 32
$ podman -v: podman version 2.0.4

Logs from attempting to connect to a Fedora 32 toolbox named grails-dev:

[34 ms] Start: Resolving remote
[38 ms] Setting up container grails-dev

[39 ms] Start: Run: podman inspect --type container grails-dev
[297 ms] Start: Check Docker is running
[298 ms] Start: Run: podman info
[741 ms] Start: Inspecting container
[742 ms] Start: Run: podman inspect --type container ca9d153d0a5c2b4191be68bb3244b700416a87f5682b3db47b1f7f3a18dddcb7
[1190 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=f54745c4-c0f0-4f9b-80be-0274471e9fbc1597217678921 ca9d153d0a5c2b4191be68bb3244b700416a87f5682b3db47b1f7f3a18dddcb7 /bin/sh
[1232 ms] Start: Run in container: uname -m
[1609 ms] x86_64
[1609 ms] 
[1610 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[1614 ms] NAME=Fedora
VERSION="32 (Container Image)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora 32 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
[1614 ms] 
[1614 ms] Start: Run in container: cat /etc/passwd
[1620 ms] Start: Run in container: test -d /root/.vscode-server-insiders
[1624 ms] 
[1624 ms] 
[1625 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server-insiders/data/Machine' && { > '/root/.vscode-server-insiders/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[1632 ms] 
[1632 ms] 
[1632 ms] Exit code 1
[1633 ms] Start: Run in container: test -d /root/.vscode-server-insiders/bin/2277c8e2a3e1cc630a6397301ba54a1dccd8a60d
[1639 ms] 
[1639 ms] 
[1639 ms] Start: Launching Remote-Containers helper.
[1642 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[1659 ms] Start: Run in container: gpgconf --list-dir agent-socket
[1662 ms] /root/.gnupg/S.gpg-agent
[1663 ms] 
[1663 ms] Start: Run in container: gpgconf --list-dir homedir
[1667 ms] /root/.gnupg
[1667 ms] 
[1667 ms] Start: Run in container: ls '/root/.gnupg/private-keys-v1.d' 2>/dev/null
[1672 ms] 
[1672 ms] 
[1672 ms] Exit code 2
[1672 ms] Start: Run in container: mkdir -p -m 700 '/root/.gnupg'
[1680 ms] 
[1680 ms] 
[1680 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /root/.vscode-server-insiders/bin/2277c8e2a3e1cc630a6397301ba54a1dccd8a60d/node /tmp/vscode-remote-containers-d8def71c0ee26c51418e0da1049888df62f6a191.js $*; }; f' || true
[1683 ms] Start: Run: gpgconf --list-dir homedir
[1699 ms] 
[1699 ms] 
[1700 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-d8def71c0ee26c51418e0da1049888df62f6a191.js' >/tmp/vscode-remote-containers-d8def71c0ee26c51418e0da1049888df62f6a191.js
[1709 ms] 
[1709 ms] 
[1709 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server-insiders/data/Machine' && { > '/root/.vscode-server-insiders/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[1718 ms] 
[1721 ms] 
[1721 ms] Exit code 1
[1721 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-d8def71c0ee26c51418e0da1049888df62f6a191.js' >/tmp/vscode-remote-containers-server-d8def71c0ee26c51418e0da1049888df62f6a191.js
[1767 ms] 
[1767 ms] 
[1767 ms] Start: Run in container: gpgconf --list-dir homedir
[1767 ms] Start: Run: podman exec -i -u root -e REMOTE_CONTAINERS_SOCKETS=["/tmp/vscode-ssh-auth-d8def71c0ee26c51418e0da1049888df62f6a191.sock","/root/.gnupg/S.gpg-agent"] -e REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-d8def71c0ee26c51418e0da1049888df62f6a191.sock ca9d153d0a5c2b4191be68bb3244b700416a87f5682b3db47b1f7f3a18dddcb7 /root/.vscode-server-insiders/bin/2277c8e2a3e1cc630a6397301ba54a1dccd8a60d/node /tmp/vscode-remote-containers-server-d8def71c0ee26c51418e0da1049888df62f6a191.js
[1797 ms] /root/.gnupg
[1797 ms] 
[1797 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink -f /proc/$pid/cwd ; xargs -0 < /proc/$pid/environ ; xargs -0 < /proc/$pid/cmdline ; } ; echo ; done 2>/dev/null
[1816 ms] Start: Run: gpg-connect-agent updatestartuptty /bye
[2301 ms] Remote-Containers server: Error: open executable: Permission denied: OCI runtime permission denied error
[2416 ms] Remote-Containers server terminated (code: 126, signal: null).
[3446 ms] Start: Run in container: /root/.vscode-server-insiders/bin/2277c8e2a3e1cc630a6397301ba54a1dccd8a60d/server.sh --disable-user-env-probe --use-host-proxy --port 0 --extensions-download-dir /root/.vscode-server-insiders/extensionsCache
[3458 ms] Error: stream is closed
        at /home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:208142
        at c (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:224558)
        at EventEmitter.e.<computed> [as exec] (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:224281)
        at u (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:202138)
        at new Promise (<anonymous>)
        at /home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:202116
        at J (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:81982)
        at W (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:81174)
        at processTicksAndRejections (internal/process/task_queues.js:85:5)
        at async Object.t.injectIntoContainer (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:77410)
        at async Object.t.resolve (/home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:238639)
        at async /home/dmouse/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.134.0/dist/extension.js:1:688588

When I manually go into the toolbox and attempt to run the server.sh command (as root within the container due to permission issues; the toolbox command usually drops you to a UID 1000 shell with sudo privileges), the server appears to start correctly unlike the output within VS code. Here's the output of me running the command within a toolbox enter shell (Addendum: the same results occur when entering via podman exec and running the command directly as root):

⬢[dmouse@toolbox ~]$ sudo /root/.vscode-server-insiders/bin/2277c8e2a3e1cc630a6397301ba54a1dccd8a60d/server.sh --disable-user-env-probe --use-host-proxy --port 0 --extensions-download-dir /root/.vscode-server-insiders/extensionsCache


*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*


IP Address: <MY HOST IP>
Extension host agent listening on 34411


[03:44:20] Extension host agent started.
Unable to retrieve mac address (Error: Command failed: /sbin/ifconfig -a || /sbin/ip link
/bin/sh: /sbin/ifconfig: No such file or directory
/bin/sh: /sbin/ip: No such file or directory
)
^C
⬢[dmouse@toolbox ~]$ id
uid=1000(dmouse) gid=1000(dmouse) groups=1000(dmouse),10(wheel)

It is worthy of note that the IP address shown is the address of the host despite running within a container; the user's home directory is mounted in a toolbox and the container appears to be running in host networking mode (?). I'm not 100% sure of what else toolbox sets up the container with.

@lbssousa
Copy link
Author

@debarshiray, could you give us any useful info about Toolbox here?

@debarshiray
Copy link

Very excited to see the interest in Toolbox here!

Toolbox containers are just OCI containers created, managed and run by Podman under the hood. Toolbox is a thin wrapper around Podman to set up the containers in a way that makes it easy to use them as pet development environments without being burdened by the cognitive load of being inside a container.

Assuming that you have already created a container (with toolbox create ...), entering or using the container is a matter of:

  • Calling the org.freedesktop.Flatpak.SessionHelper.RequestSession D-Bus method
  • Starting the container with podman start ... and waiting for the entry-point to have performed some run-time configuration
  • Entering it with podman exec ...

I hope that this gives you some idea about what happens underneath the toolbox enter command.

I am happy to answer more specific questions.

@likan999
Copy link

I have a PR for toolbox to make it work: containers/toolbox#587. With the patch, on your local machine, you only need an alias in ~/.ssh/config that has RemoteCommand toolbox run -T bash and everything just works as a normal SSH remote host. Of course VScode could do fancier stuff (e.g. without need to manually modify the ssh config) but at least it is no longer blocked on the toolbox side.

@likan999
Copy link

To clarify, my approach is different to the workaround mentioned in the top. The workaround starts an openssh server inside the toolbox, while my approach doesn't rely on any service running inside the toolbox.

@lbssousa
Copy link
Author

* Calling the `org.freedesktop.Flatpak.SessionHelper.RequestSession` D-Bus method

* Starting the container with `podman start ...` and waiting for the entry-point to have performed some run-time configuration

* Entering it with `podman exec ...`

@chrmarti , can these steps (specially the first one) be performed with current Remote Containers implementation, or there's some lacking piece?

@chrmarti
Copy link
Contributor

@lbssousa There is "initializeCommand" in the devcontainer.json that runs before anything else. We support Podman with a setting for the Docker executable that can be set to podman. We then start the container with podman run and connect to it with podman exec. Not sure what the entry-point needs to do, but you might need to turn off our overriding of it with "overrideCommand": false in the devcontainer.json (and then make sure the container still keeps running).

@lbssousa
Copy link
Author

lbssousa commented Oct 28, 2020

@chrmarti, I was able to attach to my Toolbox container with the following minimal devcontainer.json:

{
  "name": "Toolbox",
  "image": "registry.fedoraproject.org/f33/fedora-toolbox:33",
  "initializeCommand": "gdbus call --session --dest org.freedesktop.Flatpak --object-path /org/freedesktop/Flatpak/SessionHelper --method org.freedesktop.Flatpak.SessionHelper.RequestSession"
  "overrideCommand": false // I see no difference in setting this option or not
}

However, VSCode always connect to that container as root (remoteUser does not work for me, since it seems my host user is not actually created inside the container: toolbox enter still does some magic to make my host user available to the container).

@debarshiray, is there something else I need to do? What shoud I do with the output of gdbus call ... org.freedesktop.Flatpak.SessionHelper.RequestSession command?

EDIT: My mistake. With the approach above, I'm not attaching to the same container created by toolbox create, but, instead, creating a new container from the same image. When I try to attach to the correct Toolbox container, I get the same error reported earlier:

Remote-Containers server: Error: open executable: Permission denied: OCI runtime permission denied error

@debarshiray
Copy link

What shoud I do with the output of gdbus call ... org.freedesktop.Flatpak.SessionHelper.RequestSession
command?

Note that this isn't necessary any more for Toolbox containers created with current toolbox.git, which will be released as Toolbox 0.0.97.

It's possible to detect whether it's necessary or not by running podman inspect on the container and checking if it bind mounts anything at /run/host/monitor.

The overall idea was to somehow keep the time zone configuration of the container synchronized with the host. So far, Toolbox made use of the flatpak-session-helper utility to keep a copy of the host's configuration files in $XDG_RUNTIME_DIR/.flatpak-helper/. This location was then bind mounted inside the container at /run/host/monitor, and symbolic links were set up inside the container's /etc.

Most of this is done as part of the podman create ... call that creates the Toolbox container, and the entry point of the container itself. However something needs to launch the flatpak-session-helper before starting the container so that things are available in $XDG_RUNTIME_DIR/.flatpak-helper/, and this is what that gdbus call ... command does.

@debarshiray
Copy link

Remote-Containers server: Error: open executable: Permission denied: OCI runtime
permission denied error

This could be Podman territory, but we need some more information to be sure. Which is the exact command that's failing?

I don't know very well how vscode-remote-release works, so I will throw some more information out there. :)

All Toolbox containers have their entry points set to toolbox init-container .... This is important because it's used to self-configure the container at run-time. Yes, /usr/bin/toolbox is also the entry point. This is achieved by bind mounting /usr/bin/toolbox from the host into the container and then using it.

/usr/bin/toolbox enter uses podman start ... to start the container and then podman exec ... to enter it. If you are instead using podman run ... to start the container, then that might be running into some problems. It might be possible to navigate them, but it might also be better to just use podman start ... if that's possible.

Note that the podman exec ... invocation also takes a long list of arguments that includes environment variables and what not.

Would it somehow be possible to just use toolbox enter to enter the container?

@chrmarti
Copy link
Contributor

We are trying to stay out of adaptions specific to one CLI or runtime. Podman-support comes from Podman's CLI being compatible with the Docker CLI, so we didn't have to change much (except for the version check).

@lbssousa
Copy link
Author

@debarshiray , I'll keep investigating the Remote-Containers's approach here. For now, it seems @likan999 's PR approach is the easiest one.

@lbssousa
Copy link
Author

lbssousa commented Nov 5, 2020

I finally realized that, although Toolbox environment are simple OCI containers instantiated by Podman under the hood, they are not expected to be handled the same way a typical application container is, but, instead, more like a WSL distro. Thus, VSCode's Remote-Containers extension may not be the best way to attach to Toolbox containers - we may need another Remote-Toolbox one, similar to Remote-WSL. I'm not sure, however, if such an extension should be developed by Microsoft itself, or if can be developed by the community, since VSCode's Remote Development infrastructure seems to be closed-source. @chrmarti

In the case such an extension to be created, it should be interesting, in Toolbox side, if folder ~/.vscode-server could be bound to an internal volume, so it wouldn't be exposed to host system's user home folder. @debarshiray

@chrmarti chrmarti assigned egamma and Chuxel and unassigned chrmarti Nov 5, 2020
@debarshiray
Copy link

I finally realized that, although Toolbox environment are
simple OCI containers instantiated by Podman under the
hood, they are not expected to be handled the same way
a typical application container is, but, instead, more like a
WSL distro.

Indeed. Toolbox containers are meant to be long-lived so-called pet containers.

In the case such an extension to be created, it should be
interesting, in Toolbox side, if folder ~/.vscode-server could
be bound to an internal volume, so it wouldn't be exposed
to host system's user home folder.

You mean some location on the host OS that's outside $HOME should be bind mounted at ~/.vscode-server inside the container? Sounds doable. What kind of host location do you have in mind?

@HarryMichal
Copy link

Hi! I'm working on Toolbox together with @debarshiray. I just spent a little while debugging this and came up with a few points:

  1. The Toolbox usecase seems to only apply to Remote-Containers: Attach to a running container, therefore there's no need to worry about creating nor starting the container (the magic behind Toolbox is not in the images but in the way the containers are created using toolbox create)
  2. Running the extension in VSCode flatpak is tough but it's doable (mentioning this to give info about my dev env)
  3. As already described, the connection to a toolbox fails with:
Remote-Containers server: Error: open executable: Permission denied: OCI runtime permission denied error

I managed to reproduce this error with the following command:

podman exec -u root golang /root/.vscode-server/bin/d2e414d9e4239a252d1ab117bd7067f125afd80a/server.sh --disable-user-env-probe --use-host-proxy --disable-telemetry --port 0 --extensions-download-dir /root/.vscode-server/extensionsCache

The golang container is a toolbox created with toolbox create golang.

While playing with it, I discovered that the problem can be solved by passing the --tty option to podman exec (but I recommend also passing the -i option to prevent your terminal session getting stuck). This is something that also GNOME Builder does in its Podman plugin.

@lbssousa
Copy link
Author

lbssousa commented Nov 7, 2020

@HarryMichal @chrmarti it seems not to be just a matter of attaching to a running (Toolbox) container. It may be needed to pass additional command-line arguments when attaching to that container. I'm not sure if VSCode's Remote-Containers extension currently supports such a feature (e.g. attach-only devcontainer.json file).

@HarryMichal
Copy link

I'm no expert when it comes to the Remote-Containers extensions as I never got it to work on my system. My use process was based on (from my point of view) the expected workflow:

  1. Click the button in the left bottom corner
  2. Choose an option you think fits your needs the most
  3. Follow the instructions

And yes, it seems it is needed to extend the extension code for the options I mentioned to get toolboxes running. That is completely up to the devs if they see fit to add them, unless somebody else comes up with a different solution (plus maybe a different cause of the problem).

@chrmarti
Copy link
Contributor

chrmarti commented Nov 9, 2020

I don't see why -t and -i would affect the permissions error. Also: This seems to work with other Podman containers. Maybe I'm missing something?

@lbssousa
Copy link
Author

lbssousa commented Nov 10, 2020

@chrmarti is it currently possible to attach to a running container from its container name (not previously built from a Dockerfile or docker-compose.yml) as a non-root user, like we do for application containers setting remoteUser property in devcontainer.json? I think this may be needed in order to attach properly to a Toolbox pet container.

Remote-Containers' documentation says:

select Remote-Containers: Open Named Configuration File... from the Command Palette

but I'm unable to find this option. Instead, I'm only finding Remote-Containers: Open Attached Container Configuration File...

@lbssousa
Copy link
Author

lbssousa commented Nov 10, 2020

@chrmarti @Chuxel @egamma
To be more specific: toolbox run -c my-toolbox [command...] calls, under the hood, a command like this:

podman --log-level error exec \
    --detach-keys= \
    --interactive \
    --tty \
    --user ${USER} \
    --workdir ${HOME} \
    --env=COLORTERM=${COLORTERM} \
    --env=DBUS_SESSION_BUS_ADDRESS=${DBUS_SESSION_BUS_ADDRESS} \
    --env=DESKTOP_SESSION=${DESKTOP_SESSION} \
    --env=DISPLAY=${DISPLAY} \
    --env=LANG=${LANG} \
    --env=SHELL=${SHELL} \
    --env=SSH_AUTH_SOCK=${SSH_AUTH_SOCK} \
    --env=TERM=${TERM} \
    --env=VTE_VERSION=${VTE_VERSION} \
    --env=XDG_CURRENT_DESKTOP=${XDG_CURRENT_DESKTOP} \
    --env=XDG_DATA_DIRS=${XDG_DATA_DIRS} \
    --env=XDG_MENU_PREFIX=${XDG_MENU_PREFIX} \
    --env=XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR} \
    --env=XDG_SESSION_DESKTOP=${XDG_SESSION_DESKTOP} \
    --env=XDG_SESSION_TYPE=${XDG_SESSION_TYPE} \
    --env=TOOLBOX_PATH=/usr/bin/toolbox \
    my-toolbox \
    capsh \
    --caps= \
    -- \
    -c 'exec "$@"' \
    /bin/sh [command ...]

Note: toolbox enter is nearly the same as toolbox run ${SHELL} -l

Is it possible to reproduce such command-line arguments with Remote-Containers extension when attaching to a running container?

@debarshiray
When I try to mimic toolbox run by running the command above explictly in bash prompt after starting my-toolbox container, I've got the following error:

WARN[0000] Failed do add conmon to systemd sandbox cgroup: Unit libpod-conmon-XXXXXXXXXXXXX.scope already exists.
Error: container create failed (no logs from conmon): EOF

Is there something else I should do to prevent this?

Nevermind, it was just an empty string missing as argument to option --detach-keys.

@chrmarti
Copy link
Contributor

You get Remote-Containers: Open Named Configuration File... only while already attached to a container. "remoteUser" also works with attach configs.

The files are in $HOME/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/nameConfigs/${container name}.json (.config/Code\ -\ Insiders/... for VS Code Insiders).

We user docker/podman exec for attaching to an already running container. Are you asking if that can be changed? Currently not.

@lbssousa
Copy link
Author

lbssousa commented Nov 11, 2020

So, in order to Remote-Containers to mimic toolbox enter properly, it should support a file $HOME/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/nameConfigs/${TOOLBOX_CONTAINER_NAME}.json like this:

{
  "remoteUser": "${localEnv:USER}",
  "workspaceFolder": "${localEnv:HOME}",
  "remoteEnv": {
    "TOOLBOX_PATH": "/usr/bin/toolbox",
    "COLORTERM": "${localEnv:COLORTERM}",
    "DBUS_SESSION_BUS_ADDRESS": "${localEnv:DBUS_SESSION_BUS_ADDRESS}",
    "DESKTOP_SESSION": "${localEnv:DESKTOP_SESSION}",
    "DISPLAY": "${localEnv:DISPLAY}",
    "LANG": "${localEnv:LANG}",
    "SHELL": "${localEnv:SHELL}",
    "SSH_AUTH_SOCK": "${localEnv:SSH_AUTH_SOCK}",
    "TERM": "${localEnv:TERM}",
    "VTE_VERSION": "${localEnv:VTE_VERSION}",
    "XDG_CURRENT_DESKTOP": "${localEnv:XDG_CURRENT_DESKTOP}",
    "XDG_DATA_DIRS": "${localEnv:XDG_DATA_DIRS}",
    "XDG_MENU_PREFIX": "${localEnv:XDG_MENU_PREFIX}",
    "XDG_RUNTIME_DIR": "${localEnv:XDG_RUNTIME_DIR}",
    "XDG_SESSION_DESKTOP": "${localEnv:XDG_SESSION_DESKTOP}",
    "XDG_SESSION_TYPE": "${localEnv:XDG_SESSION_TYPE}",
  },
  "execArgs": [
    "capsh",
    "--caps=",
    "--",
    "-c 'exec \"$@\"'",
    "/bin/sh",
    "${localEnv:SHELL}",
    "-l"
  ]
}

(UPDATE 2020-11-12: There's no need to set TOOLBOX_PATH here, because it's already defined in the container's environment. There's also no need to set workspaceFolder, since it already goes to my home folder by default after attaching)

Moreover, it should spawn podman exec command with flags --interactive and --tty.

However, it seems neither ${localEnv:ENV_VAR} nor execArgs are currently supported for attached containers configuration. execArgs support is mandatory here, but the lack of ${localEnv:ENV_VAR} support could be circumvented if toolbox provides a way to export a suitable $HOME/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/nameConfigs/${TOOLBOX_CONTAINER_NAME}.json with those values properly set. @debarshiray (UPDATE 2020-11-12): ${localEnv:VAR} are supported in attached containers configuration for both remoteUser and remoteEnv properties.

When I try to attach to my Toolbox container with such a configuration (setting explicitly my username in remoteUser and my home dir in workspaceFolder), it ends up with the following error:

[2 ms] Remote-Containers 0.148.1 in VS Code 1.51.0 (fcac248b077b55bae4ba5bab613fd6e9156c2f0c).
[11 ms] Start: Resolving remote
[14 ms] Setting up container fedora-toolbox-33

[15 ms] Start: Run: podman inspect --type container fedora-toolbox-33
[83 ms] Start: Check Docker is running
[84 ms] Start: Run: podman info
[180 ms] Start: Inspecting container
[180 ms] Start: Run: podman inspect --type container 9c9e9954f7a86b2b181eeb3fa98bc1f8151d30adf3979577e9e08b954dc35358
[236 ms] Start: Run: podman exec -i -u laercio -e VSCODE_REMOTE_CONTAINERS_SESSION=7be72b4f-5d91-48a7-9f0a-c4cd512302ef1605100356707 9c9e9954f7a86b2b181eeb3fa98bc1f8151d30adf3979577e9e08b954dc35358 /bin/sh
[239 ms] Start: Run in container: uname -m
[325 ms] x86_64
[326 ms] 
[326 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[330 ms] NAME=Fedora
VERSION="33 (Container Image)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
[330 ms] 
[331 ms] Start: Run in container: cat /etc/passwd
[333 ms] Start: Run in container: test -d /root/.vscode-server
[334 ms] 
[334 ms] 
[334 ms] Exit code 1
[334 ms] Start: Run in container: test -d /root/.vscode-remote
[335 ms] 
[335 ms] 
[335 ms] Exit code 1
[335 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[337 ms] 
[337 ms] mkdir: cannot create directory '/root': Permission denied
[337 ms] Exit code 1
[338 ms] Start: Run in container: test -d /root/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c
[339 ms] 
[339 ms] 
[339 ms] Exit code 1
[339 ms] Installing VS Code Server for commit fcac248b077b55bae4ba5bab613fd6e9156c2f0c
[339 ms] Start: Run in container: mkdir -p /root/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c_1605100357608
[345 ms] 
[346 ms] mkdir: cannot create directory '/root': Permission denied
[346 ms] Exit code 1
[349 ms] Command in container failed: mkdir -p /root/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c_1605100357608

The problem here is that Remote-Containers is trying to install Remote VSCode server at /root/.vscode-server rather than $HOME/.vscode-server. Is there a way to make it work?

@chrmarti
Copy link
Contributor

chrmarti commented Nov 12, 2020

It looks up HOME first on the container's environment variables (using inspect) and then in /etc/passwd for that user. One of the two must be /root here, try updating that.

${localEnv:VAR} also works for attach configs, maybe HOME and/or USER are not set for the VS Code processes?

We don't have execArgs for any of our configs currently.

@lbssousa
Copy link
Author

lbssousa commented Nov 12, 2020

In fact, after running podman inspect ${TOOLBOX_CONTAINER_NAME}, I've found HOME=/root in its environment. After giving write permission to container's /root folder for my user, I was finally able to attach to my Toolbox container.

I've verified that ${localEnv:VAR} is working properly - in fact, even "remoteUser": "${localEnv:USER}" works. It only fails if I try to set "workspaceFolder": "${localEnv:HOME}", but I just realized that it's not needed at all - after removing that line from my config, Remote-Containers goes to my home folder by default after attaching to container.

I was still unable to pass arguments capsh --caps= -- -c 'exec "$@"' /bin/sh [my-shell] to podman exec. Nevertheless, I've got correctly my shell prompt after attaching to my Toolbox container. @debarshiray , is this really needed?

In short, there seems some missing pieces in Remote-Containers extension to make my life easier when attaching to a Toolbox container:

  • Provide some way to set an alternative destination for .vscode-server folder, at least when remoteUser is set (raising the precedence of remoteUser's home folder as defined in /etc/passwd over container's HOME environment variable would be enough)
  • Provide some way to allow one to pass extra command-like arguments when spawning podman exec (...) [my-shell]. "postAttachCommand": "capsh --caps= -- -c 'exec "$@"' /bin/sh [my-shell]" didn't work here. It can be achieved via properties terminal.integrated.shell.linux and terminal.integrated.shellArgs.linux (see below)

@chrmarti
Copy link
Contributor

You could unset HOME with:

	"containerEnv": {
		"HOME": ""
	}

in the devcontainer.json.

@lbssousa
Copy link
Author

lbssousa commented Nov 12, 2020

Unsetting HOME in containerEnv had no effect here: it still tries to install .vscode-server under /root. VSCode's documentation says the container needs to be rebuilt for this to take effect, which is out of question for my case.

About those capsh ... arguments, I've found a workaround. I've just added these settings to attached container config:

{
  "settings": {
    "terminal.integrated.shell.linux": "/usr/sbin/capsh",
    "terminal.integrated.shellArgs.linux": [
      "--caps=",
      "--",
      "-c",
      "exec \"$@\"",
      "/bin/sh",
      "${localEnv:SHELL}",
      "-l"
    ]
  }
}

One more suggestion to make my life easier is allowing access to command Remote-Containers: Open Named Configuration File... in command palette prior to attaching to container, or just providing a similar command that allows me to create an attach config before attaching to container.

@chrmarti
Copy link
Contributor

Did you rebuild the container after changing containerEnv? I had unsetting working, try setting it to the desired path if it doesn't.

@lbssousa
Copy link
Author

lbssousa commented Nov 13, 2020

No, I didn't rebuilt my container after setting "containerEnv". I tried both empty string and my home folder path, but none of tem has worked.

TBH, I'm not worried about this anymore: keeping .vscode-server under /root folder within the container allows me not leaking it to my host system's home folder.

@lbssousa
Copy link
Author

lbssousa commented Nov 14, 2020

There's one last thing that still makes me unconfortable with this solution: since Toolbox is a very special case of container-based development, in which host system's user home folder is directly bound to the container, Remote-Containers extension ends up overwriting my ~/.gitconfig's credential.helper value, replacing the original value (e.g. store) with something like "!f() { /root/.vscode-server/bin/e5a624b788d92b8d34d1392e4c4d9789406efe8f/node /tmp/vscode-remote-containers-0f1183fad6e028a06ad0d6f114ae48cb785a1d17.js $*; }; f". Is there a way to prevent this?

@chrmarti
Copy link
Contributor

There is a Remote-Containers setting to turn this off.

@lbssousa
Copy link
Author

lbssousa commented Nov 16, 2020

Oh, I've just found it here. Thanks!

For anyone interested, I've summarized my configuration to make Remote-Containers work with my Toolbox container here.

@debarshiray
Copy link

It's really amazing that you have managed to get this far!

The part about HOME being set to /root makes me curious. A Toolbox container uses two UIDs. One is UID 0. This isn't the actual UID 0 on the host, but some other UID on the host mapped to 0 inside the container's user namespace. This is used to run the container's entry point. I believe this is why you see the HOME=/root. The other is the user's own UID which is the same both on the host and in the container. This is what is used to present the shell to the user.

Is Visual Studio Code going by the HOME it sees in the podman inspect output when trying to place the .vscode-server directory?

@chrmarti
Copy link
Contributor

@debarshiray Yes, we first look at HOME from podman inspect. If that is not set we look up it in /etc/passwd. Note that we install and run the server as the remoteUser which might run into permission issues under /root.

@lbssousa
Copy link
Author

Does Remote-Containers extension currently support property name for attached container config? I'm trying to customize my Toolbox display name in VSCode, but it's not working.

If not, could you amend it to issue #4013?

@lbssousa
Copy link
Author

lbssousa commented Nov 19, 2020

I guess we have now all the needed information for an eventual solution on the Toolbox side (auto-generating the needed configuration for Remote-Containers). I've opened an issue there. If @debarshiray has no other questions, I believe this issue can be closed now.

@alvarlagerlof
Copy link

Sorry for the ping, but I'm seeing a lot of issues here and I'm having trouble making sense of what works and what doesn't. Is there an updated guide/text on what works and how to set that up?

@lbssousa
Copy link
Author

@alvarlagerlof The most up-to-date solution about this issue is a third-party project: https://github.com/owtaylor/toolbox-vscode

@alvarlagerlof
Copy link

@alvarlagerlof The most up-to-date solution about this issue is a third-party project: https://github.com/owtaylor/toolbox-vscode

That does not work anymore. Any alternatives?

@heyakyra
Copy link

heyakyra commented Jan 15, 2022

This might finally get me to switch to kakoune Helix / Lapce with zellij + tmate

@daw1012345
Copy link

Almost 3 years after this issue was originally created I ran into problems trying to use Toolbox containers and VScode together. https://github.com/owtaylor/toolbox-vscode doesn't seem to be working and other articles on the topic caused other (permission) issues for me. I decided to write an updated script that works well enough for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
containers Issue in vscode-remote containers feature-request Request for new features or functionality
Projects
None yet
Development

No branches or pull requests