podman 3.4.0 breaks support for MacPorts' gvproxy location

[jre21]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

When I updated my MacPorts-installed podman from 3.3.1 to 3.4.0, podman commands started failing with a message about being unable to find gvproxy.

This is a regression, as the podman 3.3.1 client had been interacting with podman machine as expected. For reference,

Steps to reproduce the issue:

1. Ensure /opt/local/bin is on $PATH
2. Move the gvproxy binary to /opt/local/bin
3. Run podman machine start

Describe the results you received:

Podman errors out with

Error: unable to start host networking: "could not find \"gvproxy\" in one of [/usr/local/opt/podman/libexec /opt/homebrew/bin /opt/homebrew/opt/podman/libexec /usr/local/bin /usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman]"

Describe the results you expected:

I expect podman to function normally in this situation. I can think of a few possible ways to accomplish this, in rough order of preference:

1. Have podman honor $PATH when searching for the gvproxy executable. This is the most flexible approach and gets podman out of the business of needing to guess how users have organized their directory structure.
2. Check for gvproxy in the parent directory of the podman executable. This would solve many of the most common installation patterns with a potentially smaller blast radius than the above.
3. Allow the user to configure the location of the gvproxy binary, and explain how to set this configuration in the error message. This would add some friction for new users, but would at least be a one-time set-and-forget deal.
4. Add /opt/local/bin/ to the list of search locations. This unblocks MacPorts users, but presents the risk of podman developers being expected to individually support the long tail of bespoke package managers.

Additional information you deem important (e.g. issue happens only occasionally):

/opt/local/bin is where MacPorts installs all system executables (including podman and gvproxy).

Output of podman version:

This issue breaks basic functionality of the podman client and results in podman version not returning useful output.

Output of podman info --debug:

Same as above.

Package info (e.g. output of rpm -q podman or apt list podman):

$ port installed podman
The following ports are currently installed:
  podman @3.3.1_1
  podman @3.4.0_0 (active)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

macOS 11.6 on bare metal (using podman machine's VM, prior to this bug appearing)

[MikeNicholls]

I had a similar problem with Homebrew, I saw that #11449 made the location configurable so I was able to fix it on my machine by editing ~/.config/containers/containers.conf to add the directory containing gvproxy to helper_binaries_dir.

Does this solve it for you @jre21?

[engine]
  helper_binaries_dir = [
  	"/opt/local/bin"
  ]

[Luap99]

We do not want gvproxy in $PATH. gvproxy is a binary to forward ports for podman machine. It should not be used as a normal user. If the package manager installs it in a different path it should also change the default containers.conf or overwrite the the default directory list at compile time.

[rhatdan]

Is this us packaging this or someone else?

[jre21]

Thanks, MikeNicholls, that solved the immediate problem.

@Luap99 I don't feel that my concerns have been fully addressed. Regardless of the podman developers' wishes, some packagers are going to place gvproxy somewhere the developers didn't intend. Right now this leads to an extremely poor user experience where we're told what's wrong (podman can't find gvproxy) but not how to fix it. Including some note in the error message which guides users toward the above configuration option would help immensely with empowering users to solve their own environment issues. And yes, ideally packagers (in this case MacPorts) would do this on users' behalf, but when that doesn't happen a friendly pointer goes much farther than a merely descriptive error message.

[MikeNicholls]

Is this us packaging this or someone else?

Looks like @judaew is the maintainer.
https://ports.macports.org/port/podman/details/

[Luap99]

@jre21 How should the error look like in your opinion?

[chadcatlett]

@Luap99 my thought is containers/common/pkg/config could be changed to also search filesystem locations that MacPorts manages too.

I understand not wanting gvproxy in the main search path for a user, so having the config package look for it in MacPort's libexec directory, the gvisor port could also be updated to install it to that location.

[judaew]

Hi, I'm a maintainer podman port in MacPorts.

Thanks @chadcatlett, I've already moved gvisor to ${prefix}/libexec directory in a recent revision.

I have added a patch for containers/common/pkg/config/config_darwin.go where the paths for MacPorts are defined at the top. Since in MacPorts /opt/local path is the default value for ${prefix}, but can be changed by the user, I will not suggest this patch as a PR for podman.

[jre21]

For the error message, I'd like to see something like "If it is installed in a different location, set helper_binaries_dir in the [engine] section of containers.conf." after printing the current search locations. Another possibility would be inserting a link to a documentation page which explains the issue in more detail.

[Luap99]

@jre21 Interested in opening a PR to change the message?
You can find the code here: https://github.com/containers/common/blob/4077ae5946112560044d5f1cb5876ac0188a77b2/pkg/config/config.go#L1154-L1158

[marchmallow]

Folks.. I read all this and still not clear how to configure my mac properly to run podman, this is where brew installs it:

$ brew list podman
Warning: Treating podman as a formula.
/usr/local/Cellar/podman/3.4.1/bin/podman
/usr/local/Cellar/podman/3.4.1/bin/podman-remote
/usr/local/Cellar/podman/3.4.1/etc/bash_completion.d/podman
/usr/local/Cellar/podman/3.4.1/libexec/gvproxy
/usr/local/Cellar/podman/3.4.1/share/fish/vendor_completions.d/podman.fish
/usr/local/Cellar/podman/3.4.1/share/man/ (160 files)
/usr/local/Cellar/podman/3.4.1/share/zsh/site-functions/_podman

brew does not add it to the $PATH

So I added /usr/local/Cellar/podman/3.4.1/bin/ to my $PATH, after reading discussion here then I added path to gvproxy to ~/.config/containers/containers.conf which now looks like following:

[containers]

[engine]
  active_service = "podman-machine-default"
  [engine.service_destinations]
    [engine.service_destinations.podman-machine-default]
      uri = "ssh://core@localhost:50173/run/user/1000/podman/podman.sock"
      identity = "/Users/gallomas/.ssh/podman-machine-default"
    [engine.service_destinations.podman-machine-default-root]
      uri = "ssh://root@localhost:50173/run/podman/podman.sock"
      identity = "/Users/gallomas/.ssh/podman-machine-default"
  helper_binaries_dir = [
        "/usr/local/Cellar/podman/3.4.1/libexec/"
  ]

Now failure is:

$ podman machine start
WARN[0000] Failed to decode the keys ["engine.service_destinations.podman-machine-default-root.helper_binaries_dir"] from "/Users/gallomas/.config/containers/containers.conf". 
WARN[0000] Failed to decode the keys ["engine.service_destinations.podman-machine-default-root.helper_binaries_dir"] from "/Users/gallomas/.config/containers/containers.conf". 
WARN[0000] Failed to decode the keys ["engine.service_destinations.podman-machine-default-root.helper_binaries_dir"] from "/Users/gallomas/.config/containers/containers.conf". 
Error: unable to start host networking: "could not find \"gvproxy\" in one of [/usr/local/opt/podman/libexec /opt/homebrew/bin /opt/homebrew/opt/podman/libexec /usr/local/bin /usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman]"

So it seems I did not get this container conf right but where do I find the documentation to understand how this config needs to be edited? Not very clear to me how it expects entries to be added

[marchmallow]

So I finally figured the config.. looked at doc here: https://github.com/containers/common/blob/main/docs/containers.conf.5.md

And changed it to:

[containers]

[engine]
  helper_binaries_dir=["/usr/local/Cellar/podman/3.4.1/bin","/usr/local/Cellar/podman/3.4.1/libexec"]
  active_service = "podman-machine-default"
  [engine.service_destinations]
    [engine.service_destinations.podman-machine-default]
      uri = "ssh://core@localhost:50173/run/user/1000/podman/podman.sock"
      identity = "/Users/gallomas/.ssh/podman-machine-default"
    [engine.service_destinations.podman-machine-default-root]
      uri = "ssh://root@localhost:50173/run/podman/podman.sock"
      identity = "/Users/gallomas/.ssh/podman-machine-default"

[baude]

So I added /usr/local/Cellar/podman/3.4.1/bin/ to my $PATH, after reading discussion here then I added path to gvproxy to ~/.config/containers/containers.conf which now looks like following

We did not want gvproxy to be part of the PATH. This was done intentionally.