Issue with port forward in corporate proxy envinronment

[kishorekkota]

/kind bug

Description

PODMAN 4.x seems to have addressed the issue with PODMAN Machine carrying the proxy setting local machine and set it in the PODMAN VM Machine. With this release, it did address the issue around Connection Timeout thats seen while running PODMAN Machine in Mac OS.

But this introduced another issue, that could be specific how my company works.

I am able to run PODMAN container, but when trying to set port forwarding, some of the steps needed to perform port forwarding are routed thru PROXY for some reason and they are failing. My company does not allow traffic to pass thru if the HTTP Traffic is pointed to an API that does not classify content categorization.

Here are the logs between simply running container vs running container with port forwarding.

Running the container alone.

DEBU[0014] DoRequest Method: GET URI: http://d/v4.0.2/libpod/containers/7154e7a416e61846852edd6de6eff5cafc77bee2f3a3e658376f46d70d593fae/json
DEBU[0014] DoRequest Method: POST URI: http://d/v4.0.2/libpod/containers/7154e7a416e61846852edd6de6eff5cafc77bee2f3a3e658376f46d70d593fae/attach
DEBU[0015] Copying standard streams of container "7154e7a416e61846852edd6de6eff5cafc77bee2f3a3e658376f46d70d593fae" in non-terminal mode
INFO[0015] Going to start container "7154e7a416e61846852edd6de6eff5cafc77bee2f3a3e658376f46d70d593fae"
DEBU[0015] DoRequest Method: POST URI: http://d/v4.0.2/libpod/containers/7154e7a416e61846852edd6de6eff5cafc77bee2f3a3e658376f46d70d593fae/start
2022-03-23 22:50:35.440491+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
2022-03-23 22:50:35.448228+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings
2022-03-23 22:50:35.448257+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status
2022-03-23 22:50:35.448276+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue

With Port Forwarding

DEBU[0000] Found credentials for docker-upstreams-virtual.artifactory.discoverfinancial.com in credential helper containers-auth.json in file /Users/kkota/.config/containers/auth.json
DEBU[0000] DoRequest Method: POST URI: http://d/v4.0.2/libpod/images/pull
DEBU[0000] DoRequest Method: POST URI: http://d/v4.0.2/libpod/containers/create
INFO[0000] Going to attach to container "179ae42d671ab3c952d35d9e345f8dec751d9b7143f5f4a34ecaac6a1413e726"
DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.2/libpod/containers/179ae42d671ab3c952d35d9e345f8dec751d9b7143f5f4a34ecaac6a1413e726/json
DEBU[0000] DoRequest Method: POST URI: http://d/v4.0.2/libpod/containers/179ae42d671ab3c952d35d9e345f8dec751d9b7143f5f4a34ecaac6a1413e726/attach
DEBU[0000] ExitCode msg: "error preparing container 179ae42d671ab3c952d35d9e345f8dec751d9b7143f5f4a34ecaac6a1413e726 for attach: something went wrong with the request: "\r\n <title> access denied

Steps to reproduce the issue:

1. PODMAN Machine INIT

2. PODMAN Machine start

3. podman run --log-level=debug -p 5672:5672 -p 15672:15672 rabbitmq

Describe the results you received:

Container did not start as expected.

Describe the results you expected:

Container up and running.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.8

Built:      Wed Mar  2 08:04:36 2022
OS/Arch:    darwin/amd64

Server:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.16.14

Built:      Thu Mar  3 08:56:56 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 2080018555
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.15.18-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 905478144
  memTotal: 2061381632
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/2080018555/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 32m 40.09s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker-upstreams-virtual.artifactory.discoverfinancial.com:
    Blocked: false
    Insecure: true
    Location: docker-upstreams-virtual.artifactory.discoverfinancial.com
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: docker-upstreams-virtual.artifactory.discoverfinancial.com
  search:
  - docker-upstreams-virtual.artifactory.discoverfinancial.com
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 1
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/2080018555/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319416
  BuiltTime: Thu Mar  3 08:56:56 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Mac OS

[Luap99]

I think I know the problem since the port forwarding also uses http to talk to the gvproxy api it will get redirect to the proxy which cannot connect to gvproxy.

Should be easy to fix, we need to make this http request does not use a proxy.

[kishorekkota]

@Luap99 Thank you.

[kishorekkota]

Do i need to wait for a release or something i can try it again and see if this is resolved ? When i can get this version of Podman ?

[rhatdan]

We will probably back port this and release podman 4.0.3 early next week.

[kishorekkota]

thanks a bunch

[rhatdan]

@kishorekkota If you want you can open a cherrypick your self to get it into the 4.0 branch.

[kishorekkota]

@rhatdan Are there nightly builds that get released - if i do so ?

[rhatdan]

No, sorry

[kishorekkota]

Still seeing the same issue even with PODMAN 4.0.3.

What does 'd' meant in the below URL ? Does that get translated some how or what is the actual URL supposed to be ?

http://d/v4.0.2/libpod/containers/179ae42d671ab3c952d35d9e345f8dec751d9b7143f5f4a34ecaac6a1413e726/json

[Luap99]

The version inside the VM needs the update

[kishorekkota]

that is still showing 4.0.2. I have deleted VM image and did the init again. How do i force the PODMAN update in the VM ?

[Luap99]

you have to wait until it lands in coreos image
@baude Did you update the images with 4.0.3?

[kishorekkota]

Any update for me. My team is struggling with this issue and any help is greatly appreciated.