Error: `podman compose up` ends up with `dial tcp: lookup unix on 127.0.0.53:53: server misbehaving` #4573

odockal · 2023-10-31T16:22:39Z

Bug description

Could be also a question at the moment. I was following up the workflow to onboard a compose via Podman Desktop and then also (as is written in the workflow) enable Docker Compatibility mode to have all pieces to be able to podman compose up.

Operating system

Fedora 38

Installation Method

Other

Version

next (development version)

Steps to reproduce

Follow compose onboarding
Download and install Compose system-wide via Podman Desktop -> Settings -> Resources -> Compose -> Setup
Assert: docker-compose tool is installed (aka, on PATH, try docker-compose version)
Press Docker compatibility option
Enable podman systemd socket on dialog when asked
Do you want to create a symlink from /run/podman/podman.sock to /var/run/docker.sock to enable Docker compatibility without having to set the DOCKER_HOST environment variable? -> Yes
Assert: Message appears Symlink created successfully. The Podman socket is now available at /var/run/docker.sock.
Assert: Message appears: Podman systemd socket has been enabled for Docker compatibility.
Assert: I think that this message should disappear from Dashboard: Docker Socket Compatibility: Docker socket is not reachable. Docker specific tools may not work. but I still can see it
Open cli: go to some compose repo (awesome-docker/flask) and run: podman compose up
Assert: docker-compose is used by podman compose and containers are spinned up - can be seen from podman desktop
Actual result: dial tcp: lookup unix on 127.0.0.53:53: server misbehaving

Relevant log output

>>>> Executing external compose provider "/usr/local/bin/docker-compose". Please refer to the documentation for details. <<<<

error during connect: Get "http://unix:2375/run/user/1000/podman/podman.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dflask%22%3Atrue%7D%7D": dial tcp: lookup unix on 127.0.0.53:53: server misbehaving
Error: executing /usr/local/bin/docker-compose up: exit status 1

Additional context

ls -al /var/run/docker.sock
lrwxrwxrwx. 1 root root 23 Oct 31 17:16 /var/run/docker.sock -> /run/podman/podman.sock

No DOCKER_HOST set - which is alright I think.

docker-compose up
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dflask%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied

systemctl status podman.socket
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; preset: disabled)
     Active: active (listening) since Tue 2023-10-31 17:13:40 CET; 7min ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/podman/podman.sock (Stream)
     CGroup: /system.slice/podman.socket

Oct 31 17:13:40 fedora systemd[1]: Listening on podman.socket - Podman API Socket.

systemctl status podman.service              
○ podman.service - Podman API Service
     Loaded: loaded (/usr/lib/systemd/system/podman.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) since Tue 2023-10-31 15:35:19 CET; 1h 46min ago
   Duration: 6.316s
TriggeredBy: ● podman.socket
       Docs: man:podman-system-service(1)
   Main PID: 1378 (code=exited, status=0/SUCCESS)
        CPU: 540ms

Oct 31 15:35:12 fedora systemd[1]: Starting podman.service - Podman API Service...
Oct 31 15:35:13 fedora podman[1378]: time="2023-10-31T15:35:13+01:00" level=info msg="/usr/bin/podman >
Oct 31 15:35:13 fedora systemd[1]: Started podman.service - Podman API Service.
Oct 31 15:35:14 fedora podman[1378]: time="2023-10-31T15:35:14+01:00" level=info msg="Not using native>
Oct 31 15:35:14 fedora podman[1378]: 2023-10-31 15:35:14.372144869 +0100 CET m=+1.341164970 system ref>
Oct 31 15:35:14 fedora podman[1378]: time="2023-10-31T15:35:14+01:00" level=info msg="Setting parallel>
Oct 31 15:35:14 fedora podman[1378]: time="2023-10-31T15:35:14+01:00" level=info msg="Using systemd so>
Oct 31 15:35:14 fedora podman[1378]: time="2023-10-31T15:35:14+01:00" level=info msg="API service list>
Oct 31 15:35:19 fedora systemd[1]: podman.service: Deactivated successfully.

The text was updated successfully, but these errors were encountered:

odockal · 2023-10-31T16:23:14Z

Anyone could follow this: https://github.com/odockal/podman-desktop-qe/blob/main/docs/scenarios/compose/compose-testing.md

afbjorklund · 2023-10-31T19:33:33Z

Sounds like something in podman is still using the unix: syntax, which is unfortunately not supported by Docker tools.

It requires the unix:// syntax, even though it is not using any auth. It was patched in containers/podman@336055f

unix:/run/user/1000/podman/podman.sock

unix:///run/user/1000/podman/podman.sock

afbjorklund · 2023-10-31T19:39:11Z

It is the registry.DefaultAPIAddress that still uses the old syntax, should probably be changed in podman compose.

        // For local clients (Linux/FreeBSD), use the default API
        // address.
        if !registry.IsRemote() {
                return registry.DefaultAPIAddress(), nil
        }

The workaround is to set it explicitly: export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock

afbjorklund · 2023-10-31T19:46:26Z

The "permission denied" from docker-compose is normal, it needs to run with sudo unless group is changed.

Normal docker and podman-remote* will get the same error, since podman.sock is owned by the root group.

* i.e. when it is run with CONTAINER_HOST=unix:/run/podman/podman.sock, not the rootless version

Implement option to view rootful containers on Linux 🐧 #2861 (comment)

dgolovin · 2023-11-01T04:10:08Z

@odockal I have the same issues on my Fedora 38. What @afbjorklund suggested as workaround works for me.
I can ether export DOCKER_HOST and run podman compose up, or just run sudo docker-compose up,

odockal · 2023-11-02T08:34:50Z

@dgolovin Thank you for reproduction. I recall that same workaround worked for me.
@afbjorklund Thanks for clarification!

If I understand it right, the 6th step in "Steps to reproduce" is then not working as expected...

afbjorklund · 2023-11-02T09:11:07Z

I don't think the issue is related to step 6 (the symlink), since it will override the default location.

i.e. if it hadn't exported DOCKER_HOST, then docker and docker-compose would have worked...

$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               ERROR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

But since it was exporting a syntax that Docker didn't like, it failed. Fixed on main, podman 4.8?

afbjorklund · 2023-11-02T09:32:05Z

Here is the expected output:

$ docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:36 2023
 OS/Arch:           linux/amd64
 Context:           default
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied
$ podman --remote --url unix:/run/podman/podman.sock version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.7.0/libpod/_ping": dial unix ///run/podman/podman.sock: connect: permission denied

Since it requires using sudo:

[anders@lima-fedora anders]$ sudo docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:36 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:08 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.7
  GitCommit:        8c087663b0233f6e6e2f4515cee61d49f14746a8
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfcb
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[anders@lima-fedora anders]$ sudo podman --remote version
Client:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.20.8
Built:        Wed Sep 27 18:24:38 2023
OS/Arch:      linux/amd64

Server:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.20.8
Built:        Wed Sep 27 18:24:3

https://docs.docker.com/engine/install/fedora/

And it seems to be bug compatible:

[anders@lima-fedora compose-start]$ docker compose start
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dcompose-start%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied
[anders@lima-fedora compose-start]$ DOCKER_HOST=unix:///run/podman/podman.sock podman compose start
>>>> Executing external compose provider "/usr/local/bin/docker-compose". Please refer to the documentation for details. <<<<

permission denied while trying to connect to the Docker daemon socket at unix:///run/podman/podman.sock: Get "http://%2Frun%2Fpodman%2Fpodman.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dcompose-start%22%3Atrue%7D%7D": dial unix /run/podman/podman.sock: connect: permission denied
Error: executing /usr/local/bin/docker-compose start: exit status 1

https://docs.docker.com/compose/install/standalone/

Docker Compose version v2.23.0

afbjorklund · 2023-11-02T09:40:32Z

Not sure why it is not running rootless by default?

dockerd-rootless-setuptool.sh install

systemctl --user start podman.socket

Then it wouldn't need to mess with sudo and groups...

export DOCKER_HOST=unix:///run/user/1000/docker.sock

export CONTAINER_HOST=unix:/run/user/1000/podman/podman.sock

Seems like Podman only creates "connections" for Machine:

[anders@lima-fedora ~]$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT                     ERROR
default *   Current DOCKER_HOST based configuration   unix:///run/user/1000/docker.sock   
rootless    Rootless mode                             unix:///run/user/1000/docker.sock   
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.
[anders@lima-fedora ~]$ podman system connection ls
Name        URI         Identity    Default

An ugly workaround is to run the remote client as root too:

sudo podman-remote

lstocchi · 2024-05-07T10:59:44Z

@odockal just gave a look and i think this can be closed as the docker compatibility button has been disabled on linux? #5903
Tried to setup compose and seems to work fine but i cannot replicate your steps due to this change

odockal · 2024-05-10T10:42:19Z

@lstocchi Yes. The option is now gone, so I believe we can close it.

lstocchi · 2024-05-10T12:46:50Z

@odockal thanks for checking/confirming 👍

odockal added kind/bug 🐞 Something isn't working os/linux 🐧 area/extensions/compose 🎹 area/docker-compatibility 🐳 labels Oct 31, 2023

odockal changed the title ~~podman compose up ends up with dial tcp: lookup unix on 127.0.0.53:53: server misbehaving~~ Error: podman compose up ends up with dial tcp: lookup unix on 127.0.0.53:53: server misbehaving Nov 1, 2023

chmanie mentioned this issue Nov 13, 2023

Improve docker compatibility JoinColony/colonyCDapp#1316

Merged

jeffmaury self-assigned this Apr 24, 2024

nichjones1 assigned lstocchi and unassigned jeffmaury Apr 24, 2024

nichjones1 assigned odockal and unassigned lstocchi May 9, 2024

odockal closed this as completed May 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: `podman compose up` ends up with `dial tcp: lookup unix on 127.0.0.53:53: server misbehaving` #4573

Error: `podman compose up` ends up with `dial tcp: lookup unix on 127.0.0.53:53: server misbehaving` #4573

odockal commented Oct 31, 2023

odockal commented Oct 31, 2023

afbjorklund commented Oct 31, 2023

afbjorklund commented Oct 31, 2023 •

edited

Loading

afbjorklund commented Oct 31, 2023 •

edited

Loading

dgolovin commented Nov 1, 2023

odockal commented Nov 2, 2023

afbjorklund commented Nov 2, 2023

afbjorklund commented Nov 2, 2023 •

edited

Loading

afbjorklund commented Nov 2, 2023 •

edited

Loading

lstocchi commented May 7, 2024

odockal commented May 10, 2024

lstocchi commented May 10, 2024

Error: podman compose up ends up with dial tcp: lookup unix on 127.0.0.53:53: server misbehaving #4573

Error: podman compose up ends up with dial tcp: lookup unix on 127.0.0.53:53: server misbehaving #4573

Comments

odockal commented Oct 31, 2023

Bug description

Operating system

Installation Method

Version

Steps to reproduce

Relevant log output

Additional context

odockal commented Oct 31, 2023

afbjorklund commented Oct 31, 2023

afbjorklund commented Oct 31, 2023 • edited Loading

afbjorklund commented Oct 31, 2023 • edited Loading

dgolovin commented Nov 1, 2023

odockal commented Nov 2, 2023

afbjorklund commented Nov 2, 2023

afbjorklund commented Nov 2, 2023 • edited Loading

afbjorklund commented Nov 2, 2023 • edited Loading

lstocchi commented May 7, 2024

odockal commented May 10, 2024

lstocchi commented May 10, 2024

Error: `podman compose up` ends up with `dial tcp: lookup unix on 127.0.0.53:53: server misbehaving` #4573

Error: `podman compose up` ends up with `dial tcp: lookup unix on 127.0.0.53:53: server misbehaving` #4573

afbjorklund commented Oct 31, 2023 •

edited

Loading

afbjorklund commented Oct 31, 2023 •

edited

Loading

afbjorklund commented Nov 2, 2023 •

edited

Loading

afbjorklund commented Nov 2, 2023 •

edited

Loading