errors with varlink via systemd

[cevich]

/kind bug

Description

When executing in a tight loop, or multiple simultaneous podman-remote throws EOF errors.

Steps to reproduce the issue:

1. make podman podman-remote
2. make install PREFIX=/usr
3. systemctl daemon-reload
4. systemctl enable io.podman.socket
5. systemctl enable io.podman.service
6. systemctl start io.podman.socket
7. while podman-remote run -it --rm alpine echo "hi"; do :; done

Describe the results you received:

Either:

hi
...cut several repeats...
hi
Error: unexpected EOF
...

or

hi
...cut several repeats...
hi
Error: Virtual Read failed, 0: EOF
...

Describe the results you expected:

A continuous stream of hi and no errors.

Additional information you deem important (e.g. issue happens only occasionally):

Found while discovering/debugging issue #3998

Also sometimes reproducible with:
podman-remote run --rm alpine echo "hello" & podman-remote run --rm alpine echo "hello"

Output of podman version:

(paste your output here)

Output of podman info --debug:

# podman-remote version
Client:
Version:            1.5.2-dev
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Service:
Version:            1.5.2-dev
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Package info (e.g. output of rpm -q podman or apt list podman):

af8fedc

Additional environment details (AWS, VirtualBox, physical, etc.):

Reproduced on a VM with 2 vCPUs (no threads) and 4 GiB memory.

[vrothberg]

I can reproduce immediately. Interestingly, I cannot reproduce when avoiding the socket activation and running sudo podman varlink -t 0 unix:/run/podman/io.podman directly.

[vrothberg]

I get a 100 percent error rate when doing a podman-remote run alpine echo "hi" > /dev/null.
...that was a red herring.

[vrothberg]

It looks like the logic to disable sdnotify was wrong. Opened #4089 to address the issue. It fixes the issues locally but lets wait what the CI has to say.

[cevich]

Oh this is fantastic! Thanks @vrothberg 😺