Socket Broken Pipe

[sumit0900]

I tried running the code that exists in podman/api_connection.py and facing a Socket broken pipe issue.
Below is the stack trace for the error observed. Is there any workaround to fix this?
Traceback (most recent call last):
File "", line 1, in
File "", line 35, in request
File "/usr/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 971, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

[mwhahaha]

What version of podman are you using? Also can you provide additional information about the connection string being used and how you are setting up the podman api service?

[sumit0900]

podman --version

podman version 2.0.5

Below is the code I am trying to run which includes the connection string

import json
import logging
import socket
import time
import urllib.parse
from contextlib import AbstractContextManager
from http import HTTPStatus
from http.client import HTTPConnection
class ApiConnection(HTTPConnection, AbstractContextManager):
    """
    ApiConnection provides a specialized HTTPConnection
    to a Podman service.
    """
    def __init__(self, url, base="/v2.0.0/libpod", *args, **kwargs):  # pylint: disable-msg=W1113
        if url is None or not url:
            raise ValueError("url is required for service connection.")
        super().__init__("localhost", *args, **kwargs)
        supported_schemes = ("unix", "ssh")
        uri = urllib.parse.urlparse(url)
        if uri.scheme not in supported_schemes:
            raise ValueError(
                "The scheme '{}' is not supported, only {}".format(uri.scheme, supported_schemes)
            )
        self.uri = uri
        self.base = base
    def connect(self):
        """Connect to the URL given when initializing class"""
        if self.uri.scheme == "unix":
            sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
            sock.connect(self.uri.path)
            self.sock = sock
        else:
            raise NotImplementedError("Scheme {} not yet implemented".format(self.uri.scheme))
    def request(self, method, url, body=None, headers=None, *, encode_chunked=False):
        """Make request to Podman service."""
        if headers is None:
            headers = {}
        super().request(method, url, body, headers, encode_chunked=encode_chunked)
        response = super().getresponse()
        # Errors are mapped to exceptions
        if HTTPStatus.OK <= response.status < HTTPStatus.MULTIPLE_CHOICES:
            pass
        elif HTTPStatus.NOT_FOUND == response.status:
            print(
                "Request {}:{} failed: {}".format(
                    method,
                    url,
                    HTTPStatus.NOT_FOUND.description or HTTPStatus.NOT_FOUND.phrase,
                ),
                response,
            )
        elif (
            response.status >= HTTPStatus.BAD_REQUEST
            and response.status < HTTPStatus.INTERNAL_SERVER_ERROR
        ):
            print(
                "Request {}:{} failed: {}".format(
                    method,
                    url,
                    response.reason or "Response Status Code {}".format(response.status),
                ),
                response,
            )
        elif response.status >= HTTPStatus.INTERNAL_SERVER_ERROR:
            print(
                "Request {}:{} failed: {}".format(
                    method,
                    url,
                    HTTPStatus.INTERNAL_SERVER_ERROR.description
                    or HTTPStatus.INTERNAL_SERVER_ERROR.phrase,
                ),
                response,
            )
        return response
    def join(self, path, query=None):
        """Create a service URL.  Join base + path + query parameters"""
        path = self.base + path
        if query is not None:
            query = urllib.parse.urlencode(query)
            path = path + "?" + query
        return path
    @staticmethod
    def quote(value):
        """Quote value for use in a URL"""
        return urllib.parse.quote(value)
    def __exit__(self, exc_type, exc_value, traceback):
        super().close()

if __name__ == "__main__":  # pragma: no cover
    with ApiConnection("unix:///run/podman/podman.sock") as api:
        url = api.join("/containers/json?all={}".format(False))
        response = api.request("GET", url)
        print("List response received")
        time.sleep(5)
        url = api.join("/version")
        response = api.request("GET", url)
        print(json.loads(response.read()))

And the output is 
List response received
Traceback (most recent call last):
  File "<stdin>", line 8, in <module>
  File "<stdin>", line 30, in request
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

[mwhahaha]

I'm using 2.2.1 and I get:

List response received
Traceback (most recent call last):
  File "/home/aschultz/Documents/workspace/containers/podman-py/test.py", line 96, in <module>
    response = api.request("GET", url)
  File "/home/aschultz/Documents/workspace/containers/podman-py/test.py", line 39, in request
    response = super().getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1337, in getresponse
    raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent

The issue seems to be with your code. You aren't reading the initial response so it fails. When I updated the code to use the provided interfaces from this module (note: I have a unreleased list_containers which is commented out here), I get:

import podman
import time
import json

with podman.ApiConnection("unix:///run/podman/podman.sock") as api:
    #response = podman.containers.list_containers(api)
    url = api.join("/containers/json?all={}".format(False))
    response = api.request("GET", url)
    print("List response received")
    #print(response)
    print(json.loads(response.read())) # <- you are missing this in the posted code
    time.sleep(5)
    url = api.join("/version")
    response = api.request("GET", url)
    print(json.loads(response.read()))

List response received
[]
{'Platform': {'Name': 'linux/amd64/fedora-33'}, 'Components': [{'Name': 'Podman Engine', 'Version': '2.2.1', 'Details': {'APIVersion': '2.0.0', 'Arch': 'amd64', 'BuildTime': '2020-12-08T07:37:50-07:00', 'Experimental': 'true', 'GitCommit': '', 'GoVersion': 'go1.15.5', 'KernelVersion': '5.10.7-200.fc33.x86_64', 'MinAPIVersion': '2.0.0', 'Os': 'linux'}}], 'Version': '2.2.1', 'ApiVersion': '1.40', 'MinAPIVersion': '1.24', 'GitCommit': '', 'GoVersion': 'go1.15.5', 'Os': 'linux', 'Arch': 'amd64', 'KernelVersion': '5.10.7-200.fc33.x86_64', 'Experimental': True, 'BuildTime': '2020-12-08T07:37:50-07:00'}

[sumit0900]

I tried after adding print code as suggested but the same issue persists. I have only added a statement to print the length as the list container output is huge.Below is the output

with ApiConnection("unix:///run/podman/podman.sock") as api:
        url = api.join("/containers/json?all={}".format(False))
        response = api.request("GET", url)
        print("List response received")
        output = json.loads(response.read())
        print(len(output))
        time.sleep(5)
        url = api.join("/version")
        response = api.request("GET", url)
        print(json.loads(response.read()))
List response received
17
Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "<stdin>", line 30, in request
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

Is there anything related to podman version? Tried in 2 VMS and the same issue appeared.

[mwhahaha]

It's possible. Filing a bug against podman itself would be the next course of action

[mwhahaha]

Worked for me on 2.0.5 with and without selinux enforcing

[centos@host podman-py]$ rpm -qa | grep podman
podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64
podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64
[centos@host podman-py]$ sudo podman --log-level debug system service --time=0 unix:///run/podman/podman.sock

[centos@host podman-py]$ cat test.py
import podman
import time
import json

with podman.ApiConnection("unix:///run/podman/podman.sock") as api:
    #response = podman.containers.list_containers(api)
    url = api.join("/containers/json?all={}".format(False))
    response = api.request("GET", url)
    print("List response received")
    #print(response)
    print(json.loads(response.read())) # <- you are missing this in the posted code
    time.sleep(5)
    url = api.join("/version")
    response = api.request("GET", url)
    print(json.loads(response.read()))
[centos@host podman-py]$ sudo python3 test.py
List response received
[{'Command': ['bash'], 'Created': 1610555970, 'CreatedAt': '', 'Exited': False, 'ExitedAt': -62135596800, 'ExitCode': 0, 'Id': 'c1c54252e2e32e7b02f7773f9740601d3d9c5281f9688daaa89e8188dfea3c55', 'Image': 'registry.fedoraproject.org/fedora:latest', 'ImageID': '79fd58dc76113dac76a120f22cadecc3b2d1794b414f90ea368cf66096700053', 'IsInfra': False, 'Labels': {'license': 'MIT', 'name': 'fedora', 'vendor': 'Fedora Project', 'version': '33'}, 'Mounts': [], 'Names': ['podman-container2'], 'Namespaces': {}, 'Pid': 0, 'Pod': '', 'PodName': '', 'Ports': None, 'Size': None, 'StartedAt': 1610555971, 'State': 'running'}, {'Command': ['bash'], 'Created': 1610555969, 'CreatedAt': '', 'Exited': False, 'ExitedAt': -62135596800, 'ExitCode': 0, 'Id': 'f7b4604bba82cd3f7b65ffc6f69ca00c08a56782c6ba8296ecd8dff1f2a11c95', 'Image': 'registry.fedoraproject.org/fedora:latest', 'ImageID': '79fd58dc76113dac76a120f22cadecc3b2d1794b414f90ea368cf66096700053', 'IsInfra': False, 'Labels': {'license': 'MIT', 'name': 'fedora', 'vendor': 'Fedora Project', 'version': '33'}, 'Mounts': [], 'Names': ['podman-container1'], 'Namespaces': {}, 'Pid': 0, 'Pod': '', 'PodName': '', 'Ports': None, 'Size': None, 'StartedAt': 1610555970, 'State': 'running'}]
{'Platform': {'Name': 'linux/amd64/"centos"-8'}, 'Components': [{'Name': 'Podman Engine', 'Version': '2.0.5', 'Details': {'APIVersion': '1.0.0', 'Arch': 'amd64', 'BuildTime': '2020-09-24T17:58:13Z', 'Experimental': 'true', 'GitCommit': '', 'GoVersion': 'go1.14.7', 'KernelVersion': '4.18.0-240.1.1.el8_3.x86_64', 'MinAPIVersion': '1.0.0', 'Os': 'linux'}}], 'Version': '2.0.5', 'ApiVersion': '1.0.0', 'MinAPIVersion': '1.0.0', 'GitCommit': '', 'GoVersion': 'go1.14.7', 'Os': 'linux', 'Arch': 'amd64', 'KernelVersion': '4.18.0-240.1.1.el8_3.x86_64', 'Experimental': True, 'BuildTime': '2020-09-24T17:58:13Z'}
[centos@host podman-py]$

[mwhahaha]

I reproduced it using the podman.service. Please file a bug over against containers/podman. I believe it's because of the sleep because it'll shutdown the service fairly quickly after the initial request as the default time option is 5 seconds. Removing the sleep seems to let it work. Anyway it's due to the service invocation and needs to be reported against that.

[sumit0900]

Thanks, @mwhahaha for the analysis. Is there any workaround to increase the timeout from client-side so that we don't have to create a socket connection for every podman API request?

[mwhahaha]

Unfortunately not clientside. The timeout is server side. If you continue to make requests over and over within a 5 second period you shouldn't get this error. You can either adjust the server side to always be running (or extend the timeout) or write a wrapper to catch connection related issues and reconnect/retry. Since only file sockets are supported, the connection overhead is very minimal