Failed import caused copr-ping to fail

[fedora-copr-github-bot]

Original issue: https://pagure.io/copr/copr/issue/2103
Opened: 2022-02-23 08:02:10
Opened by: praiskup

https://copr-dist-git.fedorainfracloud.org/per-task-logs/3540324.log

Downloading https://download.copr.fedorainfracloud.org/results/@copr/copr-ping/srpm-builds/03540324/buildtag-0-0.src.rpm
https://download.copr.fedorainfracloud.org/results/@copr/copr-ping/srpm-builds/03540324/buildtag-0-0.src.rpm
Exception raised during package import.
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7efc54f7ad70>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='download.copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /results/@copr/copr-ping/srpm-builds/03540324/buildtag-0-0.src.rpm (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc54f7ad70>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/copr_dist_git/helpers.py", line 162, in download_file
    r = get(url, stream=True, verify=False)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='download.copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /results/@copr/copr-ping/srpm-builds/03540324/buildtag-0-0.src.rpm (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc54f7ad70>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/copr_dist_git/importer.py", line 75, in do_import
    srpm_path = helpers.download_file(
  File "/usr/lib/python3.10/site-packages/copr_dist_git/helpers.py", line 164, in download_file
    raise FileDownloadException(str(e))
copr_dist_git.exceptions.FileDownloadException: HTTPSConnectionPool(host='download.copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /results/@copr/copr-ping/srpm-builds/03540324/buildtag-0-0.src.rpm (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc54f7ad70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
sending a response for task {'build_id': 3540324}
Sending back: 
{"build_id": 3540324}

---

praiskup commented at 2022-02-23 09:04:14:

These import tasks should never fail, they should keep re-trying.

If this can not be fixed, we have to at least let the user know that import failed
(the build is just "failed" without telling the user what happened).
https://copr.fedorainfracloud.org/coprs/g/copr/copr-ping/build/3540324/

(builder-live.log is OK, backend.log is OK)

---

praiskup commented at 2022-04-25 17:22:55:

Happened again today:

[15:20:55][ERROR][copr_dist_git.importer][importer:92][pid:2025080] Exception raised during package import.
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f25bbdeeda0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='download.copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /results/@copr/PyPI/srpm-builds/04312187/python-mcvariable-0.1.0-1.fc35.src.rpm (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f25bbdeeda0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/copr_dist_git/helpers.py", line 162, in download_file
    r = get(url, stream=True, verify=False)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='download.copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /results/@copr/PyPI/srpm-builds/04312187/python-mcvariable-0.1.0-1.fc35.src.rpm (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f25bbdeeda0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/copr_dist_git/importer.py", line 75, in do_import
    srpm_path = helpers.download_file(
  File "/usr/lib/python3.10/site-packages/copr_dist_git/helpers.py", line 164, in download_file
    raise FileDownloadException(str(e))
copr_dist_git.exceptions.FileDownloadException: HTTPSConnectionPool(host='download.copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /results/@copr/PyPI/srpm-builds/04312187/python-mcvariable-0.1.0-1.fc35.src.rpm (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f25bbdeeda0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

---

praiskup commented at 2022-04-26 19:09:11:

[16:25:29][ERROR][copr_dist_git.importer][importer:64][pid:2165460] HTTPSConnectionPool(host='copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /backend/import-completed/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f25bbdefe20>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f25bbdefe20>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='copr.fedorainfracloud.org', port=443): Max retries exceeded with url: /backend/import-completed/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f25bbdefe20>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/copr_dist_git/importer.py", line 61, in post_back_safe
    return self.post_back(data_dict)
  File "/usr/lib/python3.10/site-packages/copr_dist_git/importer.py", line 54, in post_back
    return post(self.post_back_url, auth=self.auth, data=json.dumps(data_dict), headers=self.headers)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 519, in send

---

praiskup commented at 2022-04-30 08:57:09:

Commit 081eb5e5 relates to this ticket

[praiskup]

Ok, seems to be resolved.

[praiskup]

(Triage: we use the dispatcher from backend, so the tasks are tested multiple times we believe)