[Failing Test]: GHA Java_GCP_IO_Direct is failing consistently at "Publish JUnit Test Results" step

[ahmedabu98]

What happened?

Stack trace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:1114: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 406, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line [48](https://github.com/apache/beam/actions/runs/6396080511/job/17361147230?pr=28397#step:8:49)6, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
  [Previous line repeated 7 more times]
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/github/GithubRetry.py", line 220, in increment
    return super().increment(method, url, response, error, _pool, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/apache/beam (Caused by ReadTimeoutError("HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/action/publish_test_results.py", line 538, in <module>
    main(settings, gha)
  File "/action/publish_test_results.py", line 262, in main
    Publisher(settings, gh, gha).publish(stats, results.case_results, conclusion)
  File "/action/publish/publisher.py", line 188, in __init__
    self._repo = gh.get_repo(self._settings.repo)
  File "/usr/local/lib/python3.8/site-packages/github/MainClass.py", line 376, in get_repo
    headers, data = self.__requester.requestJsonAndCheck("GET", url)
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 456, in requestJsonAndCheck
    *self.requestJson(
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 617, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 722, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 763, in __requestRaw
    response = cnx.getresponse()
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 167, in getresponse
    r = verb(
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line [51](https://github.com/apache/beam/actions/runs/6396080511/job/17361147230?pr=28397#step:8:52)9, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/apache/beam (Caused by ReadTimeoutError("HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)"))

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

This happens between

https://github.com/apache/beam/actions/runs/6342568605/job/17228698749 - 2023-09-28 18:05

and

https://github.com/apache/beam/actions/runs/6351240606/job/17252181160 - 2023-09-29 12:02

where there is timeout in two runs in between

https://github.com/apache/beam/actions/runs/6345499725/job/17237578555 - 2023-09-29 00:06
https://github.com/apache/beam/actions/runs/6347733612/job/17243172251 - 2023-09-29 06:02

[Abacn]

I could not find a possible change on beam side; and EnricoMi/publish-unit-test-result-action does not have new release in that time period.

From the log, the exception happens between

publish - INFO - Finished reading 202 files in 0.52 seconds

and

publish - INFO - Publishing failure results for commit ...

Persumably its failing connect to 'api.github.com'. This happens after moving this workflow to highmem: #28719

[Abacn]

CC: @volatilemolotov

[Abacn]

#28804 reverted - keep the issue open to resolve the credential / connection issue on highmem cluster

[Abacn]

Note that the enroll of highmem runner for python postcommit is also reverted due to another error on highmem cluster:

beam_PostCommit_Python (Run Python PostCommit 3.11)
Command ['sh', '-c', "pip install cython==0.29.36 numpy --config-settings=setup-args='-Dallow-noblas=true' && pip install --upgrade setuptools"] failed with code 1.

[volatilemolotov]

https://github.com/apache/beam/actions/runs/6402157556

Its working on main now. Ill investigate what might be different with highmem group (usually nothing is, except machine type and requests)

[volatilemolotov]

@Abacn https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_GCP_IO_Direct.yml test is now flaky but it is passing from time to time.

Not sure what changes happened