pytest parallel process failing with INTERNALERROR in azure machine

[shashankmotruri]

pytest parallel processes are failing with INTERNALERROR

Verisons using:
pytest==8.3.1
pytest-xdist==3.6.1
azure-cli==2.70.0
Python (Linux) 3.9.21

Command ran:
pytest
-v
-n 4
-m 'sit'
--cache-clear
--json-report
......
......

But the same command with-n 1 works fine

INTERNALERROR> def worker_internal_error(
INTERNALERROR>         self, node: WorkerController, formatted_error: str
INTERNALERROR>     ) -> None:
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR> E                 result = yield
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 174, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 195, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/assertion/__init__.py", line 176, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/unittest.py", line 429, in pytest_runtest_protocol
INTERNALERROR> E                 res = yield
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/faulthandler.py", line 87, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line 113, in pytest_runtest_protocol
INTERNALERROR> E                 runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line [132](https://github.com/yugabyte/yugabyte-cloud/actions/runs/14407462005/job/40409253750#step:20:133), in runtestprotocol
INTERNALERROR> E                 reports.append(call_and_report(item, "call", log))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line 246, in call_and_report
INTERNALERROR> E                 ihook.pytest_runtest_logreport(report=report)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 255, in pytest_runtest_logreport
INTERNALERROR> E                 self.sendevent("testreport", data=data)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 98, in sendevent
INTERNALERROR> E                 self.channel.send((name, kwargs))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 912, in send
INTERNALERROR> E                 self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1190, in _send
INTERNALERROR> E                 message.to_io(self._io)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 576, in to_io
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
INTERNALERROR> E                 header = struct.pack("!bii", self.msgcode, self.channelid, len(self.data))
INTERNALERROR> E             struct.error: 'i' format requires -2147483648 <= number <= 2147483647
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py:232: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line [138](https://github.com/yugabyte/yugabyte-cloud/actions/runs/14407462005/job/40409253750#step:20:139), in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line 217, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)

[RonnyPfannschmidt]

More context needed

Execnet doesn't call into urllib

[webknjaz]

Any repro?

[RonnyPfannschmidt]

I just tool note that thr error is intermixed

The real issue seems to be the crashitem data length

[RonnyPfannschmidt]

There's need for a investigation on why the report has more than 2gb of data

[shashankmotruri]

We are using --json-report to generate a report summarizing all the tests. The pytest command worked fine without this arg.

[RonnyPfannschmidt]

where is --json-report from ?

[RonnyPfannschmidt]

https://github.com/mattcl/pytest-json - this one is unmaintainedand by a 3rd party

the officail plugin with a json-lines report is https://github.com/pytest-dev/pytest-reportlog