Skip to content

pytest parallel process failing with INTERNALERROR in azure machine #1194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
shashankmotruri opened this issue Apr 14, 2025 · 7 comments
Open

pytest parallel process failing with INTERNALERROR in azure machine #1194

shashankmotruri opened this issue Apr 14, 2025 · 7 comments

Comments

@shashankmotruri
Copy link

pytest parallel processes are failing with INTERNALERROR

Verisons using:
pytest==8.3.1
pytest-xdist==3.6.1
azure-cli==2.70.0
Python (Linux) 3.9.21

Command ran:
pytest
-v
-n 4
-m 'sit'
--cache-clear
--json-report
......
......

But the same command with-n 1 works fine

INTERNALERROR> def worker_internal_error(
INTERNALERROR>         self, node: WorkerController, formatted_error: str
INTERNALERROR>     ) -> None:
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR> E                 result = yield
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 174, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 195, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/assertion/__init__.py", line 176, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/unittest.py", line 429, in pytest_runtest_protocol
INTERNALERROR> E                 res = yield
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/faulthandler.py", line 87, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line 113, in pytest_runtest_protocol
INTERNALERROR> E                 runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line [132](https://github.com/yugabyte/yugabyte-cloud/actions/runs/14407462005/job/40409253750#step:20:133), in runtestprotocol
INTERNALERROR> E                 reports.append(call_and_report(item, "call", log))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line 246, in call_and_report
INTERNALERROR> E                 ihook.pytest_runtest_logreport(report=report)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 255, in pytest_runtest_logreport
INTERNALERROR> E                 self.sendevent("testreport", data=data)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 98, in sendevent
INTERNALERROR> E                 self.channel.send((name, kwargs))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 912, in send
INTERNALERROR> E                 self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1190, in _send
INTERNALERROR> E                 message.to_io(self._io)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 576, in to_io
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
INTERNALERROR> E                 header = struct.pack("!bii", self.msgcode, self.channelid, len(self.data))
INTERNALERROR> E             struct.error: 'i' format requires -2147483648 <= number <= 2147483647
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py:232: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line [138](https://github.com/yugabyte/yugabyte-cloud/actions/runs/14407462005/job/40409253750#step:20:139), in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line 217, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
@RonnyPfannschmidt
Copy link
Member

More context needed

Execnet doesn't call into urllib

@webknjaz
Copy link
Member

Any repro?

@RonnyPfannschmidt
Copy link
Member

I just tool note that thr error is intermixed

The real issue seems to be the crashitem data length

@RonnyPfannschmidt
Copy link
Member

There's need for a investigation on why the report has more than 2gb of data

@shashankmotruri
Copy link
Author

We are using --json-report to generate a report summarizing all the tests. The pytest command worked fine without this arg.

@RonnyPfannschmidt
Copy link
Member

where is --json-report from ?

@RonnyPfannschmidt
Copy link
Member

https://github.com/mattcl/pytest-json - this one is unmaintainedand by a 3rd party

the officail plugin with a json-lines report is https://github.com/pytest-dev/pytest-reportlog

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@RonnyPfannschmidt @webknjaz @shashankmotruri