pytest-xdist issues when libtmux > v0.16.1

[cidrblock]

We use libtmux for testing ansible-navigator and have the version pinned at 0.16.1.

I've installed the latest version v0.19.1 and if the tests are run with pytest-xdist and the number of CPUs > 1 the following errors are generated:

E           libtmux._internal.query_list.ObjectDoesNotExist: Could not find pane_id=%204 for list-panes

I'm still digging into it, but wanted to get an issue logged early.

What more information can I provide? How can I help?

Thanks, brad

(TY for libtmux BTW, I'm not sure what we would do without it)

[tony]

@cidrblock Thank you! If you could provide more of the traceback, I can give my full attention to it this weekend

• tmux version
• full traceback (to see where in libtmux it's tripping up)
• a snippet of libtmux code where the error is happening (or perhaps linking to the code in the repo)
• steps to reproduce in ansible navigator (I assume its happening there)
• If time isn't a problem: Enough to reproduce it reliably and independent of ansible-navigator, if possible (if that's too intensive I can clone the repo and try to reproduce locally)

(TY for libtmux BTW, I'm not sure what we would do without it)

Thank you!

[cidrblock]

I'm trying to create a repro, will be in touch.

thanks, Brad

[tony]

Sounds good!

[tony]

@cidrblock This doesn't necessarily need a full reproduction (that may be prohibitively difficult)

If there's a more fuller traceback / etc. I can look at that as well.

[tony]

@cidrblock If you try v0.21.0 (which includes #475), is anything different?

[cidrblock]

I need to get back to this.... Will try next week. I few other things popped up that needed attention.

talk soon

[tony]

@cidrblock Sounds good. This is a priority since we definitely want libtmux to be xdist compatible

[sgherdao]

Hi, I think I am facing the same issue.

To reproduce the problem:

1. Attach a tmux session

tmux new -A -s main

2. run the following code repro.py

import libtmux
tmux_server = libtmux.Server()
session = tmux_server.new_session(session_name="repro-bug", kill_session=True)
window = session.windows[0]
window.rename_window("BOOM")

The traceback

Traceback (most recent call last):
  File "bug-repro-libtmux.py", line 6, in <module>
    window.rename_window("BOOM")
  File "/...site-packages/libtmux/window.py", line 480, in rename_window
    self.refresh()
  File "/...site-packages/libtmux/window.py", line 84, in refresh
    return super()._refresh(
           ^^^^^^^^^^^^^^^^^
  File "/...site-packages/libtmux/neo.py", line 175, in _refresh
    obj = fetch_obj(
          ^^^^^^^^^^
  File "/...site-packages/libtmux/neo.py", line 244, in fetch_obj
    raise ObjectDoesNotExist(
libtmux._internal.query_list.ObjectDoesNotExist: Could not find window_id=@2 for list-windows

Done some digging

❯ export PYTHONBREAKPOINT=ipdb.set_trace

❯ python bug-repro-libtmux.py

Some debugging with comments

ipdb> bt
bug-repro-libtmux.py(6)<module>()
4 session = tmux_server.new_session(session_name="repro-bug", kill_session=True)
5 window = session.windows[0]
----> 6 window.rename_window("BOOM")

.../libtmux/window.py(480)rename_window()
479
--> 480 self.refresh()
481

.../libtmux/window.py(84)refresh()
83 assert isinstance(self.window_id, str)
---> 84 return super().\_refresh(
85 obj_key="window_id",

.../libtmux/neo.py(175)\_refresh()
174 assert isinstance(obj_id, str)
--> 175 obj = fetch_obj(
176 obj_key=obj_key, obj_id=obj_id, list_cmd=list_cmd, server=self.server

> .../libtmux/neo.py(243)fetch_obj()

    242     breakpoint()

--> 243 if obj is None:
244 raise ObjectDoesNotExist(
#
# We can see that only the window for the current session is returned 
# and not the one we are looking for so ObjectDoesNotExist will raise
#
ipdb> obj_id
'@17'
ipdb> obj_key
'window_id'
ipdb> pp [(x["window_name"], x["window_id"]) for x in obj_formatters_filtered]
[('python3.11', '@16')]
#
# Re-run fetch_obj but we will modify list_extra_args from None to ["-a"]
#
ipdb> jump 234

> .../libtmux/neo.py(234)fetch_obj()

    233 ) -> OutputRaw:

--> 234 obj_formatters_filtered = fetch_objs(
235 server=server, list_cmd=list_cmd, list_extra_args=list_extra_args
# 
# HACK make libtmux run tmux list-windows -a (list all windows of all sessions)
# 
ipdb> list_extra_args = ["-a"] 
ipdb> pp [(x["window_name"], x["window_id"]) for x in obj_formatters_filtered]
[('python3.11', '@16')]
ipdb> until 238

> .../libtmux/neo.py(238)fetch_obj()

    237

--> 238 obj = None
239 for \_obj in obj_formatters_filtered:

# You can see now that the window from the new session is returned

ipdb> pp [(x["window_name"], x["window_id"]) for x in obj_formatters_filtered]
[('python3.11', '@16'), ('BOOM', '@17')]
ipdb> obj_id
'@17'
ipdb> obj_key
'window_id'

# 
# and the script exits gracefully 
# 
ipdb> c

hope this gives some pointers and thx for the lib :)

[sgherdao]

A workaround that seems to work for me is to use .cmd() to bypass the code.

import libtmux
import os
tmux_server = libtmux.Server()
session = tmux_server.new_session(session_name="repro-bug", kill_session=True)
window = session.windows[0]
window.cmd("rename-window", "BOOM")

[tony]

@cidrblock Pinning this, in re: ansible/ansible-navigator#1380

Thank you as well for the additional info @sgherdao

[ssbarnea]

@tony Any chance we could get a fix on this bug. Being forced to use an ancient version of libtmux is more like a playing with a grenade. I tried latest release and i still not working, so we are stuck with 0.16.1 for now.

Usually I would have tried to make a fix it myself but sadly my tmux knowledge is not up for that.

[tony]

@ssbarnea Yes, happy to escalate this.

[tony]

@cidrblock @ssbarnea @sgherdao The above issue will be resolved by #523.

TLDR: It was caused by refreshing not passing -a to list-windows / list-panes.

[tony]

@cidrblock @ssbarnea @sgherdao:

• v0.28.0 (PyPI, release)
• ansible-navigator related fixes continuing at Test suite: Support for libtmux 0.30.1+ ansible/ansible-navigator#1687.