With different python versions rsync can hangs on

[mitzkia]

At the end of my investigation of this issue I have realized it happens because of my fault.
So I know pytest/xdist works correctly here.
My expectation would be that pytest could send a Warning message (or a notification) about the possible fix of this issue.

Without going into the details the fix is to use same python versions on both side, in my case the correct pytest.ini is:

[pytest]
addopts =
            --tx id=debian-latest-192.168.9.111//ssh="-i docker/ssh-config/id_rsa 192.168.9.111 -l root//python=python3"
rsyncdirs = .

So, I've been debugging the following case (for a while when I found the root cause):
1, Use such ("wrong") pytest.ini:

[pytest]
addopts =
            --tx id=debian-latest-192.168.9.111//ssh="-i docker/ssh-config/id_rsa 192.168.9.111 -l root//python=python"
rsyncdirs = .

2, Start pytest with such command:

python3 -m pytest --dist=each ...

3, Realize that pytest hangs on, at (with PYTEST_DEBUG=1):

      pytest_xdist_setupnodes [hook]
          config: <_pytest.config.Config object at 0x7f6d09e91358>
          specs: [<XSpec 'id=debian-latest-192.168.9.111//ssh=-i docker/ssh-config/id_rsa 192.168.9.111 -l root//python=python'>]
debian-latest-192.168.9.111 I      finish pytest_xdist_setupnodes --> [] [hook]
    setting up nodes [config:nodemanager]
    early skip of rewriting module: thread [assertion]
      pytest_xdist_newgateway [hook]
          gateway: <Gateway id='debian-latest-192.168.9.111' receive-live, thread model, 0 active channels>
      early skip of rewriting module: __builtin__ [assertion]
[debian-latest-192.168.9.111] linux2 Python 2.7.13 cwd: /root/pyexecnetcache
debian-latest-192.168.9.111 C      finish pytest_xdist_newgateway --> [] [hook]
      pytest_xdist_rsyncstart [hook]
          source: /home/micek/.local/lib/python3.6/site-packages/py
          gateways: [<Gateway id='debian-latest-192.168.9.111' receive-live, thread model, 1 active channels>]
      finish pytest_xdist_rsyncstart --> [] [hook]

stacktrace log with --fulltrace:

config = <_pytest.config.Config object at 0x7f6d09e91358>, doit = <function _main at 0x7f6d0a39b598>

    def wrap_session(config, doit):
        """Skeleton command line program"""
        session = Session(config)
        session.exitstatus = EXIT_OK
        initstate = 0
        try:
            try:
                config._do_configure()
                initstate = 1
>               config.hook.pytest_sessionstart(session=session)

/usr/local/lib/python3.6/dist-packages/_pytest/main.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_HookCaller 'pytest_sessionstart'>, args = (), kwargs = {'session': <Session pytest_framework>}, notincall = set()

    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()
        if self.argnames:
            notincall = set(self.argnames) - set(['__multicall__']) - set(
                kwargs.keys())
            if notincall:
                warnings.warn(
                    "Argument(s) {} which are declared in the hookspec "
                    "can not be found in this hook call"
                    .format(tuple(notincall)),
                    stacklevel=2,
                )
>       return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py:258: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.config.PytestPluginManager object at 0x7f6d0c2ebf28>, hook = <_HookCaller 'pytest_sessionstart'>
methods = [<HookImpl plugin_name='dsession', plugin=<xdist.dsession.DSession object at 0x7f6d0893f4a8>>, <HookImpl plugin_name='...6d08c079e8>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6d08c076d8>>]
kwargs = {'session': <Session pytest_framework>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pluggy._tracing._TracedHookExecution object at 0x7f6d0a3645f8>, hook = <_HookCaller 'pytest_sessionstart'>
hook_impls = [<HookImpl plugin_name='dsession', plugin=<xdist.dsession.DSession object at 0x7f6d0893f4a8>>, <HookImpl plugin_name='...6d08c079e8>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6d08c076d8>>]
kwargs = {'session': <Session pytest_framework>}

    def __call__(self, hook, hook_impls, kwargs):
        self.before(hook.name, hook_impls, kwargs)
        outcome = _Result.from_call(lambda: self.oldcall(hook, hook_impls, kwargs))
        self.after(outcome, hook.name, hook_impls, kwargs)
>       return outcome.get_result()

/usr/local/lib/python3.6/dist-packages/pluggy/_tracing.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   outcome = _Result.from_call(lambda: self.oldcall(hook, hook_impls, kwargs))

/usr/local/lib/python3.6/dist-packages/pluggy/_tracing.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hook = <_HookCaller 'pytest_sessionstart'>
methods = [<HookImpl plugin_name='dsession', plugin=<xdist.dsession.DSession object at 0x7f6d0893f4a8>>, <HookImpl plugin_name='...6d08c079e8>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6d08c076d8>>]
kwargs = {'session': <Session pytest_framework>}

    self._inner_hookexec = lambda hook, methods, kwargs: \
        hook.multicall(
            methods, kwargs,
>           firstresult=hook.spec_opts.get('firstresult'),
        )

/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:61: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.dsession.DSession object at 0x7f6d0893f4a8>, session = <Session pytest_framework>

    @pytest.mark.trylast
    def pytest_sessionstart(self, session):
        """Creates and starts the nodes.
    
        The nodes are setup to put their events onto self.queue.  As
        soon as nodes start they will emit the worker_workerready event.
        """
        self.nodemanager = NodeManager(self.config)
>       nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)

/usr/local/lib/python3.6/dist-packages/xdist/dsession.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.workermanage.NodeManager object at 0x7f6d089aee80>, putevent = <bound method Queue.put of <queue.Queue object at 0x7f6d0895f198>>

    def setup_nodes(self, putevent):
        self.config.hook.pytest_xdist_setupnodes(config=self.config, specs=self.specs)
        self.trace("setting up nodes")
        nodes = []
        for spec in self.specs:
>           nodes.append(self.setup_node(spec, putevent))

/usr/local/lib/python3.6/dist-packages/xdist/workermanage.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.workermanage.NodeManager object at 0x7f6d089aee80>, spec = <XSpec 'id=debian-latest-192.168.9.111//ssh=-i docker/ssh-config/id_rsa 192.168.9.111 -l root//python=python'>
putevent = <bound method Queue.put of <queue.Queue object at 0x7f6d0895f198>>

    def setup_node(self, spec, putevent):
        gw = self.group.makegateway(spec)
        self.config.hook.pytest_xdist_newgateway(gateway=gw)
>       self.rsync_roots(gw)

/usr/local/lib/python3.6/dist-packages/xdist/workermanage.py:74: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.workermanage.NodeManager object at 0x7f6d089aee80>, gateway = <Gateway id='debian-latest-192.168.9.111' not-receiving, thread model, 0 active channels>

    def rsync_roots(self, gateway):
        """Rsync the set of roots to the node's gateway cwd."""
        if self.roots:
            for root in self.roots:
>               self.rsync(gateway, root, **self.rsyncoptions)

/usr/local/lib/python3.6/dist-packages/xdist/workermanage.py:61: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.workermanage.NodeManager object at 0x7f6d089aee80>, gateway = <Gateway id='debian-latest-192.168.9.111' not-receiving, thread model, 0 active channels>
source = local('/home/micek/.local/lib/python3.6/site-packages/py'), notify = None, verbose = 2, ignores = ['.*', '*.pyc', '*.pyo', '*~']

    def rsync(self, gateway, source, notify=None, verbose=False, ignores=None):
        """Perform rsync to remote hosts for node."""
        # XXX This changes the calling behaviour of
        #     pytest_xdist_rsyncstart and pytest_xdist_rsyncfinish to
        #     be called once per rsync target.
        rsync = HostRSync(source, verbose=verbose, ignores=ignores)
        spec = gateway.spec
        if spec.popen and not spec.chdir:
            # XXX This assumes that sources are python-packages
            #     and that adding the basedir does not hurt.
            gateway.remote_exec(
                """
                import sys ; sys.path.insert(0, %r)
            """
                % os.path.dirname(str(source))
            ).waitclose()
            return
        if (spec, source) in self._rsynced_specs:
            return
    
        def finished():
            if notify:
                notify("rsyncrootready", spec, source)
    
        rsync.add_target_host(gateway, finished=finished)
        self._rsynced_specs.add((spec, source))
        self.config.hook.pytest_xdist_rsyncstart(source=source, gateways=[gateway])
>       rsync.send()

/usr/local/lib/python3.6/dist-packages/xdist/workermanage.py:148: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.workermanage.HostRSync object at 0x7f6d08983cc0>, raises = True

    def send(self, raises=True):
        """ Sends a sourcedir to all added targets. Flag indicates
        whether to raise an error or return in case of lack of
        targets
        """
        if not self._channels:
            if raises:
                raise IOError("no targets available, maybe you "
                              "are trying call send() twice?")
            return
        # normalize a trailing '/' away
        self._sourcedir = os.path.dirname(os.path.join(self._sourcedir, 'x'))
        # send directory structure and file timestamps/sizes
        self._send_directory_structure(self._sourcedir)
    
        # paths and to_send are only used for doing
        # progress-related callbacks
        self._paths = {}
        self._to_send = {}
    
        # send modified file to clients
        while self._channels:
            print("self._channels - before self._receivequeue.get(): [%s]" % self._channels)
>           channel, req = self._receivequeue.get()

/usr/local/lib/python3.6/dist-packages/execnet/rsync.py:128: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <queue.Queue object at 0x7f6d089836d8>, block = True, timeout = None

    def get(self, block=True, timeout=None):
        '''Remove and return an item from the queue.
    
        If optional args 'block' is true and 'timeout' is None (the default),
        block if necessary until an item is available. If 'timeout' is
        a non-negative number, it blocks at most 'timeout' seconds and raises
        the Empty exception if no item was available within that time.
        Otherwise ('block' is false), return an item if one is immediately
        available, else raise the Empty exception ('timeout' is ignored
        in that case).
        '''
        with self.not_empty:
            if not block:
                if not self._qsize():
                    raise Empty
            elif timeout is None:
                while not self._qsize():
>                   self.not_empty.wait()

/usr/lib/python3.6/queue.py:164: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Condition(<unlocked _thread.lock object at 0x7f6d08938d50>, 0)>, timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               KeyboardInterrupt

Also added own print debugs near: execnet/rsync.py -> send()

self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_vendored_packages', 'apipkg.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_vendored_packages', '__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_vendored_packages', 'iniconfig.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_log', 'log.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_log', '__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_log', 'warning.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_error.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['__metainfo.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_xmlgen.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_path', 'local.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_path', 'common.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_path', 'cacheutil.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_path', 'svnwc.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_path', '__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_path', 'svnurl.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_io', 'terminalwriter.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_io', 'saferepr.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_io', '__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_io', 'capture.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', 'code.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', '_assertionnew.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', '_assertionold.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', '_py2traceback.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', '__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', 'source.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_code', 'assertion.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['test.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_version.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_builtin.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_std.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_process', 'forkedfunc.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_process', 'cmdexec.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_process', '__init__.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'send', (['_process', 'killproc.py'], None))]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]
channel - after self._receivequeue.get(): [<Channel id=5 open>]
req - after self._receivequeue.get(): [(b'list_done', None)]
self._channels - before self._receivequeue.get(): [{<Channel id=5 open>: <function NodeManager.rsync.<locals>.finished at 0x7f6d089c7e18>}]