[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails #61419

dmacvicar · 2022-01-04T23:46:19Z

Description

salt-ssh on a Python 3.9 host and a Python 3.6.15 target fails with ModuleNotFoundError: No module named '_contextvars' error:

Setup

My setup on the host is a Nix environment with Salt 3004 and a target machine with openSUSE and Python 3.9.15 on ARM, but from what I see in the code, it should happen on a Python > 3.7 host / Python 3.6 target.

Steps to Reproduce the behavior

Create a salt-ssh self-contained tree with a Saltfile, using a modern Python install.
Have a 3.6 target. You will need contextvars backport package (python3-contextvars)

After running salt-ssh rpi01 grains.items, it will fail:

        Traceback (most recent call last):
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/loader/context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.root_05dd7c_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/cli/caller.py", line 14, in <module>
            import salt.loader
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/loader/__init__.py", line 14, in <module>
            import salt.config
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/config/__init__.py", line 101, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/config/__init__.py", line 89, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/grains/core.py", line 32, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/utils/templates.py", line 20, in <module>
            import salt.utils.jinja
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/utils/jinja.py", line 21, in <module>
            import salt.fileclient
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/fileclient.py", line 15, in <module>
            import salt.client
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/client/__init__.py", line 33, in <module>
            import salt.payload
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/payload.py", line 15, in <module>
            import salt.loader.context
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/loader/context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.root_05dd7c_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
    stdout:

I believe the reason could be (I could be wrong):

/var/tmp/.root_05dd7c_salt/pyall/salt/loader/context.py does:

try:
    # Try the stdlib C extension first
    import _contextvars as contextvars
except ImportError:
    # Py<3.7
    import contextvars

The fallback import should end importing contextvars backport. but because sys.path includes /var/tmp/.root_05dd7c_salt/py3/contextvars.py first, which is a copy of the 3.9 stdlib, which also depends on _contextvars!! native module:

from _contextvars import Context, ContextVar, Token, copy_context
__all__ = ('Context', 'ContextVar', 'Token', 'copy_context')

which brings Python 3.6 back into importing something that is not in Python 3.6, even if the backport package is installed on the machine.

So there are a few things I can't make sense of:

the fallback import with underscore / not underscore

The contextvars backport states:

The good news is that the standard library always takes the precedence over site packages, so even if a local contextvars module is installed, the one from the standard library will be used. Therefore you can simply list “contextvars” in your requirements.txt or setup.py files.

So I assume just importing just context in salt/loader/context.py should attempt to load the 3.7 module, or fallback to the backport.

contextvars being included in the thin!. The whole point of the backport is to implement contextvars without depending on _contextvars, so if we are on a newer Python, this module is already included, and if we are on < 3.6, the 3.7 module can't be used. What is the point of copying it?

Fixes and workarounds

Just deleting rm /var/tmp/.root_05dd7c_salt/py3/contextvars.py makes things work, as Python 3.6 would then use the installed contextvars backport.
I tried to use python 3.9 on the target machine. It would still use the "wrong" py3 dir module, but it would work as _contextvars is available. salt-ssh does not allow me to select /usr/bin/python3.9 as the interpreter to use in the target machine, so it uses /usr/bin/python3.

My intuition tells me there is no point in packing contextvars in the thin, unless we are packing the backport package.

Expected behavior

I expect salt-ssh rpi01 grains.items to work

salt --versions-report

Host:

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.14.6
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.2
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.11.0
        pygit2: Not Installed
        Python: 3.9.6 (default, Jun 28 2021, 08:57:49)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 21.0.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: opensuse-tumbleweed 20211215 n/a
        locale: utf-8
       machine: x86_64
       release: 5.15.8-1-default
        system: Linux
       version: openSUSE Tumbleweed 20211215 n/a

Target:

python3-3.6.13-3.81.2.aarch64
python3-contextvars-2.4-bp153.1.14.noarch

The text was updated successfully, but these errors were encountered:

OrangeDog · 2022-01-05T09:32:20Z

Duplicate of #59942?
@dwoz there are other reports there that it wasn't fixed.

robertchen · 2022-05-10T13:36:24Z

same issue

rittycat · 2022-05-10T13:59:17Z

Also same issue, would love to see a workaround for this even if a real fix won't come for a good while

rittycat · 2022-05-11T05:47:39Z

The issue seems to come from these lines:

salt/salt/utils/thin.py

Lines 429 to 435 in 38341cb

    
           modules = find_site_modules("contextvars") 
        
           if modules: 
        
               contextvars = modules[0] 
        
           else: 
        
               contextvars = py_contextvars 
        
           log.debug("Using contextvars %r", contextvars) 
        
           mods.append(contextvars)

contextvars is part of the standard library, so I'm not really understanding why this is packaging it if any python version from 3.7 up would already have it.

Given that a 3.7+ master is not going to have the backport (Or prefer it if its installed), this means that an incompatible version of contextvars gets packaged before being sent to the 3.6 target.

I was able to solve this in our system by commenting these lines, then using the pre-flight script to install the contextvars backport. Not sure right now how to best handle this without using pre-flight

ptitdoc · 2022-06-02T13:32:40Z

Same here, commenting these lines on my salt-ssh client side (running with python 3.10), and installing contextvars using pip on my python3.6 server was successfull.

Python 3.10 contextvars.py is packaged in the salt-ssh thin but this contextvars.py version is not compatible if servers are running python 3.6/3.7 anyway.

piterpunk · 2022-06-06T23:07:04Z

This is already fixed, but is a bit counter-intuitive: you need to install contextvars backport package even when the salt-master is using Python 3.7+

An execution returning the already known error:

# salt-ssh 'ijurn.oci' -w -t test.ping
ijurn.oci:
    ----------
    retcode:   
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/scripts.py", line 435, in salt_call
            import salt.cli.call
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/caller.py", line 12, in <module>
            import salt.channel.client
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/channel/client.py", line 13, in <module>
            import salt.crypt
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/crypt.py", line 26, in <module>
            import salt.payload
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/payload.py", line 12, in <module>
            import salt.loader.context
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/__init__.py", line 15, in <module>
            import salt.config
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 100, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/grains/core.py", line 31, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/utils/templates.py", line 27, in <module>
            from salt.loader.context import NamedLoaderContext
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.opc_706a4b_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
        [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
        ModuleNotFoundError: No module named '_contextvars'
        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/scripts.py", line 435, in salt_call
            import salt.cli.call
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/caller.py", line 12, in <module>
            import salt.channel.client
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/channel/client.py", line 13, in <module>
            import salt.crypt
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/crypt.py", line 26, in <module>
            import salt.payload
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/payload.py", line 12, in <module>
            import salt.loader.context
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/__init__.py", line 15, in <module>
            import salt.config
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 100, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/grains/core.py", line 31, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/utils/templates.py", line 27, in <module>
            from salt.loader.context import NamedLoaderContext
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.opc_706a4b_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
    stdout:

Then, install contextvars and run the same command:

# pip install contextvars
Collecting contextvars
  Using cached contextvars-2.4-py3-none-any.whl
Requirement already satisfied: immutables>=0.9 in /usr/lib/python3.9/site-packages (from contextvars) (0.18)
Installing collected packages: contextvars
Successfully installed contextvars-2.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
# salt-ssh 'ijurn.oci' -w -t test.ping
ijurn.oci:
    True

In the requires.txt, even when using Python 3.7+, the contextvars module is listed as a requirement:

Jinja2
msgpack>=0.5,!=0.5.5
PyYAML
MarkupSafe
requests>=1.0.0
distro>=1.0.1
contextvars
psutil>=5.0.0
pyzmq>19.0.2 ; python_version >= "3.9"
pycryptodomex>=3.9.8

And it's there exactly to make salt-ssh works OK with targets using versions of Python lower than 3.7, as @bryceml stated in #60483:

if downstream packages want to remove contextvars from the requirements on py3.7+, that's up to them, it just won't work with salt-ssh targets less than py3.7.

Take care that the thin package should be regenerated after contextvars installation.

lastmikoi · 2022-06-28T19:08:42Z

Thanks for the detailed walkthrough @piterpunk.

I've followed your instructions and can confirms that on my setup (Archlinux source salt-ssh on a Python 3.9 virtualenv, Rockylinux 8 target using the system Python 3.6), installing contextvars in the source virtualenv then re-generating the thin package does not seem to be sufficient to work the issue around.

This is particularly confusing considering contextvars==2.4 is already installed on the target host, as part of a regular salt-3004 on python3.6 install.

salt --versions-report

Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: 1.15.0
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.15.0
        pygit2: 1.5.0
        Python: 3.9.12 (main, May 15 2022, 19:24:24)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 21.0.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3
 
System Versions:
          dist: arch  
        locale: utf-8
       machine: x86_64
       release: 5.18.6-arch1-1
        system: Linux
       version: Arch Linux

salt-call --versions-report (on target host)

Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Apr 12 2022, 06:55:39)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 8.6 Green Obsidian
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-372.9.1.el8.x86_64
        system: Linux
       version: Rocky Linux 8.6 Green Obsidian

EDIT: Turns out I've stumbled upon this issue already back in January and I found another workaround on a duplicate issue, #59942 (comment) that's related to an issue with typing_extensions. My memory is poor !

To whom it may concern, if Piter's instructions aren't sufficient because your issue is caused by typing_extensions, install typing_extensions (version 4.1.1 maximum, 4.2.0 introduces a breaking change...), then follow those instructions #59942 (comment)

fs30000 · 2022-12-16T20:31:26Z

I just installed salt-ssh on a centos 7 via rpm repo. When connecting to another centos 7 machine, after installing python3.x86_64 and python36-distro.noarch (because got another error as well), got this error:

salt-ssh -v -i '*' test.ping
Executing job with jid 20221216202157353872

vm34:
----------
retcode:
1
stderr:
Traceback (most recent call last):
File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 10, in
import _contextvars as contextvars
ModuleNotFoundError: No module named '_contextvars'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/salt-call", line 27, in <module>
        salt_call()
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/scripts.py", line 435, in salt_call
        import salt.cli.call
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/call.py", line 3, in <module>
        import salt.cli.caller
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/caller.py", line 12, in <module>
        import salt.channel.client
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/channel/client.py", line 13, in <module>
        import salt.crypt
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/crypt.py", line 26, in <module>
        import salt.payload
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/payload.py", line 12, in <module>
        import salt.loader.context
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/__init__.py", line 15, in <module>
        import salt.config
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 100, in <module>
        _DFLT_IPC_WBUFFER = int(_gather_buffer_space() * 0.5)
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
        import salt.grains.core
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/grains/core.py", line 31, in <module>
        import salt.modules.cmdmod
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/modules/cmdmod.py", line 32, in <module>
        import salt.utils.templates
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/utils/templates.py", line 27, in <module>
        from salt.loader.context import NamedLoaderContext
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 13, in <module>
        import contextvars
    ModuleNotFoundError: No module named 'contextvars'
    [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
    ModuleNotFoundError: No module named 'contextvars'
    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 10, in <module>
        import _contextvars as contextvars
    ModuleNotFoundError: No module named '_contextvars'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/salt-call", line 27, in <module>
        salt_call()
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/scripts.py", line 435, in salt_call
        import salt.cli.call
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/call.py", line 3, in <module>
        import salt.cli.caller
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/caller.py", line 12, in <module>
        import salt.channel.client
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/channel/client.py", line 13, in <module>
        import salt.crypt
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/crypt.py", line 26, in <module>
        import salt.payload
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/payload.py", line 12, in <module>
        import salt.loader.context
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/__init__.py", line 15, in <module>
        import salt.config
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 100, in <module>
        _DFLT_IPC_WBUFFER = int(_gather_buffer_space() * 0.5)
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
        import salt.grains.core
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/grains/core.py", line 31, in <module>
        import salt.modules.cmdmod
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/modules/cmdmod.py", line 32, in <module>
        import salt.utils.templates
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/utils/templates.py", line 27, in <module>
        from salt.loader.context import NamedLoaderContext
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 13, in <module>
        import contextvars
    ModuleNotFoundError: No module named 'contextvars'
stdout:

EDIT: OK, doing pip3 install contextvars fixed it

dmacvicar added Bug broken, incorrect, or confusing behavior needs-triage labels Jan 4, 2022

github-actions bot assigned twangboy Jan 4, 2022

OrangeDog added dependency underlying Salt dependency issue Salt-SSH labels Jan 5, 2022

OrangeDog added Duplicate Duplicate of another issue or PR - will be closed and removed needs-triage labels Jan 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails #61419

[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails #61419

dmacvicar commented Jan 4, 2022

OrangeDog commented Jan 5, 2022 •

edited

robertchen commented May 10, 2022

rittycat commented May 10, 2022

rittycat commented May 11, 2022

ptitdoc commented Jun 2, 2022

piterpunk commented Jun 6, 2022

lastmikoi commented Jun 28, 2022 •

edited

fs30000 commented Dec 16, 2022 •

edited

[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails #61419

[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails #61419

Comments

dmacvicar commented Jan 4, 2022

OrangeDog commented Jan 5, 2022 • edited

robertchen commented May 10, 2022

rittycat commented May 10, 2022

rittycat commented May 11, 2022

ptitdoc commented Jun 2, 2022

piterpunk commented Jun 6, 2022

lastmikoi commented Jun 28, 2022 • edited

fs30000 commented Dec 16, 2022 • edited

salt-ssh -v -i '*' test.ping Executing job with jid 20221216202157353872

OrangeDog commented Jan 5, 2022 •

edited

lastmikoi commented Jun 28, 2022 •

edited

fs30000 commented Dec 16, 2022 •

edited

salt-ssh -v -i '*' test.ping
Executing job with jid 20221216202157353872