Extensions

Writing a QuNex extension

An extension lets you add your own commands to QuNex — in Python, MATLAB or bash — without touching the QuNex installation. Your command is invoked as qunex my_command --parameter=value, takes its parameters from the same places every QuNex command does, writes the same logs, and can be sent to the same scheduler.

This page is the practical guide: enough to write a working extension and to recognise the ways one usually fails. The QuNex extensions whitepaper carries the full picture, and Command registry and command types specifies the docstring format, which is the same for an extension as for the suite's own commands.

The one idea to hold on to

Two things have to be true, and that is all.

• QuNex has to be able to see the folder. It looks in a fixed set of search roots on every call and takes every qx_* folder it finds there to be an extension.
• The registry has to know your commands. It is built from the docstrings in your code by qunex build_qx_extensions, which writes a qx_commands.yaml into your extension folder. Nothing rebuilds it for you.

From those two QuNex works out the rest by itself: it knows which folder each command came from, so your Python modules go on the Python path, your matlab folder onto MATLABPATH for a MATLAB call, your bash scripts are looked up under your own bash folder, and your bin folder goes on PATH for the QuNex process. Nothing has to be set up in advance.

The one habit: rebuild the registry after changing a docstring.

Setting one up

An extension is a folder whose name starts with qx_, sitting inside a search root. There are three roots: $QUNEXPATH/qx_extensions, $TOOLS/qx_extensions, and anything listed in QUNEXEXTENSIONSFOLDERS. The last is the one to use for code you maintain yourself.

# a search root of your own, holding one extension
mkdir -p $HOME/qunex_extensions/qx_example/python

# QUNEXEXTENSIONSFOLDERS names the root -- the folder that *contains* qx_example.
# QuNex reads it on every call; nothing has to be re-sourced
export QUNEXEXTENSIONSFOLDERS="$HOME/qunex_extensions"

Sourcing qunex_environment.sh is not needed for any of this to work, but it does print a report, which is a quick way to confirm a search root is set the way you think it is:

QuNex extensions identified
---> Registering extension qx_example
    ... setting QXEXAMPLEPATH to '/home/user/qunex_extensions/qx_example'
    ... added /home/user/qunex_extensions/qx_example/python to QXEXTENSIONSPY

What sourcing adds is that the per-extension variables and the bin folder land in your shell rather than in QuNex's process. That is worth having while you work on an extension by hand, and it is the only way to call a bin script by name at your own prompt. Inside a container it is not available at all — the environment is sourced once at start-up and cannot be sourced again — which is exactly why QuNex does not depend on it.

The folder can hold up to five things:

Folder	What goes in it
python/	Python commands and library code, plus the optional qx_modules file
matlab/	MATLAB functions; the folder is added to MATLABPATH whole
bash/	bash scripts that are QuNex commands
bin/	plain scripts, put on PATH; not QuNex commands
lib/	anything else your code needs; exported as <EXT>LIB

<EXT> is the folder name upper-cased with underscores removed, so qx_example gives QXEXAMPLEPATH, QXEXAMPLEBIN and QXEXAMPLELIB. Use them so your code can find its own data wherever it has been installed.

python/qx_modules

This file is optional, and it does one job: it names the modules QuNex should import at start-up, on every call. That means the module carrying your parameter declarations, which have to be in memory before any command runs, and nothing else.

# one module name per line; '#' starts a comment
qx_example_options

Do not list the modules holding your commands. QuNex puts your python folder on the Python path by itself and imports a command's module when the command is run, so naming it here only slows down every unrelated call. An extension that declares no parameters of its own needs no qx_modules file at all.

Writing a command

A function is a QuNex command when its docstring carries a .. qx_command: block. The format is specified on Command registry and command types; in brief, the docstring needs a call line in double backticks, a description paragraph, the qx_command block with at least a type:, and a Parameters: block.

def example_hello(example_name="world", example_times=1):
    """
    ``example_hello [--example_name=<name>] [--example_times=<n>]``

    Greets whoever is named -- the smallest useful command an extension can ship.

    ..  qx_command:
        type: utility

    Parameters:
        --example_name (str, default 'world'):
            Who to greet.

        --example_times (int, default 1):
            How many times to greet them.
    """

    print("\n".join(["Hello, %s!" % example_name] * int(example_times)))

Your extension's code can import from the QuNex library directly — the suite's python folder is already on the path:

import qx_utilities.general.log as gl            # the log classes
import qx_utilities.general.exceptions as ge     # CommandError, CommandFailed

A processing command has the fixed signature (sinfo, options, overwrite=False, thread=0), is called once per session, and returns its log object. The three processing.* types, and what arrives as sinfo under each, are on Command registry and command types; the log object is on Logging.

Declaring parameters

This is where extensions most often go subtly wrong, because two separate declarations are involved and they do different jobs.

The docstring says what the command accepts. Its Parameters: block, with the function signature, is what QuNex checks a command line against, what it narrows the run's parameters down to, and what it prints in the parameter report. For a utility command it is a gate: an undeclared parameter is rejected outright.

arglist says what a parameter defaults to and how its value is read. It lives in a module named in qx_modules:

# qx_example_options.py


def is_set(value):
    """A converter is any callable: this one reads the flag, or a written yes."""
    return value in (True, "yes", "true", "TRUE", "True", "1")


# each entry is [name, default, converter], with an optional fourth description
arglist = [
    ["# ---- qx_example settings"],       # one element: a heading in the listing
    ["example_name", "world", str, "Who to greet."],
    ["example_times", 1, int],            # "3" from the command line becomes 3
    ["example_shout", False, is_set],
]

# parameters that may be given without a value: --example_shout means =True
flaglist = [
    ["example_shout", True],
]

The rule that follows: a parameter a processing command reads out of options needs an arglist entry. A processing command is handed the merged options dictionary, so options["example_greeting"] raises KeyError on any run where nobody named the parameter and nothing gave it a default. Declare it in the docstring so the command accepts it; declare it in arglist so it is always there.

The converter has to be something QuNex can call. A type annotation such as Optional[str] looks like a type but cannot be applied to a value; QuNex reports it and leaves the parameter unconverted rather than failing the run.

A few more optional declarations can sit in the same module, collected across every extension: extra_parameters, deprecated_commands, deprecated_parameters, deprecated_values, to_impute, towarn_parameters and logskip_commands. The whitepaper describes each.

MATLAB and bash

The docstring format is the same; only its placement differs.

For MATLAB, put the qx_command block in the help comment block after the function line, and use type: matlab. QuNex calls the function by name, adding your matlab folder to MATLABPATH for the call, and passes the arguments positionally, in the order of the function line, quoted according to the type documented for each. Document every argument's type: an undocumented one is passed through with a warning, and one badly quoted argument shifts all the rest. Give each argument a default inside the function, since QuNex passes an empty value for arguments the user did not name.

If your MATLAB code has subfolders, list them one per line in matlab/matlabpaths, relative to the matlab folder, and they are added to MATLABPATH too.

For bash, the docstring is the text inside the script's usage() heredoc and the qx_command block must declare language: bash. QuNex passes each declared parameter as --name='value'. A script has no signature, so the Parameters: block is its interface: a parameter it does not declare is never passed, and one it declares must be spelled the way the script reads it.

Keep bash/ and bin/ straight: a script in bash/ is a QuNex command, a script in bin/ is on PATH under its own name and is not. Both have to be executable — a copy that lost its file modes fails with Permission denied rather than command not found.

Building the registry

# rebuild the registry of the extension called qx_example
qunex build_qx_extensions --extensions=example

This builds the registry of the extensions you name and leaves the QuNex installation's own registry alone, which is what you want as an extension author and what a read-only installation — a shared cluster install, or a container — requires. Run it after every change to a command's declaration: a new or removed command, a rename, or an edit to the qx_command block, the call line, the description or the Parameters: entries.

You have to name what to build. Give the name with or without the qx_ prefix, give several separated by commas, or give all for every extension QuNex can find:

qunex build_qx_extensions --extensions=qx_example,mytools   # two of them
qunex build_qx_extensions --extensions=all                  # all of them

--extensions=check builds nothing and lists what QuNex can see instead — each extension, the search root it was found under, and whether it has a registry yet. That is the quickest way to confirm QuNex is looking where you think it is:

--> Extensions QuNex can see (2):
    qx_example  /opt/qx_extensions/qx_example  [registry built]
    qx_mytools  /opt/qx_extensions/qx_mytools  [no registry yet]

build_qx_registry is the command underneath, and it rebuilds the suite's own registry as well as the extensions'. You want it when you are working on QuNex itself, not when you are working on an extension.

The registry is written beside your code, as qx_commands.yaml in the extension folder, and holds paths relative to that folder. It travels with the extension: build it once and the same file works wherever the folder is copied or mounted.

Read the output. A command that fails to register is not an error, so the symptom is absence:

--> Leaving the core command registry as it is: /opt/qunex/qx_commands.yaml
    -> registering qx_example_commands.example_hello
...
--> Built 1 extension registry:
    - extension:example: /home/user/qunex_extensions/qx_example/qx_commands.yaml

A command is excluded when its docstring has no call line, no qx_command block, or no type:. A duplicate command name or alias within one registry stops the build.

Developing in a container

Nearly everyone runs QuNex from a container, so it is worth setting your extension up to be worked on that way from the start. Bind the folder that holds your qx_* folder onto /opt/qx_extensions, which is $TOOLS/qx_extensions inside the container and one of the roots QuNex always searches:

# the extension lives on the host and is bound in; nothing is set up inside
qunex_container --interactive \
    --container=/path/to/qunex_suite-latest.sif \
    --bind=/home/user/qunex_extensions:/opt/qx_extensions

From there the cycle is the ordinary one. Edit the code on the host, rebuild inside the container, run the command:

qunex build_qx_extensions --extensions=example
qunex example_hello --example_name=Container

The registry is written beside your code, and your code is on a bound folder, so the rebuild lands on the host and the next container you start already has it.

What you cannot do is rebuild the suite's registry. It lives on the container image, which is read-only, and build_qx_registry says so rather than failing obscurely:

---> ERROR in completing build_qx_registry:
     Registry location is not writable
     ...
         qunex build_qx_extensions --extensions=<name>

Nothing about registering an extension needs it, which is why build_qx_extensions is the command to reach for. The same message appears if the extension folder itself is read-only — an extension baked into an image rather than bound in — and the answer there is to bind it in from the host.

Two container properties shape the rest. The environment is sourced once, when the container starts, and cannot be sourced again, so an extension installed afterwards will not have its variables or its bin folder in your shell; QuNex finds its commands anyway, because it reads the search roots on every call. And an extension bound in before the container starts is registered by the environment as usual, so its bin scripts are callable at the prompt too.

Overriding a QuNex command

An extension command with the same name as one the suite provides replaces it for every run in that environment. Nothing on disk changes, and removing the extension restores the original.

It is a real tool — the way to change a command's behaviour for a study or a site without patching the installation — but use it deliberately. The run says so above its parameter table, once, and the same line goes into the log:

---> Command check_study is provided by extension example, replacing the core command of the same name

A command from an extension that replaces nothing prints the first half of that line on its own; a command from the suite prints none. To see what an override does before running it, ask for qunex <command> --help, which renders the docstring of whichever implementation is active.

When it does not work

Symptom	Cause
Requested command is not supported	The registry does not have the command: build_qx_extensions not run, or the command not registered — check the build output. If nothing was found at all, QUNEXEXTENSIONSFOLDERS is probably naming the extension rather than the folder containing it; qunex build_qx_extensions --extensions=check lists what QuNex can actually see.
Registry location is not writable	The registry cannot be written where it has to go. For the suite's own registry that is normal inside a container, and the message names build_qx_extensions as what to run instead. For an extension it means the extension folder itself is read-only — bind it in from the host rather than building it onto a container image.
WARNING: extensions folder '...' does not exist	QUNEXEXTENSIONSFOLDERS names a folder that is not there. QuNex skips it and carries on with the other roots.
ModuleNotFoundError: No module named '...'	Something your command imports is missing. If the name is one of your own modules, the registry describes source that has moved — rebuild. If it is a third-party package, install it into the Python environment QuNex runs under.
KeyError on your own parameter	A processing command read a parameter from options with no arglist entry — or with one in a module that qx_modules does not name, so QuNex never imported it.
ERROR: Extra argument ... is not valid	A utility command was given a parameter it does not declare. Add it to the signature and the Parameters: block, then rebuild.

A worked extension with a Python utility command, a Python processing command, a MATLAB command, a bash command and a bin script is shipped with this wiki, in Examples/qx_example. The QuNex extensions whitepaper installs and runs it, step by step.