docs: container publishing
b46ba44
docs: build_qx_extensions, and developing an extension in a container
An extension author is now given `build_qx_extensions` rather than
`build_qx_registry --build_core=no`. The extensions page, the whitepaper
and the worked example all use it: naming what to build, with or without
the `qx_` prefix, several at once or `all`, and `--extensions=check` to ask
what QuNex can see. `--build_core` survives only in the whitepaper's
account of `build_qx_registry`'s own parameters, where the distinction
between the suite's registry and an extension's genuinely is the subject.
Each of the three now covers developing an extension **inside** a
container, which none of them did. The extensions page had no container
section at all. The point in each is the same: the registry is written
beside the extension's own code, so with the folder bound in from the host
the rebuild lands there and survives the container, while the suite's own
registry cannot be rebuilt from inside because it is on a read-only image.
The error QuNex gives for that is shown, along with the reason it names
`build_qx_extensions` rather than leaving the reader to work it out.
The extensions page's troubleshooting table gains a row for a registry
location that cannot be written, and its "Requested command is not
supported" row now points at `--extensions=check` for the case where QuNex
cannot see the extension at all. The command registry page stays about the
core registry, and gains a pointer for the extension author and a note on
the location check.
Every quoted output is from a real run. The whitepaper's summary line was
stale on two counts: it read "In addition to core" where core had been left
alone, and counted "1 extension registries".
a92ff77
docs: document how QuNex extensions are written and run
QuNex can be extended with commands of your own, and until now nothing
described how. The mechanism had also changed: an extension used to
register its commands explicitly, and since the arrival of the command
registry a command is registered by carrying a `.. qx_command:` block in
its docstring, exactly as a command of the suite is. Parameters are
still declared explicitly, and the division of labour between the two
declarations involved was nowhere written down.
Two pages are added and linked from Home. `Whitepaper-Extensions.md`
covers the mechanism end to end: what an extension is made of, where
QuNex looks for one, how commands and their parameters are declared, the
path from a command line to the code, the MATLAB and bash cases, and how
an extension replaces a command the suite provides. `Extensions.md` is
the working guide — the same ground condensed to what is needed while
writing one, cross-linked to the whitepaper and to the command registry
page. `Command-registry.md` gains a pointer to both and a missing row
for the `matlab` command type.
Both pages describe the mechanism as it now behaves, which is not how it
behaved when they were first drafted. An extension no longer depends on
the environment script having seen it: it resolves its Python, MATLAB
and bash code from the registry the command came from, which is the only
way an extension can work inside a running container, where the
environment is sourced once at start-up and cannot be sourced again. So
the pages present two conditions — a folder QuNex can see, and a
registry that describes it — and state separately what sourcing the
environment adds, which is that the per-extension variables land in the
user's shell rather than in QuNex's process. Along with that:
`python/qx_modules` is optional and only names the modules to import
eagerly; an `arglist` entry may carry a description as a fourth element,
and a converter that cannot be called is reported rather than fatal; a
run whose command comes from an extension says so above its parameter
table; `build_qx_registry --build_core=no` is what an extension author
runs, and an extension's registry travels with it because it holds
relative paths. Both pages carry a section on extensions and containers,
which is the deployment shape most installations actually have.
The comments shipped with `Examples/qx_example` are corrected to match:
`qx_modules` no longer claims to be required, the example's `arglist`
shows the four-element form, and the README tells the reader that the
example arrives with its registry already built.
Every claim that changed was checked by running the example extension
against a QuNex installation rather than by reading the source. That run
also corrected one claim nobody had verified: `qunex --a` prints one
`- name: description` line per command, not a space-separated list.
d45d9d1
docs: add the qx_example extension, registry included
A worked example of a QuNex extension, to be read alongside the
extensions documentation: one python utility command, one python
processing command, a MATLAB command and a bash command, plus the
declaration module an extension uses to give its parameters defaults and
types, and a plain script in `bin` that is deliberately not a command.
Every file is commented for somebody writing their first extension
rather than for somebody maintaining this one. `python/qx_modules` says
what it is for and why the module holding the commands is not listed in
it; `python/qx_example_options.py` says what an `arglist` entry declares
that a docstring does not; `bin/example_helper.sh` says why it is not
reachable as `qunex example_helper.sh`.
`qx_commands.yaml` is committed with it. It is what makes the example
runnable straight from a copy -- bind or copy the folder into an
extensions root and the commands are there, with no build step first --
and a published extension should carry one in any case, being invisible
to QuNex until somebody builds it. The file holds relative paths only,
so it serves wherever the extension is put.
It does have to be rebuilt whenever a command's docstring changes:
QUNEXEXTENSIONSFOLDERS=<the folder holding qx_example> \
qunex build_qx_registry --build_core=no
12842d0
docs: added pooh publishing guide
9696fee
docs: document log.rule() and how a report's prose is written
`ReportLog.rule()` draws the full width rule that separates one part of a
session report from the next. It takes `before` and `after` for the
blank lines around it, and `char` for what it is drawn with, so a report
can divide at more than one weight without the rules going to different
lengths.
*Logging* gains a row in the vocabulary table, a short block on
horizontal division beside the one on nesting, and two new pitfalls: a
paragraph of prose folded into one string with its own `\n `, and a
parameter block written as one format string. The existing pitfall on
hand-typed indentation now names rules too, since that is the same
mistake and it has a method now.
*Whitepaper-Logging* gains two sections. "Furniture is drawn, not typed"
covers `blank`, `rule` and `framed` together, and states the property
`rule` exists to protect -- the width does not change with the
character. "Prose in a report" covers the preamble at the head of a
session report: a module level triple-quoted constant rather than a
string carrying its own wrapping, `textwrap.fill` where it has
substitutions in it, and the parameters a command quotes back kept as a
list the report loops over rather than a format string that drifts from
`options`.
The section on `raw()` is now two shapes rather than three. Framing was
one of them, and it is no longer something `raw()` is reached for.
*Developer utilities* gains the three calls in its example block.
41d0bdc
docs: record the batch tier unsets and where each is applied
`Invocation` gains the two run level parameters that refuse a batch
tier, and the two implementation points a developer needs before
touching them.
The header filter runs twice, around `check_deprecated_parameters`,
because the names a header states and the names it contributes are not
the same set: `_log: study` contributes `log` and `comlog_folders`,
the second under a name nobody wrote. Filtering on both sides is what
makes either spelling work, and the remap being idempotent is what
makes the second pass harmless.
The per-session filter lives in `update_options`, which `gp.run` alone
calls -- so only processing commands ever see that tier, and only they
are affected by unsetting it.
9e33e00
docs: record how a recipe tier is narrowed and when it is withheld
`Invocation` gains the two rules a command author has to know about
the recipe tier. A step can be told not to inherit a parameter, in
either of two spellings read and removed by `recipe._unset_names`
before `select_parameters` runs -- and neither can be an absent value,
because `''` and `None` are values commands accept. And a run wide
parameter naming a file the recipe itself writes is withheld without
being asked: `recipe.PRODUCERS` maps a command to the parameter naming
what it writes and the run wide parameter that names the same file,
and `_withheld_by_production` decides from whether the file exists and
which step writes that path, in the four row table now on the page.
The note on why that decision is made once, before the first step,
rather than at each step: a batch file left by an earlier run is
otherwise indistinguishable from the one this run has just written,
and a recipe has to do the same thing twice.
The sample provenance report in `Invocation` and in the invocation
whitepaper is updated to the column order the report now uses --
parameter, source, value.
3474e10
docs: describe how a command gets its sessions and its parameters
The developer wiki covered the command registry, the reusable classes
and logging, and said nothing about the step between a command line and
a running command: which sessions a command acts on, and where each of
its parameter values comes from. That is now one whitepaper and one
working guide, in the shape the logging pages already set.
`Whitepaper-Invocation` is the full account, in thirteen sections: the
two entry points and the dispatcher, the three steering parameters and
where they are normalised, the one parser and why it imports nothing
but the standard library, the five parameter tiers and the three
dictionaries the merge returns, the order inside `runCommand` and why
it is that order, how each command class is handed its parameters, the
provenance report, the routes a command can arrive by, and the
container splice. Every snippet is quoted from the tree and annotated.
`Invocation` is the working guide condensed from it -- the ideas, the
calls, the rules, the pitfalls -- so that a command author gets the key
points without reading the whitepaper.
Three existing pages carried statements that are no longer true or were
never complete:
- `Developer-utilities` located `SessionList` in `general/core.py`; it
is in `general/batch_io.py`. The page also described the class without
saying how to obtain one, so it now shows `resolve_sessions()`.
- `Command-registry` described the docstring as documentation. It is
also the answer to "what may this command be given", since the
parameter tiers are narrowed to what a command declares -- and for a
matlab or bash command the documented parameters are the whole
interface, there being no python signature to read. That is the fact
a command author most needs from that page.
- `Home` lists both new pages.
github issue: 69
114d030
docs: updated for the new logging infrastructure
Adds a logging whitepaper -- the first entry of a new whitepapers
section on the home page -- describing how logging is implemented: the
runlog and the comlog, the report object and its vocabulary, the log
files and their lifecycle, the settings, the life of a log for a
processing command, a utility command and a recipe, a worked example of
every logging method in both kinds of command, where the logs land, and
every setting with what it does.
Adds a shorter Logging page with the same material in working form: the
key ideas, the shape of each kind of command, the vocabulary table, the
rules, the pitfalls, and how to get the most out of the available
instruments.
Developer utilities and classes gains the logging classes and examples
of the report methods beside the lines they render. Command registry and
command types states that a processing function returns a log object,
in the shared signature, the type table and each of the three processing
sections.
Python only; the bash entry point's logging is not covered.
github issue: 77
be95059
docs: cleaned up unused pages
2589820
fix: put description of pages on home page in new line.
6861480
update: add information of qx_registry, command types and SessionList class
200a733
Updated QuNex container development (markdown)
b6aee06
Updated QuNex container development (markdown)
3d95bdd
Updated QuNex container development (markdown)
a020b69
Updated QuNex container development (markdown)
070dea7
Updated QuNex container development (markdown)
e2c324f
Updated QuNex container development (markdown)
5fb9340
Updated QuNex container development (markdown)
787c7fb
Updated QuNex container development (markdown)
a5c11d6
Updated QuNex container development (markdown)
c879b57
Updated QuNex container development (markdown)
76f8d66
Updated QuNex container development (markdown)
d00ed10
Updated QuNex container development (markdown)
722483e
Updated QuNex container development (markdown)
5ab0411
Updated QuNex container development (markdown)
7fb0756