Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add --conf option for registering instance config in flux-batch and flux-alloc #5232

Merged
merged 9 commits into from Jun 7, 2023
Merged

add --conf option for registering instance config in flux-batch and flux-alloc #5232

merged 9 commits into from Jun 7, 2023

Commits on Jun 7, 2023

  1. python: Jobspec: add conf parameter to from_batch/nest_command() 

    Problem: There is no way to easily create a subinstance jobspec with
an embedded alternate config file via the from_nest_command() or
from_batch_command() constructors.

Add a `conf` parameter to these two Jobspec class factory methods.
If present, this argument will place a `conf.json` entry in the RFC 37
`files` dictionary of the resulting jobspec, and will append

 -c {{tmpdir}}/conf.json

To the broker command line.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    19d8d11 View commit details
    Browse the repository at this point in the history

  2. cli: add --conf=CONF option to flux-alloc,batch 

    Problem: There are no options to specify a config file for subinstances
created by `flux-batch(1)` and `flux-alloc(1)`, but it would be
extremely useful to be able to do so.

Add a `--conf=CONF` option to these commands. The option supports
several types of arguments, and may be specified multiple times to
update the built configuration:

 - If `CONF` is a multiline string, then parse `CONF` as JSON or
   TOML and update config.
 - If `CONF` contains an `=`, then parse `CONF` as config `KEY`=`VAL`
   and update config `KEY` with `VAL`, where `VAL` is parsed as JSON.
 - If `CONF` ends in `.json` or `.toml`, then treat `CONF` as a path to
   a config file to load as JSON or TOML.
 - Future extension: Otherwise, load a "named configuration" file from
   a local repository of config files.

The config is built iteratively using a new BatchConfig class in
cli/base.py
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    e46af33 View commit details
    Browse the repository at this point in the history

  3. testsuite: add tests for flux-batch --conf option 

    Problem: There are no tests that excercise the flux-batch/alloc --conf
option.

Add a new sharness test, t2716-python-cli-batch-conf.t, to test
operation of this option and its various supported arguments.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    23accaa View commit details
    Browse the repository at this point in the history

  4. doc: document flux-batch(1) and flux-alloc(1) --conf option 

    Problem: The --conf option is undocumented.

Add documentation for this option to
man1/common/submit-other-options.rst.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    671a8c2 View commit details
    Browse the repository at this point in the history

  5. python: add flux.util.xdg_searchpath() 

    Problem: Creation of a search path based on the XDG standard may
be useful in multiple places, but the code to do this is currently
embedded in the UtilConfig class.

Move the construction of the XDG search path to its own function
for reusability.

Add an optional `subdir` parameter to the function so that callers
which need to build a set of paths in a subdir of the XDG search path
need not do this manually.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    7250f7c View commit details
    Browse the repository at this point in the history

  6. cli: support named config files in --conf submission option 

    Problem: There is currently no way to register a set of named config
files for use with the --conf=NAME job submission option.

Add support for named configuration files which are loaded from
a searchpath following the XDG specification. This allows system
administrators to register a set of standard configurations that would
be most useful to users under /etc/xdg/flux/config/*.{toml,json},
and users can easily select one by name with the --conf option.
Users can also place their own configs uner ~/.config/flux/config
and use those as named configs as well.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    1dc1747 View commit details
    Browse the repository at this point in the history

  7. testsuite: test named config support for --conf option 

    Problem: The tests in t2716-python-cli-batch-conf.t do not test named
config file support.

Add a set of tests to exercise this functionality.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    8840165 View commit details
    Browse the repository at this point in the history

  8. doc: fix incorrect XDG path in flux-jobs(1) 

    Problem: The flux-jobs(1) CONFIGURATION section specifies that
/etc/flux/xdg is the default search path when XDG_CONFIG_DIRS is not
set, but it should be /etc/xdg/flux.

Fix the incorrect path.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    b82a9d4 View commit details
    Browse the repository at this point in the history

  9. doc: document named configs in --conf submission option 

    Problem: The use of "named" configuration with the submission command
--conf option is not documented.

Document the operation and use of the --conf=NAME option for flux
submission commands.
    @grondo
    grondo committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    1db7d1b View commit details
    Browse the repository at this point in the history