capture instance config in jobspec for batch and alloc jobs

[grondo]

From an offline discussion with @garlick:

Currently, the only way to specify a configuration for a subinstance run via flux mini alloc or flux mini batch is to pass a --config-path=PATH option down to the broker via the flux-mini --broker-opts option, e.g.

flux mini alloc --broker-opts="-c /home/grondo/myconf"

However, there are several drawbacks here:

• the configuration under /home/grondo/myconf could be updated between job submission and job start
• the configuration isn't captured along with the job record in any way, besides its path
• small updates to the configuration can't be made without copying the config to a per-job directory

Instead of leaving the configuration to be read at runtime by the broker from the original filesystem path, the config should be read by flux mini batch or flux mini alloc and the resulting JSON representation placed into a well-known attribute in the jobspec. The job shell batch plugin can then write this configuration out to a temporary directory and pass it along to the brokers via the -c option.

A --setconf or similar option should also be added, which would amend the loaded (or default empty) configuration in the jobspec, allowing easy one-off updates to a more generic configuration placed on disk.

We have also previously discussed the use of "named" configuration paths. These could be specified by name on the flux mini command line, with the option of a "default" system or user provided named config that is used as a starting point when no configuration name is specified.

The system could provide a set of named configs, which could be overridden by users in a ~/.flux/configs directory or similar. As noted by @garlick, this would allow workflows to provide standard configs that could easily be selected by users.

[grondo]

cc: @trws - this is the issue I was thinking about in our recent slack discussion.

Some groundwork has been laid to get this done:

• broker: allow a file argument to -c, --config-path in TOML or JSON #4949 - allows the broker to read a single config file instead of requiring a directory and glob pattern
• Support {{tmpdir}} in shell mustache templates and simplify batch jobspec construction #4951 - allows the shell to place a config file in tmpdir and call the broker with -c {{tmpdir}}/conf.json
• rfc37: add File Archive Format RFC rfc#369 - added a File Archive Format defining how to store generic files in jobspec

with that foundation laid, I expected to have basic support for embedding config under a well known key (may need an RFC 14 update) in jobspec, with a method for updating the config on the command line.

We then hope to add a way to established some kind of named configuration scheme (perhaps should open a separate issue on this) allowing users to easily select a starting config, e.g. --conf=node-exclusive.

[garlick]

Can we close this?

[grondo]

Oh yeah, sorry. This should have been closed by #5232