Command-line tool

[pulkin]

As a foolow-up to @basnijholt presentation about adaptive-scheduler, I'd like to emphasize the need of command-line adaptive tool.

Example syntax:

~ # learns a scalar function in a box [-1, 1] x [-1, 1] with 4 pending processes at max until ctrl-c is hit
~ # my_func.sh is blocking; last floating-point value in stdout is considered as f(x)
~ adaptive-learn --depth 4 --limit none --output-format "%some%format" my_func.sh -1 1 -1 1

Example options:

positional arguments:
  [EXECUTABLE] an arbitrary executable accepting coordinates as arguments
    and outputting function value in stdout as text
    (the last valid float is taken into account)
  [X1 X2 Y1 Y2] ... pairs of floats defining multidimensional box to learn

optional arguments:
  --depth [N] the maximal number of function evaluations in parallel
  --max-fails [N] the maximal number of failures (non-zero EXECUTABLE return codes)
    accepted before exiting (unset by default);
  --limit [STR] the stop condition: 'none' - runs infinitely, 'time' - time limit,
    'eval' - successful evaluations limit;
  --limit-time [DD:HH:MM:SS] ...
  --limit-eval [N] ...
  --output-format [STR] format for text output during the run: 'csv', 'verbose-text', ...
  --non-blocking [EXECUTABLE] assumes the function evaluation is non-blocking
    (i.e. submits a job). The executable specified here checks the job status and outputs
    a float in case the job is finished, or nothing if it still in progress. It should return
    a non-zero code if the job failed in progress. Both 'submission' (first argument) and
    'check' (this argument) executables should accept a unique 6-character alphanumeric
    string ID labelling the job

Why it can be a part of this package?

1. Flattens the learning curve, especially for those who avoid programming in python

• and, in the end, does not require scripting in python at all;

2. Loosens connection between adaptive and function evaluation:

• Removes any python requirements for function evaluation;
• Requires only one working python setup;
• No infrastructure: adaptive just learns your function, wherever the data comes from, wherever it goes and however jobs (processes) are scheduled;
• Thus, becomes compatible with any other infrastructure solution;
• Becomes automatically compatible with lots of HPCs where python does not work, outdated or absent;
• Becomes automatically compatible with lots of software (Matlab, etc)

3. Does not introduce any new concepts, abstractions or workflows and keeps things simple

[akhmerov]

Thanks for the suggestion!

Command line interfaces are harder to maintain and extend. Adaptive supports out of the box vector-valued outputs, custom loss functions, and likely several other things that would be hard to support in a CLI.

This invariably means we'd only be able to provide a small fraction of the useful functionality in a CLI, and the design choice of what should be exposed via this CLI would be hard to maintain and communicate.

I think the best way to enable users to follow a workflow similar to what you described is providing a sample script in the documentation. This would enable immediate use by those interested in trying, while it would also allow tweaking the source.

[pulkin]

I am judging upon my own experience: I know a bunch of people studying phase diagrams which are scalar functions. I followed 3 or so adaptive-related presentations where all examples shown were scalar functions. Thus, I would expect that some basic CLI functionality about scalar functions will find its niche.

[akhmerov]

I fully agree.

I just want to clearly communicate to the users that the intended way of generalizing that CLI we'd provide is to modify the script rather than extend the version of it that we provide.