Info command

[dansondergaard]

A lot of flexibility could be gained by allowing some common filtering operations (should run, completed, failed, submitted, running) and allowing the user to provide a format string to print various information in a space/comma-separated format.

[dansondergaard]

@kaspermunch A specification/set of requirements for this would be nice so that we can make sure that the implementation fulfils your needs.

[kaspermunch]

It would be nice to be able to have complete freedom in what to print (not only space/comma separated fields). Like "This {jobid} depends on {inputs} and produces {outputs}".

[dansondergaard]

Target inputs and outputs are not really status information. I think stuff like that would fit better under a different subcommand such as "gwf info".

[kaspermunch]

But it would be very convenient to be able to query status and info at the same time. I often find myself looking for the output files of the targets that failed.

[dansondergaard]

Maybe that could be solved through chaining, if we could make something like this possible:

$ gwf status --failed --name-only | gwf info --format "{name} {outputs}"

I'm just concerned that the status subcommand becomes a huge beast because people want it to print everything imaginable.

[dansondergaard]

So, I've implemented the first part of this. Please, please, please test it! :)

▸ gwf status --help                                                                                                                                                                                master gwf env 13:29:11
Usage: gwf status [OPTIONS] [TARGETS]...

  Show the status of targets.

  By default, shows a progress bar for each endpoint in the workflow. If one
  or more target names are supplied, progress bars are shown for these
  targets.

  A progress bar represents the target and its dependencies, and shows how
  many of the dependencies either should run (magenta, .), are submitted
  (yellow, S), are running (blue, R), are completed (green, C), or have
  failed (red, F).

Options:
  -n, --names-only
  --all / --endpoints             Whether to show all targets or only
                                  endpoints if no targets are specified.
  -s, --status [shouldrun|submitted|running|completed]
  --help

It took some time since I had to iron out a bunch of confusing bugs and misunderstandings in the way status is reported. The changes benefited all of gwf, so I think it was worth it. I've tested with a small workflow (not unit tests yet) using the local and slurm backends, and status seems to be accurately reported now.

Next step is to implement a info command so that @kaspermunch can get the information he wants :)

[dansondergaard]

@kaspermunch suggested that the gwf info command could be even simpler, e.g.

$ gwf info TargetName
name: TargetName
options:
    cores: 2
    memory: 8g
    ...
spec:
    THIS IS THE SPEC
dependencies:
    - dep1
    - dep2
    - dep3
    ...
dependents:
    - dep1
    - dep2
    ...

[mailund]

That output is simple YAML, and I like that. There are lots of libraries for parsing it if people want to write tools they can pipe the info into, and it is very readable for humans as well.

[dansondergaard]

I (very) intentionally wrote it as YAML ;)

[kaspermunch]

"conda install -c gwforg gwf” giver mig b10 beta versionen, ikke 1.0. Er der noget jeg gør galt? Cheers, Kasper On 14 Aug 2017, 15.01 +0200, Dan Søndergaard <notifications@github.com>, wrote: Closed #141<#141> via 6dbdbb0<6dbdbb0>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#141 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABHPBLCiVwwWl76JRXZMpud3BCHeOUlUks5sYEUkgaJpZM4MBmCc>.

[dansondergaard]

Nope, see #181.