Proposal for putting "special task" config in the graph.

[hjoliver]

In [scheduling][[special tasks]] special behaviour is bestowed upon members of a list of task names, each name with the "special" configuration parameters (if any) attached to it. This works OK for sequential tasks, which have no configuration parameters; and for clock-triggered tasks, which just take a short date-time offset string, but it's a very ungainly way to specify the much less concise external trigger messages of #1316.

The problem could be solved by relaxing our separation of scheduling and runtime and declaring clock-triggers etc. under [runtime]. Special behaviour could then be inherited, although we already achieve that to some degree by allowing family names in special task declarations. I'm not expecting this to be a popular idea though!

Another solution could be to move external triggers (clock and external events) to special graph nodes, which would have the advantage of exposing all scheduling events to visualisation, something like this:

graph = """
   <CLOCK(PT6H)> => foo
   <EXTERNAL("New satellite dataset ready for use")> => bar"""

(the only subtlety here is that the clock trigger is still relative to the cycle point of the task that triggers off it, not absolute).

[matthewrmshin]

I like the graph solution.

[arjclark]

The only thing against moving clock triggering into the graph that I can immediately think of is that we'll go from one entry to N, where N is the number of graph sections foo appears in. Potentially, this becomes a bit of a maintenance nuisance but could be got round with Jinja2 variables. I suspect the result might end up something like:

{% set CLOCK_OFFSETS = {
    "foo" : "PT1H",
    "bar" : "PT30M"
    ...
    } %}

[scheduling]
    [[dependencies]]
        [[[R1]]] # some combination of cold-start tasks where suite can start at any cycle
            <CLOCK{{CLOCK_OFFSETS["foo"]}}> => foo
        [[[T00, T12]]] # regularly cycling
            <CLOCK{{CLOCK_OFFSETS["foo"]}}> => foo
        [[[T07]]] # Irregular/non-standard cycle
            <CLOCK{{CLOCK_OFFSETS["foo"]}}> & special_task => foo

[arjclark]

@dpmatthews has pointed out that the above could be reduced down to:

[scheduling]
    [[dependencies]]
        [[[R1, T00, T12, T07]]] # Common clock triggering dependencies
            <CLOCK(PT1H)> => foo
        [[[T07]]] # Irregular/non-standard cycle
            special_task => foo

[arjclark]

While we're looking into clock triggering etc. it might also be useful to one day have this kind of triggering:

[scheduling]
[[dependencies]]
graph = foo+PT1H => bar

where bar is triggered 1 hour after foo has finished.

Of course, at the moment we can just do:

[scheduling]
[[dependencies]]
graph = foo => sleepPT1Htask => bar

I've seen one case of this where it has been necessary to stagger tasks like this due to timings and limitations in downstream systems but could prove useful for others.

[benfitzpatrick]

Can we lowercase <CLOCK and <EXTERNAL? I think it would look nicer. I know we already use caps for suite polling, but we could change that as well...

[matthewrmshin]

What about $clock as opposed to <clock>?

[hjoliver]

Can we lowercase ...

yes.

What about $clock ...

I'm not sure it's a good idea to have it look like a shell variable?

[matthewrmshin]

What about $clock ...

I'm not sure it's a good idea to have it look like a shell variable?

That's the idea, as it can be considered a built-in cylc graph variable.

[hjoliver]

[meeting]

• agreed on syntax: angle brackets;
  • leave as "later".