Task proxy spawn on demand.

[hjoliver]

Spawn on Demand implementation as per proposal https://cylc.github.io/cylc-admin/proposal-spawn-on-d.html

See #3474 (superseded) for earlier discussion and some proof of efficiency.

• partially addresses spawn on demand / event driven graph #3304
• close reload and restart - smart insertion of new tasks #774
• close Users should not need to know about insertion (or task proxies generally) #2143
• close graph: optional branching #3226
• (probably closes other issues too...):
• close limit task pool size #987
• close Submit job immediately on insertion? #2643
• close Release, trigger, insert (etc.) a whole sub-graph #1314
• close reset-to-waiting when prerequisite tasks have been removed #2654

This branch:

• correctly runs the "complex" test workflow for multiple cycles (last I checked, recently)
• handles reflow properly as described in the proposal
• seems to work for all manner of manual test cases

What's still to do:

• restart: need to record upstream output status for each task (for prerequisites) in the DB, not just task status
• refine the cycle point based housekeeping to account for inter-cycle dependence (no longer needed)
• unit and functional tests (many mods needed)
• implement reflow stop mechanisms as described in the proposal
• implement cylc spawn for specific individual outputs

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).

• Appropriate tests are included (unit and/or functional).
• Already covered by existing tests.

• Appropriate change log entry included.

• (master branch) I have opened a documentation Issue at cylc/cylc-doc#132

[hjoliver]

Restart now implemented, and tests/restart (non-remote) functional tests working. (Description updated above)

[hjoliver]

Latest Travis-CI results on this branch:

• All unit tests pass
• Failed 2 functional tests out of 1867 (in 2 test scripts out of 577)
  • graphql/02-root-queries.t DONE
  • offset/04-cycle-offset-chain.t DONE
  • flakytests/ DONE

[dwsutherland]

@hjoliver - Got graphql/02-root-queries.t to pass with:

(flow) sutherlander@cortex-vbox:cylc-flow$ git diff
diff --git a/tests/graphql/02-root-queries.t b/tests/graphql/02-root-queries.t
index 02fc49682..386d769bf 100755
--- a/tests/graphql/02-root-queries.t
+++ b/tests/graphql/02-root-queries.t
@@ -144,18 +144,12 @@ cat > expected << __HERE__
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}baa"
         },
-        {
-            "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}bar"
-        },
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}foo"
         },
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}qar"
         },
-        {
-            "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}qaz"
-        },
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}qux"
         }

Ran it 3 times to check consistency (but haven't run it on a different/faster machine).

[hjoliver]

Thanks @dwsutherland - if you're happy that's the correct expected result, I'll make your change and leave the test like that for now.

I see my latest change has caused the unit tests to fail 😬 (I know why though, easy to fix though).

[hjoliver]

Unit tests fixed, 3 or 4 "absolute offset" tests broken now, as a result of previous change. Damn it, almost there.

[hjoliver]

All tests passing 💥

[hjoliver]

So I think we just need to consider absolute triggers in that bit of the validation code. I'll fix it on this branch if easy to do...

Done in latest commit. Other issues raised all addressed except for the "absolute triggers + sequential tasks" one. (I'll comment on that soon...)

[hjoliver]

@dwsutherland - feedback addressed. Your examples above work, just need to add a test for the improved absolute trigger handling [done]

[dwsutherland]

@hjoliver - Great, when trying to run the suite:

[scheduling]
    initial cycle point = 20200101T00
    [[special tasks]]
        sequential = foo
    [[dependencies]]
        #R1 = wiz
        P1M = """
#foo[-P1M] => foo
wiz[^] => foo
"""

I get the following:

(flow) sutherlander@cortex-vbox:wiz$ cylc validate wiz
TaskDefError: No cycling sequences defined for wiz

2020-07-29T12:34:40+12:00 INFO - Cold Start 20200101T0000+12
2020-07-29T12:34:40+12:00 INFO - Suite server: url=tcp://cortex-vbox:43093/ pid=10184
2020-07-29T12:34:40+12:00 INFO - Suite publisher: url=tcp://cortex-vbox:43064
2020-07-29T12:34:40+12:00 INFO - Run: (re)start=0 log=1
2020-07-29T12:34:40+12:00 INFO - Cylc version: 8.0a2
2020-07-29T12:34:40+12:00 INFO - Run mode: live
2020-07-29T12:34:40+12:00 INFO - Initial point: 20200101T0000+12
2020-07-29T12:34:40+12:00 INFO - Final point: None
2020-07-29T12:34:40+12:00 INFO - Suite shutting down - AUTOMATIC
2020-07-29T12:34:41+12:00 INFO - [client-command] identify sutherlander@cortex-vbox:cylc-uiserver
2020-07-29T12:34:41+12:00 INFO - DONE

(branch needs de-conflicted again)

Nice work! 🥇

[kinow]

Impressive work @hjoliver !!! 🍾 🍻

[TomekTrzeciak]

This one's big! I'm really happy to see this happening already for Cylc 8 🎉 🍾 🚀

[arjclark]

@hjoliver - does this need addressing still?

[hjoliver]

Good spotting. The variable name suggests we thought the value is boolean. It's actually of no consequence as (I checked) the variable is only used in if tests (where '^' vs None behaves just like True vs `False). However, I'll follow-up with a small change to make the code less obtuse.