Rose suite run proposals cylc flow.rc

docs/proposal-rose-suite-run.md

@@ -12,6 +12,7 @@ The work described in this document aims to:
 * Change the way hosts for jobs are selected to improve support for clusters.
 * Rationalize the formats of settings `*.rc` files.
 
+
 ## Background
 
 In the early days of Rose, there was the desire for Rose to be an umbrella
@@ -40,16 +41,18 @@ all suites will start up with only Cylc commands in the future.
   - [ ] Add sections for `[job platforms]`
 
 
-- [ ] Implement new `cylc-flow.rc` form. [Cylc-flow #3260](https://github.com/cylc/cylc-flow/issues/3260)
+- [ ] Implement new `cylc-flow.rc` schema. [Cylc-flow #3260](https://github.com/cylc/cylc-flow/issues/3260)
   - [ ] Check old tests for `global.rc` & `suite.rc` to ensure that functionality
     is not lost.
-  - [ ] Devise tests for the new flow.rc
-  - [ ] Create new config specification, perhaps called `config_schema.py` &
-    remove the config schemas folder.
+  - [ ] Devise tests for the new `cylc-flow.rc`
+  - [ ] Create new config schema module, called `cylc.flow.config_schema`
+  - [ ] Remove the `cylc.flow.cfgspec` folder.
 
 
 - [ ] Implement Cluster support functionality. [Cylc-flow #2199](https://github.com/cylc/cylc-flow/issues/2199)
-  - [ ] Modify `task_job_mgr.py` to use the new variables.
+  - [ ] Modify modules to use the new variables:
+    - [ ]  `task_job_mgr.py`
+    - [ ] `task_remote_mgr`    
   - [ ] Randomize login host used
 
 
@@ -74,6 +77,7 @@ all suites will start up with only Cylc commands in the future.
 
 These functionalities are currently provided by Rose, but should really be part
 of Cylc:
+
 * On start up and reload, install suite on suite server cluster (cylc servers).
 * On start up and reload, validate suite.
 * On start up, archive old log directory.
@@ -107,6 +111,7 @@ While we consider the above, we may also want to consider the following:
     subsets of settings:
     ![Venn Diagram showing expected common usage](img/flow_rc_settings_locs.svg)
   * Migrate relevant settings from `rose.conf` and `rose-suite.conf`.
+    __UPDATE THIS__
   * Settings such as `run directory` and `work directory` may need better names
     (users think of "work" as a sub-directory of the run directory, but `run
     directory` and `work directory` are configured separately, and the latter
@@ -149,10 +154,12 @@ configuration logic. Some points to consider:
 * Users will configure tasks to run on clusters instead of hosts/batch systems.
 * If relevant, improve alignment with DRMAA Open Grid Forum API?
 
+
 So, for example, a suite `suite-flow.rc` might look like this:
 (Although a detailed specification should also be created)
 ```ini
 [cylc]
+
     ...
 
 [scheduling]
@@ -244,8 +251,10 @@ sources on installation. Other things to consider:
 ### Suite Validation
 
 The `rose suite-run` command calls `cylc validate --strict` by default.
+
 Automatic suite validation should become the default behaviour for the new
-command, as well as for `cylc reload`. 
+command, as well as for `cylc reload`.
+
 
 ### Rationalise Suite Start Up Commands
 

docs/rose-suite-run-proposal/cylc-flow-rc.md

@@ -0,0 +1,184 @@
+# Proposal example of a new style `cylc-flow.rc` file
+
+## References
+[cylc-admin PR #40](https://github.com/cylc/cylc-admin/pull/40)
+[cylc-admin work plan](../proposal-rose-suite-run.md)
+
+## Purpose
+As part of the work to transfer functionality from `rose suite run` to cylc
+it is proposed that the options available in global, user and suite config
+files are brought into alignment. This file sets out a specification for the
+combined `cylc-flow.rc` file, although for the foreseeable future `suite.rc`
+will also be read in the same way.
+
+## Preamble
+
+To reduce changes required for end users this file format is based on
+`suite.rc`: Where aspects of the file are described as unchanged this implies
+that file contents are exactly the same as they would be in a `suite.rc`.
+Users of the `global.rc` are usually admins or power users and thus changes
+to the format of this file are preferable.
+
+It should be noted that although it will be possible to modify all settings in
+all contexts, that some settings are more likely to be used in global contexts
+and some are more likely to be used in suite configs. It has been proposed
+that it ought to be possible for sysadmins to lock some settings in site
+configurations.
+
+****
+
+## specification for `cylc-flow.rc`
+
+### 1. Top level sections to come from from `suite.rc`
+
+Most sites will leave these to users (Although I could imagine adding a
+copyright message to meta by default, for example, and one user has suggested a
+very simple runtime might be added too, for training and debugging purposes.)
+
+These items are:
+```ini
+[meta]
+[scheduling]
+[vizualization]
+```
+
+* `[cylc]` will be renamed `[general]`
+
+### 2. Top level sections from site configuration
+
+It is likely that most users will continue to have these set by site admins.
+
+#### 2.1 Small changes
+
+* `[authentication]` becomes `[authorization]`
+
+* The `[suite servers]` to be renamed `[suite run platforms]` for consistency
+  with job platforms.
+
+* `[test battery]` will be removed entirely.
+
+* `[task events]` will be moved to `[runtime][[root]][[[events]]]`
+
+#### 2.2 `[suite run platforms]`
+This is the dictionary key formerly known as ``[suite servers]``. Changed only
+for the purpose of keeping the name "platforms" conistent. This is expected to
+be set only by system administrators. It should include the former top-level
+section `[[suite host self-identification]]`.
+
+#### 2.3 `[cylc]` -> `[general]`
+
+`[cylc]` is be renamed `[general]`.
+
+`global.rc[cylc]` at present contains a subset of the items available in
+`suite.rc[cylc]` for this section so it is proposed that the new fill just has
+the larger set. These items are:
+```ini
+[cylc]
+  health check interval = 600
+  task event mail interval = 300
+  [[events]]
+```
+
+### 3 Job Platforms and the deprecation of `[runtime][[TASK]][[[job]]]host`
+
+#### 3.1 `[job platforms]`
+Many of the options in this section will be very similar to `[hosts]`
+It is expected that these will mainly be set at site level, but that
+small numbers of power users may wish to over-ride them.
+
+```ini
+[job platforms]
+  [[example platform]]
+    run directory =
+    work directory =
+    task communication method =
+    submission polling intervals =
+    execution polling intervals =
+    scp command =
+    ssh command =
+    use login shell =
+    login hosts =                         # list of possible login hosts
+    batch system =                        # name of batch system
+    cylc executable =
+    global init-script =
+    copyable environment variables =
+    retrieve job logs =
+    retrieve job logs command =
+    retrieve job logs max size =      [[default directives]]
+
+    retrieve job logs retry delays =
+    task event handler retry delays =
+    tail command template =
+    [[batch systems]]
+      [[__MANY__]]
+        err tailer =
+        out tailer =
+        err viewer =
+        out viewer =
+        job name length maximum =
+        execution time limit polling intervals =
+
+
+    [[default directives]]                # This is probably something to do
+      --some-directive="directive here!"  # sometime after cylc8
+```
+
+#### 3.2 Legacy Hosts behaviour
+`[hosts]` will be deprecated but we need to keep many of its settings in
+`[job platforms]`. For back compatibility host should re-direct to
+`[runtime][[__MANY__]][[[platform]]]`. If the re-mapped `host` is part of a
+cluster defined in `[job platforms]` then that job will use that cluster.
+If a user wishes to over-ride this they can over-ride the
+`[job-platforms][[PLATFORM]]` section.
+
+
+### 4 Top level sections to merge in a more complex way
+
+#### 4.1 `[[[job]]]` & `[[[remote]]]`
+Old `[runtime][[__MANY__]][[[job]]]` & `[[[remote]]]`
+sections to be merged and rationalized, being replaced by a new
+`[runtime][[__MANY__]][[[job]]]` section.
+
+We should select the platform defined by `[job platforms]`
+
+```ini
+[runtime]
+
+[[job]]
+  platform =                            
+```
+
+I think that we will probably want users to set this in the
+[job platforms] section, leaving some of these options here as due-to-be
+deprecated back compat over-rides which will give warnings if set?
+
+```ini
+    batch system =
+    batch submit command template =
+    execution polling intervals =
+    execution retry delays =
+    execution time limit =
+    submission polling intervals =
+    submission retry delays =
+    host =
+    owner =
+    suite definition directory =
+    retrieve job logs =
+    retrieve job logs max size =
+    retrieve job logs retry delays =
+      [[[batch systems]]]
+        err tailer =
+        out tailer =
+        err viewer =
+        out viewer =
+        job name length maximum =
+        execution time limit polling intervals =
+```
+
+### Locking down global settings
+There should be a mechanism by which system administators can lock global
+settings for their sites. This should probably be a `lock=True/False` switches
+within those settings that we wish to make lockable. If unset these will
+default to `False`. If set to `True` a user who tries to over-ride that setting
+will see a warning explaining that the setting has been over-ridden by a site
+admin.