Tidy of log directory

[datamel]

Discussed in UK Team meeting:
The log directory has some discrepancies which should be tidied before release.
Renaming:

• log/workflow => log/scheduler
• log/flow-config => log/config
• file-installation-log => remote-install-log

Timestamps should be consistently at the beginning or end of the log (to be decided) and for file installation and config should include <start/restart/reload> as appropriate.

The config files end in .cylc currently which is why they currently have the date stamp at the beginning of the file name.

Note the timestamp also differs between logs, although the log is created before config load so tricky to know what is the preferred time zone.

Note that this change will have implications for cylc-review.

Before:

$ tree log
log
├── db
├── flow-config
│   ├── 20220406T124846Z-start.cylc
│   ├── 20220406T124905Z-restart.cylc
│   ├── 20220406T124913Z-reload.cylc
│   └── flow-processed.cylc
├── install
│   └── 20220406T124844Z-install.log
├── job
│   └── 1
│       ├── a
│       │   ├── 01
│       │   │   ├── job
│       │   │   ├── job-activity.log
│       │   │   ├── job.err
│       │   │   ├── job.out
│       │   │   ├── job.status
│       │   │   └── job.xtrace
│       │   └── NN -> 01
│       ├── b
│       │   ├── 01
│       │   │   ├── job
│       │   │   ├── job-activity.log
│       │   │   ├── job.err
│       │   │   ├── job.out
│       │   │   ├── job.status
│       │   │   └── job.xtrace
│       │   └── NN -> 01
│       └── c
│           ├── 01
│           │   ├── job
│           │   ├── job-activity.log
│           │   ├── job.err
│           │   ├── job.out
│           │   ├── job.status
│           │   └── job.xtrace
│           └── NN -> 01
└── workflow
    ├── file-installation-log -> file-installation-log.20220406T134924+01
    ├── file-installation-log.20220406T134850+01
    ├── file-installation-log.20220406T134924+01
    ├── log -> log.20220406T134905+01
    ├── log.20220406T134846+01
    └── log.20220406T134905+01

Proposed After:

$ tree log
log
├── db
├── config
│   ├── 20220406T124846Z-start.cylc
│   ├── 20220406T124905Z-restart.cylc
│   ├── 20220406T124913Z-reload.cylc
│   └── flow-processed.cylc
├── install
│   └── 20220406T124844Z-install.log
├── job
│   └── 1
│       ├── a
│       │   ├── 01
│       │   │   ├── job
│       │   │   ├── job-activity.log
│       │   │   ├── job.err
│       │   │   ├── job.out
│       │   │   ├── job.status
│       │   │   └── job.xtrace
│       │   └── NN -> 01
│       ├── b
│       │   ├── 01
│       │   │   ├── job
│       │   │   ├── job-activity.log
│       │   │   ├── job.err
│       │   │   ├── job.out
│       │   │   ├── job.status
│       │   │   └── job.xtrace
│       │   └── NN -> 01
│       └── c
│           ├── 01
│           │   ├── job
│           │   ├── job-activity.log
│           │   ├── job.err
│           │   ├── job.out
│           │   ├── job.status
│           │   └── job.xtrace
│           └── NN -> 01
└── scheduler
    ├── log -> log.20220406T134905+01
    ├── log.20220406T134846+01
    ├── log.20220406T134905+01
    ├── remote-install-log -> remote-install-reload-log.20220406T134924+01
    ├── remote-install-start-log.20220406T134850+01
    ├── remote-install-restart-log.20220406T134950+01
    └── remote-install-reload-log.20220406T134924+01

Could potentially be tackled at the same time as: #4314

[wxtim]

diff --git a/tmpb4 b/after
index dcef6ad3..066b6daf 100644
--- a/tmpb4
+++ b/after
@@ -1,7 +1,7 @@
 $ tree log
 log
 ├── db
-├── flow-config
+├── config
 │   ├── 20220406T124846Z-start.cylc
 │   ├── 20220406T124905Z-restart.cylc
 │   ├── 20220406T124913Z-reload.cylc
@@ -37,10 +37,11 @@ log
 │           │   ├── job.status
 │           │   └── job.xtrace
 │           └── NN -> 01
-└── workflow
-    ├── file-installation-log -> file-installation-log.20220406T134924+01
-    ├── file-installation-log.20220406T134850+01
-    ├── file-installation-log.20220406T134924+01
+└── scheduler
     ├── log -> log.20220406T134905+01
     ├── log.20220406T134846+01
-    └── log.20220406T134905+01
\ No newline at end of file
+    ├── log.20220406T134905+01
+    ├── remote-install-log -> remote-install-reload-log.20220406T134924+01
+    ├── remote-install-start-log.20220406T134850+01
+    ├── remote-install-restart-log.20220406T134950+01
+    └── remote-install-reload-log.20220406T134924+01
\ No newline at end of file

[dpmatthews]

We label the other files according to whether it's a start or restart.
To be consistent, can we do the same for the scheduler log?
Also, let's use .log as an extension consistently.
How about:

└── scheduler
    ├── remote-install.log -> 20220406T134924+01-remote-install-reload.log
    ├── scheduler.log -> 20220406T134905+01-scheduler-restart.log
    ├── 20220406T134846+01-scheduler-start.log
    ├── 20220406T134850+01-remote-install-start.log
    ├── 20220406T134905+01-scheduler-restart.log
    ├── 20220406T134950+01-remote-install-restart.log
    └── 20220406T134924+01-remote-install-reload.log

[oliver-sanders]

Further suggestion, move the remote-install logs to their own directory and have one per platform.

log/remote-install/<timestamp>-<start/restart/reload>-<install-target>.log

Having one file per remote-install would help with debugging and enable us to solve the problem of closing the file handles (which currently need to be kept open until shutdown).

[dpmatthews]

So, the proposal is now:

• Rename log/flow-config => log/config (no change to file names)
• Move scheduler logs files to log/scheduler/<timestamp>-<start/restart/reload>.log
  (log/scheduler/log is a symlink to the current log)
• Move remote install logs to their own directory:
  log/remote-install/<timestamp>-<start/restart/reload>-<install-target>.log

@hjoliver please remove the question label if you're happy

[dpmatthews]

Move scheduler logs files to log/scheduler/<timestamp>-<start/restart/reload>.log

Sorry, that should have been log/scheduler/<timestamp>-<start/restart>.log (the scheduler log doesn't change on reload)