Provide some suggestions for larger orchestrator environments

[sjmudd]

Add some suggestions on settings for orchestrator in a large environment.

[MOON-CLJ]

In addition to #1203, we have one more modification。

we comment this line out.

orchestrator/go/logic/topology_recovery.go

Line 916 in 7bf32d1

go orcraft.PublishCommand("async-snapshot", "")

because we do frequent integration tests in our product environment. and because we patch AuditPurgeDays to 31(We've already revert it to 7)，we have a lots of data in our backend db。

select count(*) from topology_recovery;
+----------+
| count(*) |
+----------+
|    62421 |
+----------+
1 row in set (0.02 sec)

select count(*) from topology_failure_detection;
+----------+
| count(*) |
+----------+
|    95159 |
+----------+
1 row in set (0.02 sec)

Snapshot reads the full amount of data in the backend db as well as the raft state each time, taking up a lot of memory and golang gc resources. Then takes up a lot of goroutine scheduling resources, then raft heartbeat timeouts and frequent elections.

So we only do snapshots periodically now.

orchestrator/go/raft/raft.go

Line 44 in 7bf32d1

snapshotInterval = 30 * time.Minute

maybe we need a config for this behavior

orchestrator/go/logic/topology_recovery.go

Line 916 in 7bf32d1

go orcraft.PublishCommand("async-snapshot", "")

cc @shlomi-noach

[sjmudd]

Hm. The snapshot thing I have configured without touching code.

"SnapshotTopologiesIntervalHours": 0,

Judging from the configuration description I think this is sufficient. You should not need to change the code to make orchestrator work the way you need it to.

If extra configuration is required I'm sure that @shlomi-noach would prefer you providing a PR with a new configuration setting to make the setting configurable as needed. The process for doing that is quite straight forward and this is much better long term. Having had experience of maintaining a patched fork of orchestrator I would really recommend that you push changes back upstream to Shlomi explaining why they are needed and if they make sense I'm sure he'll accept them.

[shlomi-noach]

CI fails because there's no incoming link to this new documentation page. Please add a link in configuration.md

[sjmudd]

@shlomi-noach good morning. I hadn't realised this was necessary and it's good to require that, so should be fixed I think with the extra commit I've pushed.

[MOON-CLJ]

orchestrator/go/logic/topology_recovery.go

Line 916 in 7bf32d1

go orcraft.PublishCommand("async-snapshot", "")

Hm. The snapshot thing I have configured without touching code. "SnapshotTopologiesIntervalHours": 0,

@sjmudd SnapshotTopologiesIntervalHours is no the same thing with snapshotInterval，snapshotInterval was about raft snapshot.

so what we suggest is to add a configuration item here below

orchestrator/go/logic/topology_recovery.go

Line 916 in 7bf32d1

go orcraft.PublishCommand("async-snapshot", "")