[feature]: Persist imported Mission Control Data across restarts.

[ziggie1984]

In the course of the SoB 2024 project where we implement an External Distributed Mission Control Coordinator it makes sense to persist the MC data we import XImportMissionControl into LND across restarts (see also #8840 for more background).

The design of the feature should include:

• an explicit flag to allow or disallow the persisting of the MC data when importing
• sanity checks for the imported data, also taking the import size into account so that we do not overflow the memory when importing a huge number of MC data
• saving the data to disc should not degrade the performance of the LND node because it's not a critical part of the system and should be treated with low priority for the scheduler.

cc @mohamedawnallah

[Roasbeef]

While we're at it, perhaps it makes sense to also introduce the concept of mission control namespaces? This way you can import some outside weights to an independent namespace, then specify later on if you want to use that or the "default" namespace.

This also has some overlap with this issue to support profiles for the configs as well: #7812. A combo of both would facilitate A/B testing of the various knobs and path finding algos we have.

[feelancer21]

In the context, one should also check whether the parameters from SetMissionControl can be persisted. Currently, the lnd.conf must also be changed if you want to have the same parameters after the next restart.
In my opinion, parameters that are changed via the API should be persisted. To avoid inconsistencies with the command line flags, these should therefore be removed and the defaults for fresh lnd instances should be hard-coded.

[ziggie1984]

In the context, one should also check whether the parameters from SetMissionControl can be persisted. Currently, the lnd.conf must also be changed if you want to have the same parameters after the next restart.
In my opinion, parameters that are changed via the API should be persisted. To avoid inconsistencies with the command line flags, these should therefore be removed and the defaults for fresh lnd instances should be hard-coded.

Good point, this is basically the idea of namespaces described by roasbeef in #7812, keeping the default (global) config as in the lnd.conf but being able to create router profiles (tagged via namespaces probably and persited to disc) and then also being able to attach those profiles to the send payment requests.

[ziggie1984]

While we're at it, perhaps it makes sense to also introduce the concept of mission control namespaces? This way you can import some outside weights to an independent namespace, then specify later on if you want to use that or the "default" namespace.

Just to clarify my understanding: So we would add a new field to the MC entry called namespace which binds the entry to the specific router profile for example. So every entry can only have 1 namespace attached to it ? If so this would increase the requirement of the MC data, LND would have to keep in memory, or maybe we would have to redesign how we handle the memory MC data so that we only have the loaded namespace related data loaded into memory otherwise the memory consumption might be harmful for the overall stability of the daemon.

Regarding this topic, @mohamedawnallah was doing some stress testing and loading up to 500_000 MC paris into LND which currently LND would just accept and that would lead to a OOM crash. I think we need to make LND more robust in the course of this PR, meaning that we only load for example most recent data into memory and fetch the other data when needed from the db, so we don't have the whole data set loaded in memory ?
I wonder if we should prioritze this architectural issue, because the current Routing Graph seem to not exceed 110000 MC pairs which LND can still handle (depending on the system obviously).