Updates to preset intervals are not applied in some scenarios

[0xgouda]

Describe the bug
In some scenarios (source config update), updates to a preset's metric fetching interval are not applied to running gatherers

To Reproduce
Steps to reproduce the behavior:

1. Run pgwatch with some preset, e.g., basic
2. Update the fetching interval for any metric in basic
3. Update the monitored source and add a standby preset.
4. Observe that the update to the fetching interval will never get applied

Expected behavior
Updates to metric fetching intervals should be applied.

Additional context
This happens because the update that added the standby preset, although not used anywhere, will cause md.Equal() in:

pgwatch/internal/reaper/reaper.go

Lines 398 to 404 in 4bae857

	if md.Equal(newMD.Source) {
	// replace with the existing connection if the source is the same
	newSrcs[i] = md
	continue
	}
	}
	r.monitoredSources = newSrcs

to be false, hence dropping the old *SourceConn of that source from r.monitoredSources and use a new one, but the running gatherer will keep looking for updates from the old *SourceConn which has been dropped and will never get updated.

[0xgouda]

This one is nasty and will probably require some precise architectural changes.

I can think of 2 approaches to resolve it, (both of them will fix #1084 and #1085 as well):

1. On source config update, stop all running gatherers, then start them again with the new config.
   • This will resolve the issue because it will ensure that all gatherers will be controlled by the latest *SourceConn object.
   • It has a disadvantage that even if we only added 1 new metric, it will stop all gatherers for this source and start them again.
2. On source config update, change the old gatherers to point to the newest *SourceConn object and hence fetch the latest configs from it.
   • This is trickier to implement and will require careful consideration for nil pointers and error logging, because the latest object's connection or other fields may not be instantiated yet, and the running gatherers may try to use them.

@pashagolub What do you think?

[pashagolub]

How about stop/start all the sources dependent on the preset?
Or just forbid to change preset if any source is using it right now.

Both are good.

[0xgouda]

How about stop/start all the sources dependent on the preset?

I guess we need a meeting to discuss this one and #1084 and #1085, as well, seeing whether we will add logic to handle a change in each field in *SourceConn separately or start/stop everything on any change?

[pashagolub]

Start/stop affected sources on any edit seems a simple decision