vmalert-tool: implement unittest

[Haleygo]

address #2945
There are two requirements to perform unittest:

1. mock an isolated VictoriaMetrics instance, can insert like vminsert, storing like vmstorage, quering like vmselect. Split or mock those functions can be very expensive, so I think we still need to reuse them by importing their packages.
2. evaluate rules exactly like vmalert.

vmalert can't implement this like #4734 said.
vmctl used to be considered, but there are unexposed parts in vmalert, so it's not be implemented at that time. Now vmctl is considered only for migration stuff.

update:
After discussion, we are adding a new tool named vmalert-tool, it's only used for unittest now.

This pull request does two things:

1. split package rule under vmalert, expose needed objects. After this, vmalert should become more functionality, other components can use vmalert library to do rule evaluation.
   Implement rule evaluation for vmsingle in the future maybe, could simplify the combination of vmsingle+vmalert, mentioned in Add remote_write support to victoria metrics (single) #1645 (comment)

2. create new vmalert-tool to bring unittest feature in vmalert: init unit test #4596.

[hagen1778]

I think we're going in the right direction. But we should try to split responsibility between packages in more clear way. Let's agree on the following:

1. main.go is responsible for initial validation of the given params and for initing components like: datasource, remote read, remote write, notifier.
2. main.go is responsible for scheduling rules execution via private component manager
3. Manager supposed to operate on the Groups level, where Group is a part of rule package
4. Rules package supposed to expose Group, Alerting and Recording structs. The Rule interface can be private, as main.go should use Group to exec rules.
5. web component is a part of main package and should operating based on the public structs from the rule package. Methods like RuleToAPI could be private and belong to main package only.
6. Unit tests should operate on rules execution via Group, the same way as in p3. It could be needed to add an extra method to a Group struct which would evaluate all the rules on the call. Additionally to Start method, which only schedules the evaluation.

This separation supposed to isolate responsibility of components and keep private fields private. It should also provide the same way for rules execution for main.go and for unit tests.

One extra comment is that for unittests it might be better to create a new tool instead of integrating into vmctl. It is right that vmctl was initially meant to become a cli tool for VM. But historically, vmctl is responsible for migration only. Creating a vmalert-tool could be easier to grasp and argue about for users. Probably, we'll move replay mode to vmalert-tool after that.

Let me know wdyt.

[Haleygo]

But historically, vmctl is responsible for migration only.

@hagen1778
Yeah, it was a bit of strange when added this subcmd to vmctl which seems all about data processing.
But having a second tool might be little inconvenient for users, maybe we can embed it into vmalert in the future and use it in web like #4691.
And I don't have better idea than vmalert-tool now, will create it)

[codecov]

Codecov Report

Attention: 546 lines in your changes are missing coverage. Please review.

Comparison is base (4f4bc51) 60.37% compared to head (6e0db01) 60.21%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4789      +/-   ##
==========================================
- Coverage   60.37%   60.21%   -0.17%     
==========================================
  Files         389      397       +8     
  Lines       73142    74019     +877     
==========================================
+ Hits        44163    44572     +409     
- Misses      26556    26957     +401     
- Partials     2423     2490      +67     

Files	Coverage Δ
app/vmalert/replay.go	42.50% <100.00%> (-20.34%)	⬇️
app/vmalert/rule/utils.go	96.59% <ø> (ø)
app/vmalert/main.go	39.08% <0.00%> (ø)
app/vmalert/manager.go	88.07% <94.73%> (+0.64%)	⬆️
app/vmalert/web.go	66.66% <85.71%> (ø)
app/vmalert/web.qtpl	72.77% <76.19%> (+1.45%)	⬆️
lib/promutils/duration.go	57.57% <0.00%> (-10.29%)	⬇️
app/vmalert/rule/recording.go	77.37% <71.42%> (ø)
app/vmalert-tool/unittest/recording.go	71.42% <71.42%> (ø)
app/vmalert/remotewrite/debug_client.go	66.07% <66.07%> (ø)
... and 11 more

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hagen1778]

Almost ready now! Thanks for hard work!

[hagen1778]

LGTM!

[valyala]

FYI, this pull request has been included in v1.95.0 release.