Loki Canary: One more round of improvements to query for missing websocket entries up to max-wait

[slim-bean]

With wait set to only 1 min when we miss a websocket entry we check for it after 1 minute but only check one time.

If promtail was delayed or another hiccup existed somewhere the entry could still make it but later.

This PR introduces max-wait where the canary will continue to check for missing entries up to this time before officially marking them lost.

There was also a repeating pattern of removing elements from a slice which I extracted into a dedicated function for better reuse

[codecov-commenter]

Codecov Report

Merging #2369 into master will increase coverage by 0.10%.
The diff coverage is 95.58%.

@@            Coverage Diff             @@
##           master    #2369      +/-   ##
==========================================
+ Coverage   61.46%   61.56%   +0.10%     
==========================================
  Files         160      160              
  Lines       13534    13544      +10     
==========================================
+ Hits         8318     8339      +21     
+ Misses       4589     4583       -6     
+ Partials      627      622       -5     

Impacted Files	Coverage Δ
pkg/canary/comparator/comparator.go	80.24% <95.58%> (+2.51%)	⬆️
pkg/logql/evaluator.go	92.88% <0.00%> (+0.40%)	⬆️
pkg/promtail/targets/file/filetarget.go	70.48% <0.00%> (+1.80%)	⬆️
pkg/promtail/targets/file/tailer.go	76.13% <0.00%> (+2.27%)	⬆️

[cyriltovena]

LGTM