[feature] Space out checks over time, to avoid CPU- and memory-usage spikes

[dnngll]

Version and OS
changedetection.io:0.49.7 and sockpuppetbrowser:0.0.1 on Rocky Linux/Docker

Is your feature request related to a problem? Please describe.
Yes. When restarting the container, all (due) watches are checked right away and at the same time (until FETCH_WORKERS is reached). This leads to CPU- and memory-usage spikes:

Describe the solution you'd like
I would love to have an option like "space out checks over time". Maybe only affecting the watches, that are using the default "Time Between Check"? Or even adjustable per watch? Or randomly, but within the "Time Between Check"?

Describe the use-case and give concrete real-world examples
To avoid the above, right now I am setting FETCH_WORKERS=1. My default "Time Between Check" is at 10 minutes. After (re)starting the service, all due checks are done at the beginning, one after another. This doesn't slow down the system to much, but is still not optimal: There will be a problem, once the sum of the duration of all checks is greater than the "Time Between Check".

A better solution would be an option like "space out checks over time":

Example 1. Assuming:

• The default "Time Between Check" is set to 10 minutes
• 10 watches use the default "Time Between Check"
• FETCH_WORKERS=1
  One check would start at the beginning of minute 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10

Example 2. Assuming:

• The default "Time Between Check" is set to 10 minutes
• 10 watches use the default "Time Between Check"
• FETCH_WORKERS=2
  Two checks would start at the beginning of minute 1, 3, 5, 7, and 9

[dgtlmoon]

can i ask, why is the container being restarted?

[dgtlmoon]

and, can you enable FAST_PUPPETEER_CHROME_FETCHER=true env var and tell me if it helps.. ?

[dnngll]

Hello,

the whole machine is restarted from time to time, and the container with it.

I will try FAST_PUPPETEER_CHROME_FETCHER=true next time I'm on it, and will report back here. Could you point me towards a description of what it does? I didn't find anything.

Besides the above - what do you think of my feature suggestion? When searching for the FAST_PUPPETEER_CHROME_FETCHER-option, I also found #2994 . Seems to me, that @huanghe2015 was thinking of something similar?

Thanks

[gety9]

@dgtlmoon

proposed feature will not only fix issue of "CPU & memory usage" but will also lead to less "blocks by websites" (issue
#2966)

current "tools" we have are not enough, there should be something like "MINIMUM_SECONDS_RECHECK_TIME" (which is for 1 url), but for domain and/or all watches, example:

-, MINIMUM_SECONDS_RECHECK_TIME (all watches, fixes CPU/memory)

-, MINIMUM_SECONDS_RECHECK_TIME_URL (current MINIMUM_SECONDS_RECHECK_TIME)

-, MINIMUM_SECONDS_RECHECK_TIME_DOMAIN (fixes block by websites)

[dgtlmoon]

@gety9 trust me when i write #3178 (comment) , the reason i wrote it is that this method uses 95% less CPU and memory..

can you try it?

[dgtlmoon]

but will also lead to less "blocks by websites" (issue #2966)

without statistical hard evidence i cannot accept this claim

[gety9]

@dgtlmoon

but will also lead to less "blocks by websites" (issue #2966)

without statistical hard evidence i cannot accept this claim

of course i can't conduct full scale experiment with 1000s of users :) but as one user and using common sense i see following:

big portion of users use cd.io to check several urls on same website, example domain1/url-of-interest1, domain1/url-of-interest2, ... domain1/url-of-interest9

domain1's anti robot sees checks to all 9 urls as same user requests, now there is no way to spread them - MINIMUM_SECONDS_RECHECK_TIME is useless (cause it's per url), FETCH_WORKERS also useless (cause it avoid simultaneous checks, but doens't spread checks).
I am getting so many errors simply cause check of url2 was too close to check of url1 (on same domain).

Don't get me wrong, cd.io is amazing software, it's very useful to me and i am grateful to all your work, but it could become so much more reliable with small additional setting - MINIMUM_SECONDS_RECHECK_TIME_DOMAIN or/and MINIMUM_SECONDS_RECHECK_TIME (not current MINIMUM_SECONDS_RECHECK_TIME_URL which is imho not very usefull at all)