[Alerting] When custom notify interval is smaller than check interval, notifications are sent on check interval, not notify interval

[lucasfcosta]

Kibana version: 8.1.0 (SHA f159f02b46d86c2e58c0ddb2b99173e94fd4a7a5)

Elasticsearch version: 8.1.0-SNAPSHOT

Server OS version: MacOS Big Sur 11.6

Browser version: Version 96.0.4664.45 (Official Build) (x86_64)

Browser OS version: MacOS Big Sur 11.6

Original install method (e.g. download page, yum, from source, etc.): Running Kibana from source (currently main), against an Elastic Stack brought up by elastic-package.

Describe the bug: When users create an alert whose check interval is bigger than the custom notify interval, the user will not be notified in the custom notify interval specified. Instead, they will be notified when the check interval is triggered.

For example: someone creates an alert which is checked every 1 minute, but its notify interval is every 30 seconds. In that case, they'll receive alerts every 1 minute, not every 30 seconds.

Steps to reproduce:

1. Go to "Add Integrations", and create a synthetic monitor which always fails, like the one below:

   step('load homepage', async () => {
       await page.goto('https://www.elastic.co');
             });
       step('hover over products menu', async () => {
       const cookieBanner = await page.$('#iubenda-cs-banner');
       await page.waitForTimeout(60*1000);
       await page.hover('css=[44data-nav-item=products]');
   });

   I recommend that you configure this monitor to run every 60 seconds.
2. Go to Hookbin and create a new endpoint to which you'll send requests when your alert rule is triggered.
3. In Kibana, access your monitor's details in the Uptime app.
4. For that monitor, create an alert rule whose check interval is greater than the notify interval (i.e. 1 minute, and 30 seconds, respectively, as in the screenshot above). Make sure to also enable a status check that will cause the rule to be activated soon, so you don't have to wait too much to see the problem.
5. Add a Webhook connector to the alert rule which is triggered both when the monitor fails and recovers, and for that connector use the URL you obtained from Hookbin in step 2.
6. On Hookbin, you will see that the requests are sent every minute (the check interval), not every 30 seconds (the notify interval).

Expected behavior: The notifications should be sent according to the custom notify interval, even when it's smaller than the check interval or, if this case is supposed to not be supported, it shouldn't be possible to set a notify interval smaller than the check interval.

Screenshots (if relevant): See above.

Errors in browser console (if relevant): None.

Provide logs and/or server output (if relevant): None.

Any additional context: I've found this issue when testing #118999, and would recommend further testers/implementers to have that fix in their branch when working on this.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

The notifications should be sent according to the custom notify interval, even when it's smaller than the check interval or, if this case is supposed to not be supported, it shouldn't be possible to set a notify interval smaller than the check interval.

@arisonl sounds to me like this is the latter - we shouldn't allow a customer to set a notify smaller than the interval.
Does that sound right to you?