Webhook processing time limit

[escopecz]

Please be sure you are submitting this against the staging branch.

Q	A
Bug fix?	Y
New feature?	Y
Automated tests included?	N
Related user documentation PR URL	/
Related developer documentation PR URL	/
Issues addressed (#s or URLs)	/
BC breaks?	N
Deprecations?	N

Description:

There are issues with webhook processing when there are too many events to send. The processing method is called recursively until the queue is empty. This was causing timeout issues in our system as the webhook processing never stopped due to new records in the webhook queue. This PR is adding 10 minutes time limit (configurable) to stop processing after the limit is hit.

This PR also improves the query getting rows from the webhook queue table and adds a new index that speeds up the query even more.

Steps to reproduce the bug:

1. Set 'webhook_limit' => 1, in local.php which means send 1 event in 1 request (so it will take longer, that's what we need for our test).
2. Create a webhook which will send the events to some website. The slower the website respond the better.
3. Create around 50 webhook events. Sequel Pro has option to duplicate row with command + D which will speed it up.
4. Run app/console m:w:p -i ID. Replace the ID with the ID of your webhook.
5. It will process the whole queue at once. Check in the webhook_logs table that processing the first and the last log was longer than 1 second.

Steps to test this PR:

1. Set 'webhook_time_limit' => 1, in local.php which means timeout the processing after 1 second.
2. Repeat the test again. Now only part of the queue will be processed. Only those events that can be processed in 1 second. Execute the command again and another amount of events will be processed.

[codecov]

Codecov Report

Merging #9573 (ecfcbc0) into features (77f0da0) will increase coverage by 0.16%.
The diff coverage is 58.33%.

@@              Coverage Diff               @@
##             features    #9573      +/-   ##
==============================================
+ Coverage       38.33%   38.50%   +0.16%     
- Complexity      33939    33940       +1     
==============================================
  Files            1982     1982              
  Lines          104948   104952       +4     
==============================================
+ Hits            40236    40409     +173     
+ Misses          64712    64543     -169     

Impacted Files	Coverage Δ	Complexity Δ
app/bundles/WebhookBundle/Model/WebhookModel.php	71.56% <42.85%> (+33.68%)	71.00 <1.00> (+2.00)
...hookBundle/Command/ProcessWebhookQueuesCommand.php	68.75% <50.00%> (+39.71%)	8.00 <0.00> (ø)
...es/WebhookBundle/Entity/WebhookQueueRepository.php	88.88% <100.00%> (+88.88%)	4.00 <0.00> (-1.00)	⬆️
app/bundles/CoreBundle/Entity/CommonRepository.php	62.71% <0.00%> (+0.13%)	320.00% <0.00%> (ø%)
app/bundles/WebhookBundle/Entity/WebhookQueue.php	94.59% <0.00%> (+8.10%)	10.00% <0.00%> (ø%)
app/bundles/WebhookBundle/Entity/Event.php	73.33% <0.00%> (+11.11%)	10.00% <0.00%> (ø%)
app/bundles/WebhookBundle/Entity/Webhook.php	51.36% <0.00%> (+13.66%)	53.00% <0.00%> (ø%)
app/bundles/WebhookBundle/Entity/LogRepository.php	17.02% <0.00%> (+17.02%)	8.00% <0.00%> (ø%)
... and 4 more

[kuzmany]

Works perfectly 👍
Approve it

[RCheesley]

I have added an issue to add this to the Dev Docs in the webhooks section: mautic/developer-documentation#187