High volume breaks Rock PDF generation due to excess processes

[TheCoreyHall]

Description

We found this issue with one of our clients on 16.2. After a lot of testing and research we were able to identify the cause, but it is difficult to reproduce due to a high volume of people, workflows, and financial data needed. Let me explain.

Our client sent an email to a list of ~1700 people with a link to a workflow. When they click the link, it takes them to a workflow that runs the "Create Contribution Statement" action and emails them the created statement. When hundreds of people started clicking the link, the workflow began failing and reported errors with the browser. After logging into the server hosting Rock, we found this in the task manager:

That is over 500 Chromium processes. These processes built up and eventually all PDF generation failed. After investigating, here is what we found:

When an action such a Create Contribution Statement or Electronic Signature are run, a 5-6 of these Chromium processes are created and then quickly finish as the PDF is generated. However, if too many people generate PDFs at the same time, the processes do not finish, and instead build up as more people use the workflow. Eventually, when the processes soon accumulate over 500, all PDF generation fails until we could restart the server and remove them. I suspect this amount differs based on server specs.

This is happening on 16.2 but its possible its happening on earlier versions. This is the first time we have tried to generate PDFs in bulk through workflows.

Actual Behavior

After too many PDFs are generated in too quick of a time frame, all PDF generation breaks until the server is restart.

Expected Behavior

Rock should handle high volumes of PDF requests.

Steps to Reproduce

I was unable to reproduce this on demo because it requires high volumes of givers to recreate, but we were able to reproduce with a fresh install of Rock that we imported some data into. Here are the steps:

1. Upload this workflow (It will generate a statement per person but it wont save to profile and will only exist in the workflow):
   Giving Statement Request - Test_202402011502.json
2. Go to Giving Analytics and set the filters to generate a list of ~2000 Givers.
3. Select the Launch Workflow button on the list and select the workflow uploaded
   
4. Launch the workflows and then login to the server to view the Chromium processes not disappearing.

Issue Confirmation

• Perform a search on the Github Issues to see if your bug or enhancement is already reported.
• Try to reproduce the problem on a fresh install or on the demo site.

Rock Version

16.2

Client Culture Setting

en-US

[TheCoreyHall]

We created a workaround with a custom script that runs every minute and kills chrome processes that are older than 90 seconds, but would love something more official.

[unlearnd]

Hi @TheCoreyHall we've fixed the issue where the processes were being orphaned and we hope this will be enough to prevent the server from being overwhelmed, but if you continue to experience high resource utlization you might also consider using a more robust PDF Generation service. Please see the PDF Generation section in the Hero Guide for more information on this topic.

[TheCoreyHall]

@unlearnd We appreciate it.