FOUR-6601: Refactor settings

[runyan-co]

Environment Setup

• Set core to this PR's branch: refactor/FOUR-6601
• Set connector-send-email to refactor/FOUR-6601
• Set package-actions-by-email to refactor/FOUR-6601
• Set package-collections refactor/FOUR-6601
• Set package-auth to refactor/FOUR-6601
• Set package-ellucian-ethos to refactor/FOUR-6601

Issue & Reproduction Steps

We used to utilize a tagged Redis cache to store settings to reduce the number of queries to the database every time the app is loaded. Over time, we've noticed the cache actually creates too many network calls to Redis (it makes a network call everytime a cached setting is called) and it uses a lot more memory by each PHP process. We also had a settings() function helper which complicates things as all settings were also in the global app config (accessed using the config() function helper).

This PR refactors the settings so they can be easily accessed through the global app configuration. This will reduce the duplicative nature of the settings() helper and simplify the app config caching process.

Solution

The SettingsServiceProvider loads all settings from the database and binds the key and config values into the app’s global configuration. If the app configuration is cached when a setting is saved or updated, the config will automatically be cleared and re-cached. Horizon will also be restarted when the configuration is re-cached to ensure it picks up the updated config values.

Refactoring settings usage in packages

Multiple packages implement and use settings, however this refactor deprecates the settings(), flush_settings(), and the cache_settings() function helpers. There are also other changes which need to be made to specific place in the below packages to ensure the settings are correctly implemented.

How to Test

Extensive testing will be required to ensure all PRs cover all use cases of settings, ensuring they are updated (even when the configuration is cached).

Related Tickets & Packages

• PR for connector-send-email
• PR for package-collections
• PR for package-auth
• PR for package-ellucian-ethos
• Ticket FOUR-6601

Code Review Checklist

• I have pulled this code locally and tested it on my instance, along with any associated packages.
• This code adheres to ProcessMaker Coding Guidelines.
• This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
• This solution fixes the bug reported in the original ticket.
• This solution does not alter the expected output of a component in a way that would break existing Processes.
• This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
• This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
• This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
• This ticket conforms to the PRD associated with this part of ProcessMaker.

[agustinbusso]

For saved search I just did a test, and it seems the settings still cached, please watch the following video @runyan-co

Screen.Recording.2022-08-19.at.10.18.42.mov

To make the changes take effect you need to restart horizon manually:

Screen.Recording.2022-08-19.at.10.22.26.mov

[agustinbusso]

Extended properties configuration working correctly:

Screen.Recording.2022-08-19.at.10.37.29.mov

SSO configuration working correctly:

Screen.Recording.2022-08-19.at.10.55.04.mov

[agustinbusso]

Horizon is terminating constantly after trying to run a process with data connector:

Screen.Recording.2022-08-19.at.10.57.11.mov

Screen.Recording.2022-08-19.at.10.59.41.mov

[agustinbusso]

ABE seems to be working correctly:

Screen.Recording.2022-08-19.at.12.54.28.mov

[agustinbusso]

SCIM settings working correctly:

Screen.Recording.2022-08-19.at.16.32.57.mov

[agustinbusso]

SAML settings working correctly:

Screen.Recording.2022-08-19.at.16.58.15.mov

[runyan-co]

Details

Hey @agustinbusso, I reviewed your screen recording and it looks like it's working correctly. Since you're running horizon in the foreground (using php artisan horizon), it's sending a shutdown signal to that particular PHP process. In a production environment (or my local dev environment, cause I'm lazy), supervisor is usually used to continually spawn new PHP processes (running horizon) as needed.

If you're interested, I included the supervisor config file I use in my local environment. You'd need to change out the path to artisan in the file and then add the file to the supervisor config directory:/usr/local/etc/supervisor.d/horizon.conf (or /opt/homebrew/etc/supervisor.d/horizon.conf for M1 MacBooks). Then restart supervisor (supervisorctl reread, supervisorctl update, supervisorctl restart all) and it should run horizon in the background for you.

[program:horizon]
process_name=%(program_name)s_%(process_num)02d
command=/opt/homebrew/bin/php /Users/alex/code/processmaker/artisan horizon
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=alex
numprocs=1
redirect_stderr=true
stdout_logfile=/Users/alex/logs/%(program_name)s_%(process_num)02d.log
stopwaitsecs=3600

[agustinbusso]

Saved search send email report working correctly

Screen.Recording.2022-08-24.at.10.52.27.mov

[agustinbusso]

Send email from connector working correctly:

Screen.Recording.2022-08-24.at.11.17.06.mov

[agustinbusso]

Docusign settings changing correctly:

Screen.Recording.2022-08-24.at.11.34.45.mov

[agustinbusso]

Test for FOUR-6599:
Send Report of save search is sent without having configurations in the email settings

Working well, settings are not cached:

Screen.Recording.2022-08-24.at.15.29.33.mov

Screen.Recording.2022-08-24.at.15.27.55.mov

[danloa]

Check list:
Configure email - OK
Test email connector - OK
Configure ABE - OK
Test ABE - OK
Collections - OK
Configure Ethos - OK
Ethos - OK