Bugfix/FOUR-6473: As a process designer, I want to be able to index fields in JSON data

[agustinbusso]

Issue & Reproduction Steps

Note: Mysql 8 required

Please read description at: https://processmaker.atlassian.net/browse/FOUR-6473

Related Tickets & Packages

• FOUR-6473
• Core PR
• package-savedsearch PR
• pmql PR

Testing Notes

• Here is a SQL dump of a collection with 418k rows. You'll need to create a collection (with ID 1) and run the sql to replace the table.
• The first 18k rows have a data field with about 190Kb and the rest about 4Kb
• The json has these fields: bigText (string), numberValue (integer between 1 and 1000), stringNumberValue (same number but as a string), and randomString (10 characters)

Testing plan

• Before using this branch, create the collection and import the attached table. Create a few saved searches on the collection records with data.stringNumberValue = "950", another saved search with data.numberValue = "123" etc.
• Verify the saved searches run slow
• Switch to this branch and restart horizon
• Run php artisan upgrade and make sure the indexes for referenced json columns were created.
• Add another saved search, this time with something like data.randomString = "X6YXefgktF". Make sure that index is created (new index creation is a queued job)
• Verify the saved searches run quickly

Example query that should return 429 quickly after the indexes have been added

SELECT count(*) from collection_1 where data->>"$.stringNumberValue" = "950";

Limitations

• The index only works on string values, but you can search for an exact numberValue by putting your pmql value in double quotes like data.numberValue = "123" and it will use the index
• >, < will not use indexes
• count(*) is still slow to get all rows and is a limitation of mysql
• like will not use the index and will be slow