Out Of Memory for for attribute filters #53
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the Out Of Memory error by pushing unique value selection to SQL level instead of iteration in PHP.
The original issue is that the bigger the database was, the more objects were selected from DB, resulting, in our case, in about 300 000 ProductAttributeValue array of objects created in memory and obviously it was not only slow but hit the memory limit fast.
Also excluded attributes where always rendered too, which didn't help cause we had 400k attributes named "Description" that were unique for each product, resulting in absurd memory requirements and hydration taking ages despite the attribute being in the excluded array.
The fix is not perfect - that GROUP BY is going to be slow when there are a lot of values for a given attribute because there are no indexes. And some attributes can be on each product - we for now dealt with it by putting such attributes into the excluded array. Also, boolean/checkbox attributes right now make no sense for filtering and need further development.
Also, doing a GROUP BY on a json, array or text field is, well, stupid, ludicrous, plain and simple - an issue that needs to be handled.