Metadata page can't save edits if it tried to load over 2000 images

[StephenChan]

Related threads:

• https://groups.google.com/g/coralnet-users/c/Zne8Hy7PIWw (recently)
• https://groups.google.com/g/coralnet-users/c/5Y2fM5A1BfU (September 2022)

The root cause is that Django is silently truncating the metadata formset past the 2000th form, resulting in an invalid formset. Relevant docs: https://docs.djangoproject.com/en/2.2/topics/forms/formsets/#validating-the-number-of-forms-in-a-formset

Regardless of validate_max, if the number of forms in a data set exceeds max_num by more than 1000, then the form will fail to validate as if validate_max were set, and additionally only the first 1000 forms above max_num will be validated. The remainder will be truncated entirely. This is to protect against memory exhaustion attacks using forged POST requests.

But Django (at least 2.2) allows the formset to be loaded anyway, and then it so happens that our edit_metadata_ajax view only reports errors from individual forms in the formset, without anticipating top-level formset errors. So the view ends up not reporting any error details, and thus no further error details are displayed on the page besides "There were error(s); couldn't save".

The latest Django still has the form limit, but it might be handled differently such that it fails loudly instead of silently: https://docs.djangoproject.com/en/dev/topics/forms/formsets/#validating-the-number-of-forms-in-a-formset

Possible fixes:

1. Raise max_num. However, some sources have upwards of 30,000 images, so we may be moving the bar to a point where memory exhaustion (or at least degraded server performance) is actually an issue.
2. When there are 2000+ results, display a message asking to narrow down the results with filters, rather than displaying the grid. Similar to issue Metadata edit performance: Ask for a search before showing any results #130
3. Pagination: issue Metadata edit: pagination #131
4. Organize the form fields more efficiently to only submit what was edited, which probably involves not using a formset (at least not in the same way): issue Metadata edit performance: Only send edited fields, not all fields #129

Probably 3 is preferred, 2 if it's simpler in the short-term; while 4 would be considered an extra rather than a standalone solution (because page performance with 30,000 rows displayed is a problem regardless).