Assess 3.4.0 performance with large databases

[asmecher]

OJS 3.4.0 introduces a lot of SQL changes that should improve performance but need to be tested.

In particular, assess the performance with a large database and the following operations/tasks:

• Submission lists/filters
• Reviewer selection
• User management
• OAI Identify ("first record date")
• Merge Users
• 3.3.x to 3.4.x upgrade
• Large tables of contents (@diegoabadan may have test data)
  • pkp/ojs@75d8dfd for stable-3_3_0 should improve performance somewhat (to be included in stable-3_3_0-15)

[asmecher]

(@alexxxmendonca, I think this captures the pain points identified by the SciELO/OJS working group, but please let me know if there were others.)

[diegoabadan]

@asmecher , is better performance expected when there are many articles published in the table of contents?

Referring to #6511

Is it desirable to run these tests now, from the git code?

[asmecher]

@diegoabadan, it's too early to start tests, but I've added that scenario to the list. I may ping you later for some test data; we don't have anything with that large a number of articles in a single TOC.

[diegoabadan]

Thanks Alec.

We have more than one case to test. :)

[mpbraendle]

Hi Alec - if that helps: we have a journal with about 6000 articles, but issues usually have 10-40 articles (so not a very large TOC).

[asmecher]

@mpbraendle, that's a fairly typical size and not likely to turn up bottlenecks in our environment. I think the delays you're experiencing will need to be resolved separately from this issue -- as posted here: https://forum.pkp.sfu.ca/t/submissions-again-loading-slowly/72688/8

[ajnyga]

We are going to 3.3 soon and will try to do testing with 3.4 right after that. This would mean probably october(ish).

[asmecher]

We'll need to audit warnings encountered with a large upgrade database. The categories of warnings from the SciELO data set are...

• Nulling non-existent genre_id ### in submission_files. [AS]
• Nulling non-existent source_submission_file_id ### in submission_files. [AS]
• Nulling non-existent uploader_user_id ### in submission_files. [AS]
• Removing orphaned controlled_vocab_entry_settings for missing controlled_vocab_entry_id ### [AS]
• Removing orphaned edit_decisions entries for missing submission_id ### [AS]
• Removing orphaned edit_decisions entry for missing editor_id ### [AS]
• Removing orphaned email_templates_settings for missing email_id ### [AS]
• Removing orphaned filters for missing filter_group ### [AS]
• Removing orphaned library_files for missing context_id ### [AS]
• Removing orphaned library_files for missing submission_id ### [AS]
• Removing orphaned note entry ID ### with nonexistent query ### [AS]
• Removing orphaned query_participants for missing user ID ### [AS]
• Removing orphaned review_assignments entry ID ### with submission_id ### [AS]
• Removing orphaned review_files for missing review_id ### [AS]
• Removing orphaned settings for missing library_file ### [AS]
• Removing orphaned settings for missing notification ID ### [AS]
• Removing orphaned settings for missing submission file ID ### [AS]
• Removing orphaned submission_comments entry for missing author_id ### [AS]
• Removing orphaned submission_comments entry for missing submission_id ### [AS]
• Removing orphaned submission_files entry ### with non-existent submission. [AS]
• Removing orphaned submission ID ### with nonexistent context ID ###. [AS]
• Removing orphaned user_interests for missing user_id ### [AS]
• Reset ### announcements with orphaned (non-null) announcement types to no announcement type. [AS]

[asmecher]

These seem fine at a glance with a large data set. Closing for internal testing.