Uploading job list is extremely slow

[nclaidiere]

Uploading job list is extremely slow, about 3 mins for a 5kb csv file (I stopped counting when I had to upload a 10k trial list, I did it overnight but I think it was more than 1h).

[dschreij]

I haven't experienced this actually! For me this is quite fast, even for larger files.
@smathot what are your experiences with this?

[nclaidiere]

Ok, so one question is does that depend on the number of participants? is the uploading creating jobs lists and session data, etc before assigning participants?

[smathot]

I have also never experienced this, but of course I've never worked with a database that's as populated as the one of @nclaidiere.

[smathot]

I am testing this now on the server of @nclaidiere and I can confirm that this is really slow, about 30 seconds for a table with 50 rows and 2 columns. (Whereas otherwise the server is fairly snappy.) I suspect that Study.processJobs() is the culprit.

[dschreij]

Did some research into this, after I also experienced bad performance during an automated test run of a study. During that run, each query took about two seconds (according to the OpenSesame console), while for @smathot executing was in millisecond domain (#92 (comment))
My assumption is that this process is so slow because of queries taking a lot longer than usual, and there are a lot of queries in this function.
I found quite some reports of mysql vastly underperforming when running in Docker (on Windows), for instance:

• https://forums.docker.com/t/mysql-slow-performance-in-docker/37179/22
• MySQL extremely slow just with docker-ce installed docker/for-linux#247
• https://jsatt.com/blog/mysql-docker-performance/

There are also some proposed solutions for which varying degrees of success are reported:

• Add :delegated to the mysql data path mapping in docker-compose.
• Map the docker data folder to a physical location on the disk. It currently points to a volume container, which is the preferred method, but other than that, there wouldn't me much difference for persistence between the two options.

I'll see if I have time to play around with these soon and see if they have any effect. If none of these work, we can also try to:

• Replace MySQL with PostgreSQL, which is another popular and widely-used open-source database. I mainly work with it for current projects, and it works really well in Docker and on Windows. Since this project uses an ORM that supports Postgres out of the box. It shouldn't be that much work to replace (but may bring some surprises too).
• Install and run the windows version of mysql locally, making Docker unnecessary. This involves some more steps when setting up and makes the system less portable, but may remedy this and other performance and memory issues.

[smathot]

My assumption is that this process is so slow because of queries taking a lot longer than usual, and there are a lot of queries in this function.

That does sound likely. That being said, uploading is also slow on servers for which the other queries go very fast.