Slow multirun performance after database setup with SimPathStart

[Mariia-Var]

Model runs extremely slowly when database is set up using SimPathsStart instead of -DBSetup

Details:
The model runs quickly when using the following terminal commands:

java -jar multirun.jar -DBSetup
java -jar multirun.jar -n 3 -s 2011 -e 2023

However, if instead of using the -DBSetup command, the database is initialised by running the SimPathStart class, the multirun process becomes significantly slower:
-This slowdown occurs regardless of whether the program is executed via the terminal (e.g., java -jar multirun.jar) or through the IDE’s built-in run tool.
-More specifically, when using SimPathStart for database setup, the initialization phase of the second and subsequent runs takes 2–4 hours to complete.

[andrewbaxter439]

Thanks Mariia, re-running the slow model versions just now to see sticking points. If you get a chance, it could be handy do add here what lines the runs get stuck on. We discussed the points of slowness before, but I think I've forgotten them!

[Mariia-Var]

Hi Andy, the model usually gets stuck at the stage "Creating population structures" (i.e. the last message shown in the terminal).

@dav-sonn (and later me) will run the model with the profiler to identify the exact lines of code that work slowly.

[andrewbaxter439]

Hi @Mariia-Var - want to try pulling the new branch at https://github.com/centreformicrosimulation/SimPaths/tree/261-slow-multirun-performance-after-database-setup-with-simpathstart and seeing if that makes the second run onwards run quickly?

[dav-sonn]

I'll do it.

[Mariia-Var]

Hi @Mariia-Var - want to try pulling the new branch at https://github.com/centreformicrosimulation/SimPaths/tree/261-slow-multirun-performance-after-database-setup-with-simpathstart and seeing if that makes the second run onwards run quickly?

Hi @andrewbaxter439, the new modification leads to an error when running SimPathsMultiRun.java after initialisation of the database with SimPathsStart.java:
first, the error is thrown at the beginning of the 2nd run;
Then the same error emerges in the first run:

Loading model parameters
Time to load parameters: 2.889 seconds.
Populating donor database indices
Completed accessing donor data from the database
Time to load tax database references: 56.184 seconds.
Country: UK. Running simulation from: 2019 to 2021
Creating population structures
Run failed: Problem sourcing data for starting population
jakarta.persistence.NonUniqueResultException: query did not return a unique result: 2
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:128)
	at org.hibernate.query.spi.AbstractSelectionQuery.getSingleResult(AbstractSelectionQuery.java:466)
	at org.hibernate.query.sqm.internal.QuerySqmImpl.getSingleResult(QuerySqmImpl.java:1017)
	at simpaths.model.SimPathsModel.getProcessed(SimPathsModel.java:3338)
	at simpaths.model.SimPathsModel.getProcessed(SimPathsModel.java:3311)
	at simpaths.model.SimPathsModel.createInitialPopulationDataStructures(SimPathsModel.java:2443)
	at simpaths.model.SimPathsModel.buildObjects(SimPathsModel.java:409)
	at microsim.engine.SimulationEngine.buildModels(SimulationEngine.java:421)
	at microsim.engine.SimulationEngine.startSimulation(SimulationEngine.java:349)
	at microsim.engine.MultiRun.go(MultiRun.java:147)
	at microsim.engine.MultiRun.run(MultiRun.java:102)

Process finished with exit code 1

[andrewbaxter439]

Apologies, this is still not producing errors in my code in any run! Will try again next week.

As a test, if you open the SQL console window, can you try SELECT * FROM Processed and see what it returns? For me it's this one-row table with the details of the last run. It seems like it's somehow returning 2 rows for you?

[dav-sonn]

I get the same error as Mariia’s, and I confirm that I have 2 rows in the SQL console window (see below).

[andrewbaxter439]

ah this is fantastic! somehow the start year changes between runs and that's causing the problem. In all of my runs I've specified start year as 2011 so never had a multiple row problem. I've pushed two new changes which should now correct that issue! Try it out if you get a chance!