Improve import times of aiida.orm and aiida.storage

[danielhollas]

1. aiida.orm went from 600ms back to 300ms by deferring import of aiida.storage.
2. Also some small improvements to both aiida.orm and aiida.storage, mainly from deferring build of pydantic models.

Main branch

Benchmark 1: python -c "import aiida.orm"
  Time (mean ± σ):     650.2 ms ±   9.2 ms    [User: 622.8 ms, System: 852.6 ms]
  Range (min … max):   641.0 ms … 663.2 ms    10 runs
 
Benchmark 1: python -c "import aiida.storage"
  Time (mean ± σ):     644.3 ms ±  12.0 ms    [User: 608.7 ms, System: 859.6 ms]
  Range (min … max):   632.4 ms … 671.6 ms    10 runs

Benchmark 1: verdi code list
  Time (mean ± σ):     926.9 ms ±  30.6 ms    [User: 871.6 ms, System: 852.1 ms]
  Range (min … max):   889.9 ms … 997.4 ms    10 runs

This branch

Benchmark 1: python -c "import aiida.orm"
  Time (mean ± σ):     282.4 ms ±   6.3 ms    [User: 296.7 ms, System: 794.5 ms]
  Range (min … max):   273.1 ms … 290.3 ms    10 runs
 
Benchmark 1: python -c "import aiida.storage"
  Time (mean ± σ):     588.4 ms ±  16.3 ms    [User: 564.5 ms, System: 857.0 ms]
  Range (min … max):   560.4 ms … 610.8 ms    10 runs

Benchmark 1: verdi code list
  Time (mean ± σ):     910.0 ms ±  29.4 ms    [User: 862.8 ms, System: 872.7 ms]
  Range (min … max):   836.6 ms … 939.5 ms    10 runs

[sphuber]

Thanks @danielhollas . I guess the CI test we added a long time ago to catch these regressions is not doing its job. Guess it can be difficult to tune where we catch real significant regressions in performance but don't get hit by false positives all the time. Appreciate your effort to optimize this manually 👍

[danielhollas]

Appreciate your effort to optimize this manually 👍

Cheers. :-)

I guess the CI test we added a long time ago to catch these regressions is not doing its job.

The tests works fine, but they only preventregressions in verdi. I've opened #6387 with some suggestions for further improvements.

I am afraid that even though I fixed the aiida.orm regression, it's practical impact will be small, since for any DB interaction one needs to load aiida.storage anyway (and one can see that the timings for verdi code list did not change much here). I am quite sad that even a very simple DB interaction takes a whole second (and it can pose real issues on AiiDAlab side). I'd like to understand more where the time is being spent. By some initial investigation, a lot of time is spent in sqlalchemy initialization. Seems like building the orm models is expensive, and we seem to be paying an upfront cost for all the DB backends unfortunately. I wonder if SQL supports something like defer_build in pydantic?