Update aiida-core to 2.5.1

[danielhollas]

For now just testing the build

[unkcpz]

Not sure why it failed. I rebase with main and seems the tests are fine now.

[unkcpz]

full-stack failed because of pydantic conflict from optimade and aiidalab-qe dependencies.

[yakutovicha]

full-stack failed because of pydantic confiliction from optimade and aiidalab-qe dependencies.

@unkcpz, would it be possible to bump pydantic in the aiidalab-qe? I assume it should be more-or-less smooth.

[unkcpz]

The pinning is coming from aiidalab/aiidalab-qe#416, which can be removed.
But aiida-quantumespresso just have pydantic 2 support in aiidateam/aiida-quantumespresso#977 but net yet release. That the main reason we are not able to bump it.

[danielhollas]

Blocked on aiida-qe release: aiidateam/aiida-quantumespresso#1002

@unkcpz in terms of optimade, does it depend directly on pydantic, or is pydantic a transitive dependency there? Would be good to have an estimate about how difficult it will be to update it.

[unkcpz]

Old version pydandic dependency comes from aiidalab-widgets-base -> optimade-client -> optimade.
The latest version of optimade (https://github.com/Materials-Consortia/optimade-python-tools/tree/v0.25.3) has pydantic v2 supported. However, the optimade-client not yet support optimade.

After moving the optimade-client to the repo we maintained, I'll bump the version. I didn't see any block, I'll try to do it ASAP.

[unkcpz]

The test failed since the stable version of QeApp require aiida-quantumespresso==4.4.0 which require pydantic v1 and conflict with aiida-core==2.5.0 image.
Sholud work after merging aiidalab/aiidalab-qe#632 and make an alpha release.

[danielhollas]

NOTE: I believe we will have to pin the pydantic version in the constraints file (in addition to pinning aiida-core). Specifically pydantic>=2. Otherwise, user could install an older version of AWB with old version of optimade that depended on pydantic v1, which would subsequently break aiida-core. This is because even if we constrain aiida-core, I don't think the constrain applies to transitive dependencies as well. But I might be mistaken, we will need to test this.

[danielhollas]

Oh, never mind, I've just checked the latest test failure and it seems that the aiida-core constraint is already working! 🎉

        ERROR: Cannot install aiidalab_qe because these package versions have conflicting dependencies.
E         ? +++++
E           
E           The conflict is caused by:
E               aiida-core 2.5.0 depends on pydantic~=2.4
E               aiida-quantumespresso 4.4.0 depends on pydantic>=1.10.8 and ~=1.10
E           
E           To fix this you could try to:
E           1. loosen the range of package versions you've specified
E           2. remove package versions to allow pip attempt to solve the dependency conflict
E           
E           ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[unkcpz]

Hi @danielhollas, for the failed test on qeapp, can we just ignore it in order to not block the release of aiidalab-docker-stack with aiida-core 2.5.0 and pydantic v2? TBH I don't like the idea to test the QeApp installation here, although it is something from our side we want to make it always working. But it should not every time the blocker of making release for the docker stack.

[danielhollas]

I don't know, I am not comfortable releasing a version for which QeApp doesn't work. I am okay to wait. Dne st 20. 3. 2024 13:51 uživatel Jusong Yu ***@***.***> napsal:

…

Hi @danielhollas <https://github.com/danielhollas>, for the failed test on qeapp, can we just ignore it in order to not block the release of aiidalab-docker-stack with aiida-core 2.5.0 and pydantic v2? TBH I don't like the idea to test the QeApp installation here, although it is something from our side we want to make it always working. But it should not every time the blocker of making release for the docker stack. — Reply to this email directly, view it on GitHub <#424 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIY64IHWBV6DOLXPTIG3XLYZGH6PAVCNFSM6AAAAABBAMK476VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBZGYYTOMRTGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[unkcpz]

Okay, if you are fine then let's wait. Just let you know, I will (already) slightly move away from QeApp maintenance in order to give more heart on aiidalab in general, so the progress will be slow down from QeApp side.

[unkcpz]

I don't know, I am not comfortable releasing a version for which QeApp doesn't work.

I had a discuss with @superstar54 on this in the morning just summarise it here, we think it might be better to have it a scheduled (or so called nightly) test on QeApp instead of a mandatory test on QeApp. We can see the test dependent loop between QeApp and aiidalab-docker-stack which make us goes back and forth to consider the relation on how to proceed on making release. Not very ideal I think. Want to see your opinion on this @danielhollas

[danielhollas]

I am currently on holiday and on a conference next week, I'll try to summarise my thinking next week Dne pá 22. 3. 2024 0:49 uživatel Jusong Yu ***@***.***> napsal:

…

I don't know, I am not comfortable releasing a version for which QeApp doesn't work. I had a discuss with @superstar54 <https://github.com/superstar54> on this, we think it might be better to have it a scheduled (or so called nightly) test on QeApp instead of a mandatory test on QeApp. We can see the test dependent loop between QeApp and aiidalab-docker-stack which make us goes back and forth to consider the relation on how to proceed on making release. Not very ideal I think. Want to see your opinion on this @danielhollas <https://github.com/danielhollas> — Reply to this email directly, view it on GitHub <#424 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIY64L6TMCEJKJK7ZOSK7DYZNWY7AVCNFSM6AAAAABBAMK476VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJUGA2TSOBQGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[danielhollas]

btw: Before this is merged, I'd like to release one last stable version where the aiida-2.4 version would be pinned, as proposed in #436

[yakutovicha]

Maybe we should set the aiida-core version to 2.5.1 . It has some bug fixes.

[unkcpz]

I think it is ready?

[danielhollas]

Looks like it. @unkcpz or @superstar54 it would be great if you could test it manually with QeApp. I'll do the same with my app.

Marking as Draft since I want to release one last image in 2.4 series.

[danielhollas]

I released last image in the 2.4.3, so merging this, people can test with the full-stack:edge image.

[unkcpz]

Cool, thanks! There are two ways to adapt aiidalab-qe to run on new image. The Qeapp can installed from appstore, I suggest @superstar54 to test it, basically start a pure full-stack container and install qeapp from appstore and test it. I think some people if they follow the aiidalab documentation will still do.

It can also run directly with the aiidalab-qe container that will build on top of full-stack:latest, I'll test it with building on top of edge.

[superstar54]

Hi @danielhollas @unkcpz I installed QEApp from the App Store using the full-stack:edge image. It works! And I noticed I didn't need to wait the code installation in the step3, which is seen in the previous version. This much because the improvement of the loading time and the verdi command. Great work! @danielhollas .Cheer!

There is one small issue on the node tree: it does not show the sub-nodes of the sub-tree, like bands/pdos, but I checked, and it still used the AWB 2.2.0a1, which does not include the latest feature implemented in aiidalab/aiidalab-widgets-base#583, If I install latest AWB from the repo, everything work great!

[danielhollas]

@superstar54 awesome! Great to hear that the loading speed has been improved, although I would be surprised if my changes influenced Step 3, unless you somehow directly invoke verdi?

@superstar54 Would you mind taking a look at aiidalab/aiidalab-qe#688?
I would like to get it in before next release, and I also plan to make analogous PR for AWB. Once that's done I'll release a new AWB version and we should be good to go.

[danielhollas]

although I would be surprised if my changes influenced Step 3, unless you somehow directly invoke verdi?

Oh, I see you do in fact invoke verdi code create so I would indeed imagine it would be faster with 2.5.1

https://github.com/aiidalab/aiidalab-qe/blob/44fc111ee8184edf7b6fe4d7e3fc4e10a3343707/src/aiidalab_qe/common/setup_codes.py#L87

@sphuber by the way, isn't there a Python API for verdi code create? Us calling verdi directly seems wasteful.

[superstar54]

isn't there a Python API for verdi code create? Us calling verdi directly seems wasteful.

Yes, there is InstalledCode Python API. However, the code setup in QEApp is run in a different thread, thus, using the InstalledCode directly will fail when storing the code. If you are interested, I wrote it in here aiidalab/aiidalab-qe#565.

[danielhollas]

I see, thanks. Still, I think you should be able to use the Python API even in a different thread, you just need to make sure to use fresh DB connection.

[superstar54]

I think you should be able to use the Python API even in a different thread, you just need to make sure to use fresh DB connection.

Thanks! Could you give a working example?

[danielhollas]

I am not that familiar with QeApp setup. If you open a new issue with a full stacktrace of what happens when you try it I am happy to help to figure it out.

[unkcpz]

Tested QeApp image with full-stack:edge, and works as expected.

[superstar54]

I think you don't need to wait the code installation because you install the app from appstore which calling https://github.com/aiidalab/aiidalab-qe/blob/main/post_install before you start the app the first time.

A user installed QEApp from the App Store, and then:

• post_install is called, and needs some time (e.g., ~200 seconds) to finish
• at the same time, the user opened QEApp, and run test job, if the user is fast enough (in my case), I reached the step 3, and the post_instlal is not finished yet, and the file is locked, so the page will show waiting to setting up the codes.

I bet you never did install from app store before

Please just run the test and check (image aiida-core 2.4.3, QEApp 2024.04.0a3 or rc0), then you will know. You need fast enough to catch this.

[unkcpz]

And I noticed I didn't need to wait the code installation in the step3, which is seen in the previous version. This much because the improvement of the loading time and the verdi command. Great work! @danielhollas .Cheer!

I didn't see much improvement of code setup from my test, here is how I tested, maybe something not correct in my test?

In the container with aiida-core==2.5.1:

In the container with aiida-core==2.4.3:

[unkcpz]

@danielhollas, can we make a release with aiida-core 2.5.1? We need it for QeApp container to be tested next week during the demo server coding day.

[danielhollas]

@danielhollas, can we make a release with aiida-core 2.5.1? We need it for QeApp container to be tested next week during the demo server coding day.

I pushed a new release just now, hopefully will be available in few hours.