migrate SSSP export archives

[ltalirz]

Currently, the SSSP export archives are migrated whenever a new container is started (and this takes time)

Just store the already migrated archives in the image.

[yakutovicha]

I am going to take another approach here.

I work on adding to @sphuber's aiida-sssp package a selector widget with downloading capability: aiidateam/aiida-sssp#25

[yakutovicha]

This way we do won't need to import pseudos at the image build/container start-up, but rather do it on-request base.

[ltalirz]

That is another possibility. Just keep in mind that this means every app using QE will need to set this up individually.

I'm not sure how aiida-sssp does things - we should just make sure that it isn't migrating either.

[sphuber]

I'm not sure how aiida-sssp does things - we should just make sure that it isn't migrating either.

The aiida-sssp install command will download the required data straight from the MC archive entry as the single source of truth. Note that I have been working on aiida-pseudo to generalize the functionality, but this will include the same functionality for the SSSP. It is possible to create a family from a pseudo tarball on disk, but it will be a different group plugin, to differentiate it from those installed directly from the MC archive

[csadorf]

Brief update on this issue: With #144 the SSSPs are no longer installed automatically and the QE app is now responsible for installing the SSSPs (aiidalab/aiidalab-qe#25), however like @ltalirz pointed out, since they are not migrated this takes a significant amount of time.

[sphuber]

Are you sure that it is the migration that takes a lot of time and not just the downloading and importing? Anyway, just to point out that you seem to be downloading the .aiida archives from the MC Archive entry. If you want to migrate these, you would have to migrate them all manually and then reupload them and create an entire new version of the MC Archive entry. Since aiida-pseudo has been released and the SSSP install is working, it might be easier to use that package instead. It will download the tarballs instead of the archives and doesn't use the import but a custom mechanism which doesn't need to migrate everytime the archive version is upped in aiida-core. Something to consider.

[csadorf]

Are you sure that it is the migration that takes a lot of time and not just the downloading and importing?

I am not sure about what exactly takes up the time here, I just know it takes a couple of minutes for the whole process. Just a bit longer than what feels comfortable to me and if we could speed it up somehow, that would be ideal.

Anyway, just to point out that you seem to be downloading the .aiida archives from the MC Archive entry. If you want to migrate these, you would have to migrate them all manually and then reupload them and create an entire new version of the MC Archive entry. Since aiida-pseudo has been released and the SSSP install is working, it might be easier to use that package instead. It will download the tarballs instead of the archives and doesn't use the import but a custom mechanism which doesn't need to migrate everytime the archive version is upped in aiida-core. Something to consider.

Very happy to use utility provided by aiida-pseudo for the QE app instead, whatever works best. Is this related to aiidalab/aiidalab-qe#6 ?

[sphuber]

It is related to that. That was opened against aiida-sssp which was the predecessor of aiida-pseudo and is now deprecated. I will have to go back to aiida-pseudo to see that it is still easy for other clients to easily use the code to install an SSSP family without having to go through our CLI interface. Will try to look at this soon

[ltalirz]

By the way, you might get some speedup already from upgrading to aiida 1.5 and benefitting from the improved import/migration code (you can easily test this locally)

[yakutovicha]

The QE app is now capable of downloading the pseudos, so the issue seems to be resolved. I am going to close it. Feel free to reopen in case you disagree.