Changing the pipeline for qe2pert tests: downloading required files instead of QE calculations.

[hurricane642]

Hi everyone!
In this PR we are completely changing the test approach for qe2pert.x. In the original version, in order to compute epr files, we did a full computation starting from SCF. This allowed us not to waste memory storing heavy files, some of which are binary.
However, this approach has 2 significant disadvantages:

1. Test time. On turbo all testing took 20 minutes, in case of running tests in docker containers it took several hours.
2. Too high variation of results. Either because of the specific compilation of QE, or because of the fact that our calculations are quite simple and inaccurate initially, we often received epr files that differed significantly from the original ones, even if qe2pert.x was working correctly. This did not allow us to test this executable file normally.
   Therefore, it was decided to use the following approach:
3. store all files needed for qe2pert.x calculations in remote storage (Box);
4. Load them automatically when qe2pert.x tests are run.
   Thanks to the answer from https://lists.quantum-espresso.org/pipermail/users/2024-August/051827.html, we learned that binaries are not a problem when switching between GNU and Intel, so for each epr file we just have a folder inside the archive. The archive itself weighs 412 MB, which is not so critical. The archive is stored in our group's Caltech Box, and the file is accessible to everyone, but I don't think that's a problem. At the same time, the time of tests on turbo was reduced to 3 minutes, in docker containers it became less than 5 minutes, and the problem with stability of calculations seems to have disappeared.

I'll be glad to receive any suggestions regarding the code and ideas on how to improve it!

[imaliyov]

This is great, Sergei!

Two questions about Box:

1. Is it on your personal account? If so, we have to migrate files to the PI's account, or create a Perturbo account. We had this problem before: once a student leaves, we have no more access to files from his/her account.
2. Do we need any keys to access files using API? If so, can these keys expire?

[hurricane642]

This is great, Sergei!

Two questions about Box:

1. Is it on your personal account? If so, we have to migrate files to the PI's account, or create a Perturbo account. We had this problem before: once a student leaves, we have no more access to files from his/her account.
2. Do we need any keys to access files using API? If so, can these keys expire?

1. No, I've used perturbo account, It's registered to the entire group, so it's should be good;
2. Basically, I put up a link that doesn't expire and is open to anyone who has the link. That is, only the link itself is needed to download files.
   There is only 1 problem - because of Box peculiarities I had to save test files as a zip-archive. As a result, every time we update the test files we will have to update it as well as the link. And here comes another problem - to create a link to download an archive, you really need to use the API. But you don't need to know any unique keys to do that - just need to become an Editor in that repository. There is a series of additional steps, they aren't very complicated, but not obvious at the beginning. I can write in the manual how it can be done.

[imaliyov]

For legacy code, like run_scf, etc., can we keep it somewhere?
Maybe in a different file?..
It is not needed now, but it does the job of all the preliminary calculations that might be needed in future for other projects.

So, now it is all commented out, right?
Maybe we can leave it uncommented by in another file?..

[hurricane642]

For legacy code, like run_scf, etc., can we keep it somewhere? Maybe in a different file?.. It is not needed now, but it does the job of all the preliminary calculations that might be needed in future for other projects.

So, now it is all commented out, right? Maybe we can leave it uncommented by in another file?..

Yes, that's a great idea! I have separated all legacy functions into a separate file in a separate folder.

[imaliyov]

Cool! Feel free to merge when you like.