Fixtures: Modularize fixtures creating AiiDA test instance and profile #5758

sphuber · 2022-11-13T16:02:27Z

The aiida_profile is a fixture that will automatically load an
existing test profile, if specified by the AIIDA_TEST_PROFILE environment
variable, or otherwise create a temporary and isolated test profile from
scratch. While very useful to make it easy to start writing tests
against an AiiDA instance, it hard-coded most if not all of the
configuration.

For example, the storage backend was hardcoded to the core.psql_dos
storage backend. This used to be fine since this was the only storage
backend available, but since aiida-core==2.1 they are pluginnable and
so it becomes necessary to be able to configure them.

The aiida_profile still works the same as in that it automatically
provides a fully loaded AiiDA test instance, and it still loads the
core.psql_dos storage by default, but it does so through the new
aiida_profile_factory fixture. This fixture can be reused in plugin
packages to achieve the same effect, but they can override any setting
of the profile configuration by providing those through the
custom_configuration argument.

These new fixtures make the aiida.manage.tests.main module obsolete.
It was added when the tests were still run with the unittest module
instead of pytest. The module is deprecated as all of its
functionality is completely provided by the new, more modular and more
succinct pytest fixtures.

sphuber · 2022-11-13T16:03:19Z

I am successfully using these fixtures in aiida-s3 to run with minimal changes against a custom storage backend.

Once we are happy with the interface of the fixtures, I will add documentation to the list of public fixtures

ltalirz

Cheers, @sphuber

The main downside I see is that it appears that this change will break the tests in pretty much all AiiDA plugins, which I consider to be quite serious (even if it is easy to fix).

Is there no way to "disable" the fixture for specific tests where it is not needed?

ltalirz · 2022-11-13T21:18:39Z

aiida/manage/tests/pytest_fixtures.py

-def aiida_profile():
-    """Set up AiiDA test profile for the duration of the tests.
+@pytest.fixture(scope='session')
+def postgres_cluster(


it looks to me like some of the code below duplicates what is already in the TemporaryProfileManager in main.py. Can we avoid that?

Not in a way that I can see that will make things more clear. The problem of the TemporaryProfileManager is that it doesn't properly separate concerns but mixes various concepts: The concept of a profile, but hard-codes the integration with a disk repository and a PostgreSQL cluster for a database. The whole point is to properly decouple this so that parts can be easily replaced where needed. I think it is better to deprecate the entire module as I have done, then try and keep bits around and make things complicated. I would have even removed it entirely instead of deprecating it, because the main use case is for plugin to use aiida_profile, but I am not sure if code out there used these classes directly.

sphuber · 2022-11-13T21:32:01Z

Is there no way to "disable" the fixture for specific tests where it is not needed?

That's the problem, since it is autouse=True and session-scoped, it will be loaded as soon as the tests run. This means everything is created and loaded instantly and for example in aiida-s3 I don't even get the chance to load another type of profile.

But I will experiment some more with aiida-s3 to see how we can switch between profiles with different backends.

sphuber · 2022-11-13T23:39:19Z

Ok, I think I may have found a way, but it will require some refactoring of the PsqlDosBackend and the PsqlDostoreMigrator. Need to clean things up though, but will do that later. In short, we should put this on hold until I have cleaned things up.

sphuber · 2022-11-17T21:26:55Z

@ltalirz with #5760 merged, I have rebased this PR and it is now ready for review. The aiida_profile fixture remains autouse=True.

ltalirz

thanks @sphuber !

I assume the relevant code is copied from the aiida.manage.tests, so I did not check all the logic down to the last detail.
If there is some change implementation that you'd like me to look at in particular, let me know.

Before I approve, could you please have a look at conftest.py? It seems to contain a number of fixtures that are very similar to the ones now present in the pytest_fixtures.py (profile_factory, isolated_config, ...).

It seems to me we may be able to get rid of those?

The rationale for placing a fixture into conftest.py vs. the pytest_fixtures.py is that the former is only for aiida-core, while fixtures in the latter become available to all plugins, and should thus be considered public API.

Edit: perhaps we should mention this rationale in the module docstrings of these two modules.

.github/system_tests/pytest/test_memory_leaks.py

.github/system_tests/pytest/test_pytest_fixtures.py

sphuber · 2022-11-17T23:08:45Z

I assume the relevant code is copied from the aiida.manage.tests, so I did not check all the logic down to the last detail. If there is some change implementation that you'd like me to look at in particular, let me know.

Actually, the idea is the same, but I didn't salvage much of the code, or at the least it has been rewritten for a large part. That being said, I have tested them against the aiida-s3 tests, so reasonably confident that they work out of the box.

Before I approve, could you please have a look at conftest.py? It seems to contain a number of fixtures that are very similar to the ones now present in the pytest_fixtures.py (profile_factory, isolated_config, ...).
It seems to me we may be able to get rid of those?

These are actually quite different. They only provide a simple AiiDA instance (.aiida folder with config.json) and are used internally just to test functionality that operates on the config file: setting options, adding/removing profiles etc. This is not something that plugin packages should have to do. And the config/profiles that are being generated are "inert" as in they don't configure a working storage backend with the required services, so it couldn't be used for running tests anyway. There may be a slight bit of overlap that can be reduced, but that would only be for internal use, which I can take care of later.

The rationale for placing a fixture into conftest.py vs. the pytest_fixtures.py is that the former is only for aiida-core, while fixtures in the latter become available to all plugins, and should thus be considered public API.

Exactly right.

Edit: perhaps we should mention this rationale in the module docstrings of these two modules.

I have updated the module docstrings accordingly

Finally, in the added commit I have improved/corrected the docstrings of the fixtures and added them to the fixture documentation with instructions and examples on how to use them.

ltalirz · 2022-11-17T23:34:28Z

Thanks!

In that case I should still have a quick look at the logic and check whether there are any changes with respect to the one in the previous test manager (tomorrow).

Also, I noticed that the system tests are still using the test manager module, and so there are a couple of AiidaDeprecationWarnings in the test logs. Perhaps it makes sense to move these over?

sphuber · 2022-11-18T09:06:30Z

Also, I noticed that the system tests are still using the test manager module, and so there are a couple of AiidaDeprecationWarnings in the test logs. Perhaps it makes sense to move these over?

These are actually testing the test manager module itself. Since we are just deprecating it and not removing them yet, it seems best to keep the tests around as well.

ltalirz

thanks @sphuber , just some final comments

ltalirz · 2022-11-18T15:36:19Z

aiida/manage/tests/pytest_fixtures.py

+def postgres_cluster(
+    database_name: str | None = None,
+    database_username: str | None = None,
+    database_password: str | None = None


with pgtest this variable is superfluous

shall I just get rid of it entirely?

The Postgres class still requires passing an explicit password. Is the idea that when using pgtest that any password will do? So should I just remove the argument from postgres_cluster and pass a random password to Postgres.create_dbuser?

The Postgres class still requires passing an explicit password.

I see.

Is the idea that when using pgtest that any password will do? So should I just remove the argument from postgres_cluster and pass a random password to Postgres.create_dbuser?

I would guess that even an empty password should do

Ok. But does it even make sense to expose the other variables? Would a user ever want to specify an explicit username? If so, then the same could hold for the database, couldn't it? So either we maybe remove both and just keep the database name or we keep all three. But maybe the database name should also not be configurable if the whole point is that it should just create a temporary one. Not sure if you would ever need to have to specify an explicit one

ltalirz · 2022-11-18T15:45:51Z

aiida/manage/tests/pytest_fixtures.py

+            yield configuration.CONFIG
+        finally:
+            if reset:
+                if current_path_variable:


I'm not entirely sure, but if AIIDA_PATH is empty at first, won't this result in AIIDA_PATH being str(dirpath_config) after the test completes?

Perhaps it does not matter for the test suite because the variable is always overridden, but it would seem cleaner to restore the environment to its previous state after the test.

Yeah, I think you are right and I agree that restoring to original state, even if that means unsetting, is the correct behavior

ltalirz · 2022-11-18T15:53:28Z

aiida/manage/tests/pytest_fixtures.py

+        """
+        config = aiida_instance
+        configuration = {
+            'default_user_email': 'test@aiida.local',


should this hardcoded default be made available as a (private) constant at the module level?

Can do. Not sure it matters though, because it people want to override it, they pass it in the custom_configuration. They wouldn't have to start messing around with a module constant.

ltalirz · 2022-11-18T16:22:33Z

All of the remaining points are optional - feel free to proceed as you see fit (no need for further review from my side)

The `aiida_profile` is a fixture that will automatically load an existing test profile, if specified by the `AIIDA_TEST_PROFILE` environment variable, or otherwise create a temporary and isolated test profile from scratch. While very useful to make it easy to start writing tests against an AiiDA instance, it hard-coded most if not all of the configuration. For example, the storage backend was hardcoded to the `core.psql_dos` storage backend. This used to be fine since this was the only storage backend available, but since `aiida-core==2.1` they are pluginnable and so it becomes necessary to be able to configure them. The `aiida_profile` still works the same as in that it automatically provides a fully loaded AiiDA test instance, and it still loads the `core.psql_dos` storage by default, but it does so through the new `aiida_profile_factory` fixture. This fixture can be reused in plugin packages to achieve the same effect, but they can override any setting of the profile configuration by providing those through the `custom_configuration` argument. These new fixtures make the `aiida.manage.tests.main` module obsolete. It was added when the tests were still run with the `unittest` module instead of `pytest`. The module is deprecated as all of its functionality is completely provided by the new, more modular and more succinct pytest fixtures.

sphuber requested review from ltalirz and chrisjsewell November 13, 2022 16:02

sphuber force-pushed the fix/5523/configurable-pytest-fixtures branch from d4acba4 to 83fc907 Compare November 13, 2022 16:19

ltalirz reviewed Nov 13, 2022

View reviewed changes

sphuber mentioned this pull request Nov 15, 2022

StorageBackend: Add the initialise method #5760

Merged

sphuber force-pushed the fix/5523/configurable-pytest-fixtures branch 3 times, most recently from 2de1930 to b5da615 Compare November 17, 2022 17:05

sphuber requested a review from ltalirz November 17, 2022 21:26

ltalirz reviewed Nov 17, 2022

View reviewed changes

.github/system_tests/pytest/test_memory_leaks.py Outdated Show resolved Hide resolved

.github/system_tests/pytest/test_pytest_fixtures.py Outdated Show resolved Hide resolved

sphuber requested a review from ltalirz November 17, 2022 23:13

sphuber force-pushed the fix/5523/configurable-pytest-fixtures branch 5 times, most recently from e671273 to c0a81a3 Compare November 18, 2022 12:00

ltalirz approved these changes Nov 18, 2022

View reviewed changes

sphuber force-pushed the fix/5523/configurable-pytest-fixtures branch from c0a81a3 to 22be5c2 Compare November 18, 2022 16:26

sphuber force-pushed the fix/5523/configurable-pytest-fixtures branch from 22be5c2 to 60054e1 Compare November 18, 2022 16:36

sphuber merged commit da5103e into aiidateam:main Nov 18, 2022

sphuber deleted the fix/5523/configurable-pytest-fixtures branch November 18, 2022 17:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixtures: Modularize fixtures creating AiiDA test instance and profile #5758

Fixtures: Modularize fixtures creating AiiDA test instance and profile #5758

sphuber commented Nov 13, 2022 •

edited

Loading

sphuber commented Nov 13, 2022 •

edited

Loading

ltalirz left a comment

ltalirz Nov 13, 2022 •

edited

Loading

sphuber Nov 13, 2022

sphuber commented Nov 13, 2022 •

edited

Loading

sphuber commented Nov 13, 2022

sphuber commented Nov 17, 2022

ltalirz left a comment •

edited

Loading

sphuber commented Nov 17, 2022 •

edited

Loading

ltalirz commented Nov 17, 2022

sphuber commented Nov 18, 2022

ltalirz left a comment

ltalirz Nov 18, 2022

sphuber Nov 18, 2022

sphuber Nov 18, 2022

ltalirz Nov 18, 2022

sphuber Nov 18, 2022

ltalirz Nov 18, 2022

sphuber Nov 18, 2022

ltalirz Nov 18, 2022

sphuber Nov 18, 2022

ltalirz commented Nov 18, 2022

Fixtures: Modularize fixtures creating AiiDA test instance and profile #5758

Fixtures: Modularize fixtures creating AiiDA test instance and profile #5758

Conversation

sphuber commented Nov 13, 2022 • edited Loading

sphuber commented Nov 13, 2022 • edited Loading

ltalirz left a comment

Choose a reason for hiding this comment

ltalirz Nov 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sphuber commented Nov 13, 2022 • edited Loading

sphuber commented Nov 13, 2022

sphuber commented Nov 17, 2022

ltalirz left a comment • edited Loading

Choose a reason for hiding this comment

sphuber commented Nov 17, 2022 • edited Loading

ltalirz commented Nov 17, 2022

sphuber commented Nov 18, 2022

ltalirz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ltalirz commented Nov 18, 2022

sphuber commented Nov 13, 2022 •

edited

Loading

sphuber commented Nov 13, 2022 •

edited

Loading

ltalirz Nov 13, 2022 •

edited

Loading

sphuber commented Nov 13, 2022 •

edited

Loading

ltalirz left a comment •

edited

Loading

sphuber commented Nov 17, 2022 •

edited

Loading