Unit test all dependency management approaches

[ehmatthes]

Right now all unit tests assume the local project is using a bare requirements.txt file. Implement an efficient approach to unit testing all three currently-supported dependency management approaches: bare requirements.txt (req_txt), Pipenv (pipenv), and Poetry (poetry).

This should roughly triple the number of unit tests. Right now there are 24 tests, and they run in about 20s. There should be about 60 tests run when this ticket is finished, and the test suite should still run in under a minute.

• First attempt: try parametrizing the reset_project() fixture.
• Pass all unit tests, for all package managers. 24 tests ~20 seconds -> 90 tests ~30 seconds. :)
• Note open tabs.
• Update documentation.
• Move relevant tasks to other issues.

[ehmatthes]

Parametrize reset_project() fixture

Parametrize reset_project() in conftest.py, using each of the three dependency management approaches. The test functions will each need the tmp_project directory, and the parameter that specifies the dependency management approach.

• Since tmp_proj_dir is already being returned, maybe I can just return the value specifying the dependency management approach that's being tested?
• Current progress: Parametrized reset_test_project() successfully. Can access that param in test function, but it's laborious: request.node.callspec.params.get("reset_test_project")
• Pass the Heroku requirements.txt test using the above approach to accessing fixture param.
  • req_txt and poetry should both match current reference requirements.txt
  • pipenv should not have a requirements.txt file.
• Write a test for Pipfile, using same approach to accessing param.
  • Should not exist for req_txt and poetry.
  • Should match new reference file for pipenv.
• Write a fixture that's autoused that simply returns the param, so the long call to access this is only written once.
• Write a test for pyproject.toml.
  • Exists for poetry, should match reference file.
  • Should not exist for req_txt or pipenv.
• Same approach for fly unit tests.
• Same approach for platform.sh unit tests.
• Is there any way to allow setting these parameters from the command line?
  • Need for this depends on how long the test suite takes overall.
  • See Dynamic scope for fixtures. I wonder if there's a similar way to dynamically set the parameters from the command line? ie if CLI pkg_manager passed, use that, otherwise, use all existing values. Could limit CLI option to specifying one specific pkg_manager.
• Run, and note rough benchmark, for entire test suite.

[ehmatthes]

Working notes

• For development work, if it's difficult to set parameters from the command line, can always manually set the list of parameters for a test run, ie to build a new reference file.
• If writing test targeting a file for which there's no reference file yet, can run the test, it should fail, then look in test project, and you can see if it's generated a correct reference file. Use pytest -x platforms/target_platform, you'll see something like FAILED platforms/platform_sh/test_platformsh_config.py::test_platform_app_yaml_file[poetry] - AssertionError.
• I can't figure out how to autouse the pkg_manager fixture, without needing to list it in a test function's parameters.

[ehmatthes]

Fly tests

• test_req_txt
• test_pyproject_toml
• test_pipfile
• test_dockerfile
• test_fly_toml

[ehmatthes]

Platform.sh tests

• requirements.txt
• pyproject.toml
• pipfile
• .platform.app.yaml

[ehmatthes]

Platformsh tests pass when run in isolation, fail when run in entire suite

Run in isolation:

$ pytest -x platforms/platform_sh
...
=== 24 passed in 15.15s ===

Entire suite:

$ pytest -x
...
collected 93 items                                                                                                                  

platform_agnostic_tests/test_invalid_cli_commands.py .........
platform_agnostic_tests/test_project_inspection.py sss
platform_agnostic_tests/test_valid_cli_commands.py ...
platforms/fly_io/test_flyio_config.py ...........................
platforms/heroku/test_heroku_config.py ...........................
platforms/platform_sh/test_platformsh_config.py F
...
FAILED platforms/platform_sh/test_platformsh_config.py::test_settings[req_txt] - AssertionError
=== 1 failed, 66 passed, 3 skipped in 23.19s ===

Most relevant output:

utils/ut_helper_functions.py:44: AssertionError
------------------------------------------------------- Captured stdout setup ---
HEAD is now at fcc4a70 Initial commit.
Configuring project for deployment...
  Deployment target: platform_sh
  Found .git dir at /private/var/folders/md/4h9n_5l93qz76s_8sxkbnxpc0000gn/T/pytest-of-eric/pytest-247/blog_project0.
  Dependency management system: poetry
  Checking current project requirements...
------------------------------------------------------- Captured stderr setup -------
rm: fly.toml: No such file or directory
rm: Dockerfile: No such file or directory
rm: .dockerignore: No such file or directory
rm: .platform.app.yaml: No such file or directory
Traceback (most recent call last):
  File "/private/var/folders/md/4h9n_5l93qz76s_8sxkbnxpc0000gn/T/pytest-of-eric/pytest-247/blog_project0/manage.py", line 22, in <module>
    main()
...
    self._inspect_project()
  File "/Users/eric/projects/django-simple-deploy/simple_deploy/management/commands/simple_deploy.py", line 332, in _inspect_project
    self.requirements = self._get_current_requirements()
  File "/Users/eric/projects/django-simple-deploy/simple_deploy/management/commands/simple_deploy.py", line 505, in _get_current_requirements
    requirements = self._get_poetry_requirements()
  File "/Users/eric/projects/django-simple-deploy/simple_deploy/management/commands/simple_deploy.py", line 631, in _get_poetry_requirements
    parsed_toml = toml.loads(self.pyprojecttoml_path.read_text())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 1132, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 1117, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/md/4h9n_5l93qz76s_8sxkbnxpc0000gn/T/pytest-of-eric/pytest-247/blog_project0/pyproject.toml'

• Something is causing it to identify this run as a poetry run (Dependency management system: poetry), but there's no pyproject.toml?
  • Maybe it's an issue of which order the fixtures are run in? (No, see notes below.)

[ehmatthes]

Fixing platform.sh tests, when run in suite

• Related question: When a test fails, why do I still see all of the dependency management files in the test repo? Is that a cleanup artifact?
  • try running ls -alh at beginning and end of reset script.
  • This clarified some files that weren't being removed, ie poetry.lock, etc. Remove extra files after git reset, then remove package management specific files. All tests pass.

[ehmatthes]

Pytest sources

• SO In which order are pytest fixtures executed? The documentation is more authoritative, but this discussion is helpful for people's experiences in trying to apply what they've read. Fixture ordering is famously hard. :/
• doc Autouse fixtures are executed first within their scope "Autouse fixtures are assumed to apply to every test that could reference them, so they are executed before other fixtures in that scope. Fixtures that are requested by autouse fixtures effectively become autouse fixtures themselves for the tests that the real autouse fixture applies to."
• Fixture availability
• SO Accessing fixture params from a test function, same topic
• docs Parametrizing fixtures
• docs article How to parametrize fixtures and test functions