Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve pytest developer experience #1368

Closed
dmos62 opened this issue May 9, 2022 · 4 comments
Closed

Improve pytest developer experience #1368

dmos62 opened this issue May 9, 2022 · 4 comments
Labels
affects: dx Related to developer experience affects: technical debt Improves the state of the codebase ready Ready for implementation type: enhancement New feature or request work: backend Related to Python, Django, and simple SQL
Milestone

Comments

@dmos62
Copy link
Contributor

dmos62 commented May 9, 2022

Problem

  • Our pytest test suites take too long to run. The db/tests/ suite alone takes 15 minutes to run on my machine (my i5-4670K is dated, but not outdated). This degrades ability to quickly iterate.
  • Also, our tests are brittle, in that whether they pass might depend on the order in which they are run. Them being order-dependent makes it hard to run tests out-of-order, which is often preferable (--ff option runs failed and unrun tests first).
  • Further, our fixtures are bloated and difficult to understand. Because of this, a developer will spend more time writing his tests (or editing others') and the result will be worse.
  • Futher, our test coverage is inefficient due to heavy overlap, meaning that many aspects of Mathesar are tested many times over. This also contributes to brittleness, since one change might affect many tests.

Proposed solution

  • To reduce runtime: refactor fixtures to allow for running multiple tests in parallel;
  • To address order-dependence: setup pytest to run tests in random order by default;
  • To improve readability: establish a coherent fixture system;
  • To improve overlap: identify most expensive tests and factor out overlapping coverage.

Additional context

Issue related to test coverage overlap: #1138

@dmos62 dmos62 added type: enhancement New feature or request affects: dx Related to developer experience affects: technical debt Improves the state of the codebase work: backend Related to Python, Django, and simple SQL status: triage labels May 9, 2022
@dmos62 dmos62 self-assigned this May 9, 2022
@dmos62 dmos62 added this to the [08.1] 2022-05 improvements milestone May 9, 2022
@dmos62 dmos62 added ready Ready for implementation and removed status: triage labels May 9, 2022
@dmos62 dmos62 removed their assignment May 9, 2022
@kgodey kgodey removed this from the [08.1] 2022-05 improvements milestone Jun 1, 2022
@kgodey
Copy link
Contributor

kgodey commented Jun 2, 2022

@dmos62 Is this fixed by your refactor?

@kgodey kgodey added this to the Unprioritized milestone Jun 2, 2022
@dmos62
Copy link
Contributor Author

dmos62 commented Jun 7, 2022

@kgodey the #1222 refactor made runtime a non-issue, improved fixture coherence and improved test hermitization.

Problems still left to address:

  • Fixture coherence/readability can be further improved;
  • order dependence is vastly improved, but random order execution is still worth exploring to further improve robustness;
  • test overlap (testing the same thing many times) is still severe;
  • instantiating a lot of engines on the backend threatens to bottleneck how many parallel test runners we can use.

@kgodey
Copy link
Contributor

kgodey commented Jun 7, 2022

There is a separate issue for connection management: #592

@dmos62 Can you open separate issues for each of the other three items so that we can work on them separately?

@dmos62
Copy link
Contributor Author

dmos62 commented Jun 8, 2022

Split this issue into four issues (see mentioned-in issues above).

@dmos62 dmos62 closed this as completed Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects: dx Related to developer experience affects: technical debt Improves the state of the codebase ready Ready for implementation type: enhancement New feature or request work: backend Related to Python, Django, and simple SQL
Projects
No open projects
Development

No branches or pull requests

2 participants