Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit tests are very very slow #207

Open
brews opened this issue Jul 1, 2022 · 0 comments
Open

Unit tests are very very slow #207

brews opened this issue Jul 1, 2022 · 0 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@brews
Copy link
Member

brews commented Jul 1, 2022

Unit tests run in CI have become very very slow. The can take about 15 minutes to complete.

I suspect dodola.services.validate() using dask.delayed is the culprit. Dask was used here as a hack to work around memory problems (see #126). The drawback to this hack has been that simple unit tests on Datasets covering a long time range take much more time to run.

If this is the case I see two options:

  1. Re-write validation to work around the original memory problems in dodola validate-dataset reads in entire zarr store? #126 while also making unit tests complete faster. i.e. replace the hack solution with a "real" solution.
  2. Find and refactor the one or two long-running unit tests so that they test the same behavior, but run faster.
@brews brews added bug Something isn't working help wanted Extra attention is needed labels Jul 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant