Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix pytest-xdist with session-scoped fixture #94

Closed
tallamjr opened this issue Jun 11, 2022 · 0 comments
Closed

Fix pytest-xdist with session-scoped fixture #94

tallamjr opened this issue Jun 11, 2022 · 0 comments
Assignees
Labels
1 - refactor DRY conflict. Refactor required 1 - tests A Failed test or tests needs updating 4 - back-burner Item on hold, low priority

Comments

@tallamjr
Copy link
Owner

Currently, it is not possible to use more than --numprocessor=1 as with pytest-xdist the session scope is actually replicated and it does multiple loads of the data in fixture per worker.

There are ways to overcome this with code examples found here, but since we load a mix of numpy files and tf.data.Prefectdataset, it becomes a little complicated which for now doesn't really matter.

With this issue fix, tests will surely be able to run fast as we can then run in parallel, but for now must be run with single worker, otherwise there is sure to be memory problems

Refs:

@tallamjr tallamjr added 1 - tests A Failed test or tests needs updating 1 - refactor DRY conflict. Refactor required 4 - back-burner Item on hold, low priority labels Jun 11, 2022
@tallamjr tallamjr self-assigned this Jun 11, 2022
tallamjr added a commit that referenced this issue Jun 12, 2022
Allow for distributed tests and reduce inference test time.

Closes #94
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1 - refactor DRY conflict. Refactor required 1 - tests A Failed test or tests needs updating 4 - back-burner Item on hold, low priority
Projects
None yet
Development

No branches or pull requests

1 participant