Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spark fixture #1698

Merged
merged 1 commit into from
Nov 6, 2023
Merged

Fix spark fixture #1698

merged 1 commit into from
Nov 6, 2023

Conversation

ADBond
Copy link
Contributor

@ADBond ADBond commented Nov 6, 2023

Type of PR

  • BUG
  • FEAT
  • MAINT
  • DOC

Is your Pull Request linked to an existing Issue or Pull Request?

Previously test_helpers depended on the spark fixture. That means this gets created whenever we request the test_helpers fixture, which is slow (and particularly annoying if you're just running e.g. duckdb tests).

Give a brief description for the solution you have provided

Now both test_helpers and spark fixtures use a helper function, with the upshot that spark is only created in test_helpers whenever a SparkTestHelper is instantiated (which should only happen when you are running spark tests, due to LazyDict)

Currently this means that spark is created afresh each time we access a SparkTestHelper, which is not ideal.

PR Checklist

  • Added documentation for changes
  • Added feature to example notebooks or tutorial (if appropriate)
  • Added tests (if appropriate)
  • Updated CHANGELOG.md (if appropriate)
  • Made changes based off the latest version of Splink
  • Run the linter

if `test_helpers` depends on the `spark` fixture, then this gets created whenever we request the `test_helpers` fixture, which is slow. Now both `test_helpers` and `spark` fixture use a helper function, with the upshot that `spark` is only created in `test_helpers` whenever `SparkTestHelper` is instantiated
@ADBond ADBond added the testing label Nov 6, 2023
@ADBond ADBond requested a review from RobinL November 6, 2023 10:41
@ADBond ADBond merged commit e575a4e into master Nov 6, 2023
6 checks passed
@ADBond ADBond deleted the fix-spark-fixture branch November 6, 2023 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants