Improve speed of link only sample test #1773
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RobinL
commented
Nov 30, 2023
| @@ -76,19 +96,12 @@ def estimate_u_values(linker: Linker, max_pairs, seed=None): | |||
| result = dataframe.as_record_dict() | |||
| dataframe.drop_table_from_database_and_remove_from_cache() | |||
| frame_counts = [res["count"] for res in result] | |||
| # total valid links is sum of pairwise product of individual row counts | |||
There was a problem hiding this comment.
Moved this code to function so it could be tested individually
RobinL
commented
Nov 30, 2023
| linker = helper.Linker( | ||
| [df_l, df_r], | ||
| settings, | ||
| input_table_aliases=["_a", "_b"], |
There was a problem hiding this comment.
this may make it slightly faster
RobinL
commented
Nov 30, 2023
|
|
||
| # max_pairs is a good deal less than total possible pairs = 9_000_000 | ||
| max_pairs = 1_800_000 | ||
|
|
||
| settings = { | ||
| "link_type": "link_only", | ||
| "comparisons": [helper.cl.levenshtein_at_thresholds("name", 2)], | ||
| "comparisons": [helper.cl.exact_match("name")], |
There was a problem hiding this comment.
no need for a CPU-heavy comparison to test this functionality
ADBond
approved these changes
Nov 30, 2023
tests/test_u_train.py
Outdated
Comment on lines
30
to
31
| helper.extra_linker_args() | ||
|
|
There was a problem hiding this comment.
is this line here by mistake?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Now we've fixed the analyse blocking test, this is now the slowest test
Combining a
sparktest withdebug_mode=Trueis very slow, because lots of intermediate tables are produced and output to parquet. (I've also tried setting the checkpointing method topersistbut that didn't help either)In this PR, I've disabled the spark test, and included a new test for the underlying calculation
https://mojdt.slack.com/archives/C060KLFJXPX/p1699978472369589
Here's some previous notes coped from slack:
I have been experimenting with this test (the second slowest).
The problem is I can’t find an easy to to make it faster
It would be possible to refactor the test to make it faster, but at a significant loss of clarity as to what the test is actually doing
reducing size of data makes the test close to useless. needs big sample because of the random element.
running debug_mode = True is really bad for Spark performance, but necessary to test the high-level API
possibly there’s a solution with a seed, but not sure how robust, and only works in spark and duckdb
I wonder if the best solution is simply to disable Spark for this test. On the basis that it’s really testing the random sample size is computed correctly - and if that’s right in Duckdb, then it should be right in Spark. (realise that isn’t completely watertight, but if it’s wrong other things should break too).