-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds regression tests for #2020 #2021
Conversation
@@ -520,7 +520,18 @@ def run_api_experiment(input_features, output_features, data_csv): | |||
shutil.rmtree(output_dir, ignore_errors=True) | |||
|
|||
|
|||
def create_data_set_to_use(data_format, raw_data): | |||
def read_csv_with_nan(path, nan_percent=0.0): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function converts nan_percent
of samples in each row of the CSV into NaN. This is important for tests that drop rows for exactly one feature– with this change, one has guarantees about the number of rows that will be dropped.
Example:
In a unit test, we are simulating predicting the targets
column which is missing 10% of samples. We choose to drop rows missing a value for targets
. With this sampling scheme, we know we will have exactly 90% of samples left.
Adds regression tests for #2020, a PR that implemented a fix for NaNs introduced via an outer join
concat
in the dask df engine.While writing these tests, I found that
PandasEngine.df_like
was also doing an outer join ofproc_cols
(implicitly throughpd.DataFrame
init) instead of an inner join, similarly leading to NaN values in columns whose preprocessing step called for dropping rows (typically OutputFeature features). This is remedied through an inner joinconcat
. We would like to be able to implement an inner joinconcat
in the dask df engine in the future. It is not currently possible in the dask df engine due to the parallel nature of dask dataframes.