-
Notifications
You must be signed in to change notification settings - Fork 19
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* dask logical type inference * add dask to requirements.txt * lint fix * refactor inference tests * more test refactors * update release notes * update test fixtures * update conftest.py * Various updates for Dask validation and testing (#260) * updates for dask validation * update release notes * comment out unused code * update time index test for dask * parameterize test series and fix tests * lint fix * add rows to sample_df and fix head * Skip Ordinal order validation for Dask dataframes (#270) * updates for dask validation * update release notes * comment out unused code * update time index test for dask * parameterize test series and fix tests * lint fix * fix ordinal test with Dask * update release notes * fix accidental change in conftest.py * fix changelog * fix changelog * Better dask datetime support (#286) * better dask datetime support * update release notes * remove test xfail * Test numeric time index with Dask input (#288) * parameterize numeric time index tests * parameterize additional tests * update numeric time index test * update pop method to work with Dask * Update DataTable.describe to work with Dask (#296) * update describe to work with Dask * update release notes * improve describe performance with Dask * fix release notes * Update DataTable.get_mutual_information() to work with Dask (#300) * update mi to work with Dask * update release notes * update test fixture * Create using Woodwork with Dask guide (#304) * add using Woodwork with Dask docs guide * update release notes * remove comments and empty cell * update dataframe wording * remove empty cell * fix title capitalization * update release notes * fix merge issues * remove unused args from to_pandas util func
- Loading branch information
1 parent
bdcc446
commit 662c1b7
Showing
16 changed files
with
851 additions
and
357 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Using Woodwork with Dask DataFrames\n", | ||
| "\n", | ||
| "Woodwork enables DataTables to be created from Dask DataFrames when working with datasets that are too large to easily fit in memory. Although creating a DataTable from a Dask DataFrame follows the same process as one would follow when creating a DataTable from a pandas DataFrame, there are a few limitations to be aware of. This guide will provide a brief overview of creating a DataTable starting with a Dask DataFrame, and will outline several key items to keep in mind when using a Dask DataFrame as input.\n", | ||
| "\n", | ||
| "First we will create a Dask DataFrame to use in our example. Normally you would create the DataFrame directly by reading in the data from saved files, but we will create it from a demo pandas DataFrame." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import dask.dataframe as dd\n", | ||
| "import woodwork as ww\n", | ||
| "\n", | ||
| "df_pandas = ww.demo.load_retail(nrows=1000, return_dataframe=True)\n", | ||
| "df_dask = dd.from_pandas(df_pandas, npartitions=10)\n", | ||
| "df_dask" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now that we have a Dask DataFrame, we can use it to create a Woodwork DataTable, just as we would with a pandas DataFrame:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "dt = ww.DataTable(df_dask, index='order_product_id')\n", | ||
| "dt.types" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "As you can see from the output above, the DataTable was created successfully, and logical type inference was performed for all of the columns. However, this brings us to one of the key issues in working with Dask DataFrames. \n", | ||
| "\n", | ||
| "In order to perform logical type inference, Woodwork needs to bring the data into memory so it can be analyzed. Currently, Woodwork reads data from the first partition of data only, and then uses this data for type inference. Depending on the complexity of the data, this could be a time consuming operation. Additionally, if the first partition is not representative of the entire dataset, the logical types for some columns may be inferred incorrectly.\n", | ||
| "\n", | ||
| "If this process takes too much time, or if the logical types are not inferred correctly, users have the ability to manually specify the logical types for each column. If the logical type for a column is specified, type inference for that column will be skipped. If logical types are specified for all columns, logical type inference will be skipped completely and Woodwork will not need to bring any of the data into memory when creating the DataTable.\n", | ||
| "\n", | ||
| "To skip logical type inference completely, and/or to correct type inference issues, you would simply define a logical types dictionary with the correct logical type defined for each column in the dataframe. Then, pass that dictionary to the call to create the DataTable as shown below:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "logical_types = {\n", | ||
| " 'order_product_id': 'WholeNumber',\n", | ||
| " 'order_id': 'Categorical',\n", | ||
| " 'product_id': 'Categorical',\n", | ||
| " 'description': 'NaturalLanguage',\n", | ||
| " 'quantity': 'WholeNumber',\n", | ||
| " 'order_date': 'Datetime',\n", | ||
| " 'unit_price': 'Double',\n", | ||
| " 'customer_name': 'FullName',\n", | ||
| " 'country': 'Categorical',\n", | ||
| " 'total': 'Double',\n", | ||
| " 'cancelled': 'Boolean',\n", | ||
| "}\n", | ||
| "\n", | ||
| "dt = ww.DataTable(df_dask, index='order_product_id', logical_types=logical_types)\n", | ||
| "dt.types" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "There are two DataTable methods that also require bringing the underlying Dask DataFrame into memory: `describe` and `get_mutual_information`. When called, both of these methods will call a `compute` operation on the DataFrame associated with the DataTable in order to calculate the desired information. This may be problematic for datasets that cannot fit in memory, so exercise caution when using these methods." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "dt.describe(include=['numeric'])" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "dt.get_mutual_information().head()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Data Validation Limitations\n", | ||
| "\n", | ||
| "When creating a DataTable several validation checks are performed to confirm that the data in the underlying dataframe is appropriate for the specified parameters. Because some of these validation steps require pulling the underlying data into memory, they are skipped when creating a DataTable from a Dask DataFrame. This section provides an overview of the validation checks that are performed with pandas input but skipped with Dask input.\n", | ||
| "\n", | ||
| "### Index Uniqueness\n", | ||
| "Normally a check is performed to verify that any column specified as the index contains no duplicate values. This check is skipped for Dask, so users must manually verify that any column specified as an index column contains unique values.\n", | ||
| "\n", | ||
| "### Data Consistency with LogicalType\n", | ||
| "If users manually define the LogicalType for a column when creating the DataTable, a check is performed to verify that the data in that column is appropriate for the specified LogicalType. For example, with pandas input if the user specifies a LogicalType of `Double` for a column that contains letters such as `['a', 'b', 'c']`, an error would be raised as it is not possible to convert the letters into numeric values with the `float` dtype associated with the `Double` LogicalType.\n", | ||
| "\n", | ||
| "With Dask input, no such error would be raised at the time of DataTable creation. However, behind the scenes, Woodwork will have attempted to convert the column physical type to `float`, and this conversion would be added to the Dask task graph, without raising an error. However, an error will be raised if a `compute` operation is called on the underlying DataFrame once Dask attempts to execute the conversion step. Extra care should be taken when using Dask input to make sure any specified logical types are consistent with the data in the columns to avoid this type of error.\n", | ||
| "\n", | ||
| "Similarly, for the `Ordinal` LogicalType, a check is typically performed to make sure that the data column does not contain any values that are not present in the defined order values. This check will not be performed with Dask input. Users should manually verify that the defined order values are complete to avoid unexpected results." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Other Limitations\n", | ||
| "\n", | ||
| "Woodwork provides the ability to read data directly from a CSV file into a DataTable, and during this process Woodwork creates the underlying dataframe so the user does not have to do so. The helper function used for this, `woodwork.read_csv`, will currently only read the data into a pandas DataFrame. At some point, we hope to remove this limitation and also allow data to be read into a Dask DataFrame, but for now only pandas DataFrames can be created by this function." | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.7.4" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| numpy>=1.19.1 | ||
| pandas>=1.1.0 | ||
| click>=7.1.2 | ||
| dask[dataframe]>=2.30.0 | ||
| scikit-learn>=0.21.3 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.