Azure Machine Learning Data Prep SDK

You will find in this repo:

• How-To Guide Notebooks for more in-depth feature examples.
• Case Study Notebooks that show in-depth scenario examples of features.
• Getting Started Tutorial for a quick introduction to the Data Prep SDK and some of its main features.

Installation

Here are the SDK installation steps.

Documentation

Here is more information on how to use the new Data Prep SDK:

• SDK overview and API reference docs that show different classes, methods, and function parameters for the SDK.
• Tutorial: Prep NYC taxi data for regression modeling and then run automated machine learning to build the model.
• How to load data is an overview guide on how to load data using the Data Prep SDK.
• How to transform data is an overview guide on how to transform data.
• How to write data is an overview guide on how to write data to different storage locations.

Known Issues

• If running version 0.1.0: To fix "Error Message: Cannot run the event loop while another loop is running", downgrade Tornado version to 4.5.3. Restart any running kernels for the change to take effect.

pip install -U tornado==4.5.3

Release Notes

2019-02-27 (version 1.0.16)

Bug fix

• Fixed a Service Principal authentication issue that was caused by an API change.

2019-02-25 (version 1.0.15)

New features

• Data Prep now supports writing file streams from a dataflow. Also provides the ability to manipulate the file stream names to create new file names.
  • How-to guide: Working With File Streams notebook

Bug fixes and improvements

• Improved performance of t-Digest on large data sets.
• Data Prep now supports reading data from a DataPath.
• One hot encoding now works on boolean and numeric columns.
• Other miscellaneous bug fixes.

2019-02-11 (version 1.0.12)

New features

• Data Prep now supports reading from an Azure SQL database using Datastore.

Changes

• Significantly improved the memory performance of certain operations on large data.
• read_pandas_dataframe() now requires temp_folder to be specified.
• The name property on ColumnProfile has been deprecated - use column_name instead.

2019-01-28 (version 1.0.8)

Bug fixes

• Significantly improved the performance of getting data profiles.
• Fixed minor bugs related to error reporting.

2019-01-14 (version 1.0.7)

New features

• Datastore improvements (documented in Datastore how-to-guide)
  • Added ability to read from and write to Azure File Share and ADLS Datastores in scale-up.
  • When using Datastores, Data Prep now supports using service principal authentication instead of interactive authentication.
  • Added support for wasb and wasbs urls.

2019-01-09 (version 1.0.6)

Bug fixes

• Fixed bug with reading from public readable Azure Blob containers on Spark.

2018-12-19 (version 1.0.4)

New features

• to_bool function now allows mismatched values to be converted to Error values. This is the new default mismatch behavior for to_bool and set_column_types, whereas the previous default behavior was to convert mismatched values to False.
• When calling to_pandas_dataframe, there is a new option to interpret null/missing values in numeric columns as NaN.
• Added ability to check the return type of some expressions to ensure type consistency and fail early.
• You can now call parse_json to parse values in a column as JSON objects and expand them into multiple columns.

Bug fixes

• Fixed a bug that crashed set_column_types in Python 3.5.2.
• Fixed a bug that crashed when connecting to Datastore using an AML image.

2018-12-07 (version 0.5.3)

Fixed missing dependency issue for .NET Core2 on Ubuntu 16.

2018-12-03 (version 0.5.2)

Breaking changes

• SummaryFunction.N was renamed to SummaryFunction.Count.

Bug fixes

• Use latest AML Run Token when reading from and writing to datastores on remote runs. Previously, if the AML Run Token is updated in Python, the Data Prep runtime will not be updated with the updated AML Run Token.
• Additional clearer error messages
• to_spark_dataframe() will no longer crash when Spark uses Kryo serialization
• Value Count Inspector can now show more than 1000 unique values
• Random Split no longer fails if the original Dataflow doesn’t have a name

2018-11-19 (version 0.5.0)

New features

• Created a new DataPrep CLI to execute DataPrep packages and view the data profile for a dataset or dataflow
• Redesigned SetColumnType API to improve usability
• Renamed smart_read_file to auto_read_file
• Now includes skew and kurtosis in the Data Profile
• Can sample with stratified sampling
• Can read from zip files that contain CSV files
• Can split datasets row-wise with Random Split (e.g. into test-train sets)
• Can get all the column data types from a dataflow or a data profile by calling .dtypes
• Can get the row count from a dataflow or a data profile by calling .row_count

Bug fixes

• Fixed long to double conversion
• Fixed assert after any add column
• Fixed an issue with FuzzyGrouping, where it would not detect groups in some cases
• Fixed sort function to respect multi-column sort order
• Fixed and/or expressions to be similar to how Pandas handles them
• Fixed reading from dbfs path.
• Made error messages more understandable
• Now no longer fails when reading on remote compute target using AML token
• Now no longer fails on Linux DSVM
• Now no longer crashes when non-string values are in string predicates
• Now handles assertion errors when Dataflow should fail correctly
• Now supports dbutils mounted storage locations on Azure Databricks

2018-11-05 (version 0.4.0)

New features

• Type Count added to Data Profile
• Value Count and Histogram is now available
• More percentiles in Data Profile
• The Median is available in Summarize
• Python 3.7 is now supported
• When you save a dataflow that contains datastores to a Data Prep package, the datastore information will be persisted as part of the Data Prep package
• Writing to datastore is now supported

Bug fixes

• 64bit unsigned integer overflows are now handled properly on Linux
• Fixed incorrect text label for plain text files in smart_read
• String column type now shows up in metrics view
• Type count now is fixed to show ValueKinds mapped to single FieldType instead of individual ones
• Write_to_csv no longer fails when path is provided as a string
• When using Replace, leaving “find” blank will no longer fail

Datasets License Information

IMPORTANT: Please read the notice and find out more about this NYC Taxi and Limousine Commission dataset here: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml

IMPORTANT: Please read the notice and find out more about this Chicago Police Department dataset here: https://catalog.data.gov/dataset/crimes-2001-to-present-398a4