Skip to content
develop
Go to file
Code

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md


Documentation | Forum | Mail List

DataPrep lets you prepare your data using a single library with a few lines of code.

Currently, you can use DataPrep to:

  • Collect data from common data sources (through dataprep.connector)
  • Do your exploratory data analysis (through dataprep.eda)
  • ...more modules are coming

Releases

Repo Version Downloads
PyPI
conda-forge

Installation

pip install -U dataprep

Examples & Usages

The following examples can give you an impression of what DataPrep can do:

Connector

Connector is an intuitive, open-source API wrapper that speeds up development by standardizing calls to multiple APIs as a simple workflow.

Connector provides a simple wrapper to collect structured data from different Web APIs (e.g., Twitter API, Yelp Fusion API, Spotify API, DBLP API), making web data collection easy and efficient, without requiring advanced programming skills.

Do you want to leverage the growing number of websites that are opening their data through public APIs? Connector is for you!

Let's check out the several benefits that Connector offers:

  • A unified API: You can fetch data using one or two lines of code to get data from many websites.

      

  • Auto Pagination: Do you want to invoke a Web API that could return a large result set and need to handle it through pagination? Connector automatically does the pagination for you! Just specify the desired number of returned results (argument _count) without getting into unnecessary detail about a specific pagination scheme.

      

  • Smart API request strategy: Do you want to fetch results more quickly by making concurrent requests to Web APIs? Through the _concurrency argument, Connector simplifies concurrency, issuing API requests in parallel while respecting the API's rate limit policy.

      

In configuration files, Connector specifies how to connect with each Web API for data gathering. If you want to connect with any of the APIs mentioned in the table below, with one line of code, you can get the most up-to-date version of the config file from our codebase and use it right away!

Many websites in different domains are currently supported. These are some examples:

Category Web API Auth Method Connector Config File(s) Jupyter Notebook / Tutorial Description
Social Media Twitter OAuth2 Twitter config file(s) Twitter Jupyter Notebook API endpoint for Tweets information retrieval.
Music Spotify OAuth2 Spotify config file(s) Spotify tutorial Comprehensive API for retrieving albums, artists, and tracks metadata.
Restaurants Yelp Bearer Token Yelp config file(s) Yelp Jupyter Notebook Leading API to access restaurant information by location.
Science DBLP No DBLP config file(s) DBLP Jupyter Notebook Open bibliographic API for computer science publications.
Social Media Youtube API Key Youtube config file(s) Youtube Jupyter Notebook API for retrieving Youtube's content information.
Finance Finnhub API Key Finnhub config file(s) Finnhub Jupyter Notebook Comprehensive API for financial, market, and economic data.
Music Musixmatch API Key Musixmatch config file(s) Coming soon Leading API for searching music lyrics.
Weather OpenWeatherMap API Key OpenWeatherMap config file(s) Coming soon API for retrieving current and historical weather data.
Lifestyle Spoonacular API Key Spoonacular config file(s) Coming soon Recipe, food, and nutritional information API.

If you want to connect with a different web API, Connector is designed to be easy to extend. You just have to write a simple configuration file to support the new web API. This configuration file describes the API's main attributes like the URL, query parameters, authorization method, pagination properties, etc.

In the following link, you can see detailed examples of how to use Connector for retrieving data from DBLP, Spotify, Yelp, and other sites, without taking an in-depth look into the web APIs documentation!: Examples.

EDA

There are common tasks during the exploratory data analysis stage, like a quick look at the columnar distribution, or understanding the correlations between columns.

The EDA (Exploratory Data Analysis) module categorizes these EDA tasks into functions helping you finish EDA tasks with a single function call.

  • Want to understand the distributions for each DataFrame column? Use plot.

  • Want to understand the correlation between columns? Use plot_correlation.

  • Or, if you want to understand the impact of the missing values for each column, use plot_missing.

You can drill down to get more information by given plot, plot_correlation and plot_missing a column name.: E.g. for plot_missing

    for numerical column usingplot:

    for categorical column usingplot:

Don't forget to checkout the examples folder for detailed demonstration!

Clean

DataPrep.Clean contains simple functions designed for cleaning and standardizing a column in a DataFrame. It provides

  • A unified API: each function follows the syntax clean_{type}(df, "column name") (see an example below)
  • Python Data Science Support: its design for cleaning pandas and Dask DataFrames enables seamless integration into the Python data science workflow
  • Transparency: a report is generated that summarizes the alterations to the data that occured during cleaning

The following example shows how to clean a column containing messy emails:

Type validation is also supported:

Below are the supported semantic types (more are currently being developed).

Semantic Types
longitude/latitude
country
email
url
phone

For more information, refer to the User Guide.

Contribute

There are many ways to contribute to DataPrep.

  • Submit bugs and help us verify fixes as they are checked in.
  • Review the source code changes.
  • Engage with other DataPrep users and developers on StackOverflow.
  • Help each other in the DataPrep Community Discord and Mail list & Forum.
  • Twitter
  • Contribute bug fixes.
  • Providing use cases and writing down your user experience.

Please take a look at our wiki for development documentations!

Acknowledgement

Some functionalities of DataPrep are inspired by the following packages.

  • Pandas Profiling

    Inspired the report functionality and insights provided in dataprep.eda.

  • missingno

    Inspired the missing value analysis in dataprep.eda.

You can’t perform that action at this time.