-
Notifications
You must be signed in to change notification settings - Fork 200
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(clean): add user guide for clean_duplication
Added the user guide for the clean_duplication function.
- Loading branch information
Showing
4 changed files
with
218 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,187 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "raw", | ||
| "metadata": { | ||
| "raw_mimetype": "text/restructuredtext" | ||
| }, | ||
| "source": [ | ||
| ".. _duplication_userguide:\n", | ||
| "\n", | ||
| "Duplicate Values\n", | ||
| "================" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "raw", | ||
| "metadata": { | ||
| "raw_mimetype": "text/restructuredtext", | ||
| "scrolled": false | ||
| }, | ||
| "source": [ | ||
| "Introduction\n", | ||
| "------------\n", | ||
| "\n", | ||
| "The function :func:`clean_duplication() <dataprep.clean.clean_duplication.clean_duplication>` creates a user interface that clusters duplicate values and allows the user to merge them into standardized values. The following clustering methods are provided:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### fingerprint\n", | ||
| "This is the process for creating a fingerprint key:\n", | ||
| "\n", | ||
| "* remove leading and trailing whitespace\n", | ||
| "* convert to lowercase\n", | ||
| "* remove punctuation and control characters\n", | ||
| "* normalize extended western characters to ASCII\n", | ||
| "* split into whitespace separated tokens\n", | ||
| "* sort tokens and remove duplicates\n", | ||
| "* join tokens back together\n", | ||
| "\n", | ||
| "### ngram-fingerprint\n", | ||
| "This is the process for creating a n-gram fingerprint key:\n", | ||
| "\n", | ||
| "* convert to lowercase\n", | ||
| "* remove punctuation, whitespace and control characters\n", | ||
| "* get string n-grams\n", | ||
| "* sort n-grams and remove duplicates\n", | ||
| "* join sorted n grams back together\n", | ||
| "* normalize extended western characters to ASCII\n", | ||
| "\n", | ||
| "A textbox is provided for choosing the n-gram size.\n", | ||
| "\n", | ||
| "### phonetic-fingerprint\n", | ||
| "Uses the double metaphone algorithm for generating phonetic-fingerprint keys. The [metaphone](https://github.com/oubiwann/metaphone) library is used.\n", | ||
| "\n", | ||
| "### levenshtein distance\n", | ||
| "Blocking is used to speed up the process, blocks are obtained where strings in the same\n", | ||
| "block share a substring of a given blocking size. Only strings within the same block are\n", | ||
| "compared using the levenshtein distance function. If two values have a distance less than \n", | ||
| "or equal to the given radius they are added to the same cluster. Textboxes are provided for choosing the block size and the radius.\n", | ||
| "\n", | ||
| "The [python-Levenshtein](https://github.com/ztane/python-Levenshtein) library is used for a fast levenshtein distance implementation.\n", | ||
| "\n", | ||
| "Clustering methods are taken from the [OpenRefine](https://github.com/OpenRefine/OpenRefine) project and the [simile-vicino](https://code.google.com/archive/p/simile-vicino/n) project, you can read more about these clustering methods [here](https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth).\n", | ||
| "\n", | ||
| "The `df_var_name` parameter sets the variable name to be used for the dataframe when creating replace function calls.\n", | ||
| "\n", | ||
| "The `page_size` parameter can be used to set the number of clusters that are displayed on each page of the user interface.\n", | ||
| "\n", | ||
| "Most of the functionality is provided through an interactive user interface which will be introduced shortly." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## An example dirty dataset" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import pandas as pd\n", | ||
| "import numpy as np\n", | ||
| "df = pd.DataFrame(\n", | ||
| " {\n", | ||
| " \"city\": [\n", | ||
| " \"Québec\",\n", | ||
| " \"Quebec\",\n", | ||
| " \"Vancouver\",\n", | ||
| " \"Vancouver\",\n", | ||
| " \"vancouver\",\n", | ||
| " \" Vancuver \",\n", | ||
| " \"Toronto\",\n", | ||
| " \"Toront\",\n", | ||
| " \"Tronto\",\n", | ||
| " \"Ottowa\",\n", | ||
| " \"otowa\"\n", | ||
| " ]\n", | ||
| " }\n", | ||
| ")\n", | ||
| "df" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 1. Default `clean_duplication()`\n", | ||
| "\n", | ||
| "By default the `df_var_name` parameter is set to \"df\" and the `page_size` variable is set to 5. Clustering methods can be toggled using the dropdown menu at the top of the GUI. Select which clusters you would like to merge using the checkboxes under the \"Merge?\" heading. Then press the \"Merge and Re-Cluster\" button to merge the cluster. If the \"export code\" checkbox is selected, code for merging the clusters will be created and added to the notebook cell. Finally, you can press the \"finish\" button to close the GUI and see the final DataFrame created." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from dataprep.clean import clean_duplication\n", | ||
| "clean_duplication(df, \"city\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 2. `df_var_name` parameter\n", | ||
| "\n", | ||
| "Pandas Series.replace function calls are created and added to the current notebook cell when merging a cluster with the \"export code\" checkbox selected. This parameter allows for changing the DataFrame variable name used in the exported code." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "clean_duplication(df, \"city\", df_var_name=\"dataframe\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 3. `page_size` parameter\n", | ||
| "This parameter allows for changing the number of clusters that are displayed on each page of the user interface." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "clean_duplication(df, \"city\", page_size=1)" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "celltoolbar": "Raw Cell Format", | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.7.3" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters