Skip to content
Easy pipelines for pandas DataFrames.
Python Jupyter Notebook Shell
Branch: master
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github tfidf vectorizer fully tested and documented Jan 4, 2020
doc more fixes induced by codefactor shell script checks Dec 25, 2019
pdpipe minor prints changes + nicer labels for tfidf vectorizer Jan 13, 2020
tests minor prints changes + nicer labels for tfidf vectorizer Jan 13, 2020
.codecov.yml setting up nice test coverage Mar 16, 2017
.coveragerc dropna and freqdrop added and tested Jan 14, 2018
.gitattributes some basic stages done Feb 16, 2017
.gitignore doc branch looks good Dec 20, 2019
.shellcheckrc added .shellcheckrc to configure codefactor shell script checks Dec 25, 2019
.travis.yml joined linting job to main testing stage Jan 7, 2020
LICENSE initial commit Jan 24, 2017 some basic stages done Feb 16, 2017
README.rst readme fix: install for development Jan 8, 2020
mit_license_badge.svg Add files via upload Apr 17, 2018
pytest.ini a bunch of nice readme updates for contributords Dec 23, 2019
setup.cfg doc branch looks good Dec 20, 2019 updating url params for Dec 21, 2019 initial commit Jan 24, 2017


pdpipe ˨

PyPI-Status PePy stats PyPI-Versions Build-Status Codecov Codefactor code quality LICENCE

Easy pipelines for pandas DataFrames (learn how!).



>>> df = pd.DataFrame(
        data=[[4, 165, 'USA'], [2, 180, 'UK'], [2, 170, 'Greece']],
        index=['Dana', 'Jane', 'Nick'],
        columns=['Medals', 'Height', 'Born']
>>> import pdpipe as pdp
>>> pipeline = pdp.ColDrop('Medals').OneHotEncode('Born')
>>> pipeline(df)
            Height  Born_UK  Born_USA
    Dana     165        0         1
    Jane     180        1         0
    Nick     170        0         0

1   Documentation

This is the repository of the pdpipe package, and this readme file is aimed to help potential contributors to the project.

To learn more about how to use pdpipe, either visit pdpipe's homepage or read the online documentation of pdpipe.

2   Installation

Install pdpipe with:

pip install pdpipe

Some pipeline stages require scikit-learn; they will simply not be loaded if scikit-learn is not found on the system, and pdpipe will issue a warning. To use them you must also install scikit-learn.

Similarly, some pipeline stages require nltk; they will simply not be loaded if nltk is not found on your system, and pdpipe will issue a warning. To use them you must additionally install nltk.

3   Contributing

Package author and current maintainer is Shay Palachy (; You are more than welcome to approach him for help. Contributions are very welcomed, especially since this package is very much in its infancy and many other pipeline stages can be added.

3.1   Installing for development


git clone

Install in development mode with test dependencies:

cd pdpipe
pip install -e ".[test]"

3.2   Running the tests

To run the tests, use:

python -m pytest

Notice pytest runs are configured by the pytest.ini file. Read it to understand the exact pytest arguments used.

3.3   Adding tests

At the time of writing, pdpipe is maintained with a test coverage of 100%. Although challenging, I hope to maintain this status. If you add code to the package, please make sure you thoroughly test it. Codecov automatically reports changes in coverage on each PR, and so PR reducing test coverage will not be examined before that is fixed.

Tests reside under the tests directory in the root of the repository. Each model has a separate test folder, with each class - usually a pipeline stage - having a dedicated file (always starting with the string "test") containing several tests (each a global function starting with the string "test"). Please adhere to this structure, and try to separate tests cases to different test functions; this allows us to quickly focus on problem areas and use cases. Thank you! :)

3.4   Code style

pdpip code is written to adhere to the coding style dictated by flake8. Practically, this means that one of the jobs that runs on the project's Travis for each commit and pull request checks for a successfull run of the flake8 CLI command in the repository's root. Which means pull requests will be flagged red by the Travis bot if non-flake8-compliant code was added.

To solve this, please run flake8 on your code (whether through your text editor/IDE or using the command line) and fix all resulting errors. Thank you! :)

3.5   Adding documentation

This project is documented using the numpy docstring conventions, which were chosen as they are perhaps the most widely-spread conventions that are both supported by common tools such as Sphinx and result in human-readable docstrings (in my personal opinion, of course). When documenting code you add to this project, please follow these conventions.

Additionally, if you update this README.rst file, use python checkdocs to validate it compiles.

3.6   Adding doctests

Please notice that for pdoc3 - the Python package used to generate the html documentation files for pdpipe - to successfully include doctests in the generated documentation files, the whole doctest must be indented in relation to the opening multi-string indentation, like so:

class ApplyByCols(PdPipelineStage):
    """A pipeline stage applying an element-wise function to columns.

    columns : str or list-like
        Names of columns on which to apply the given function.
    func : function
        The function to be applied to each element of the given columns.
    result_columns : str or list-like, default None
        The names of the new columns resulting from the mapping operation. Must
        be of the same length as columns. If None, behavior depends on the
        drop parameter: If drop is True, the name of the source column is used;
        otherwise, the name of the source column is used with the suffix
    drop : bool, default True
        If set to True, source columns are dropped after being mapped.
    func_desc : str, default None
        A function description of the given function; e.g. 'normalizing revenue
        by company size'. A default description is used if None is given.

        >>> import pandas as pd; import pdpipe as pdp; import math;
        >>> data = [[3.2, "acd"], [7.2, "alk"], [12.1, "alk"]]
        >>> df = pd.DataFrame(data, [1,2,3], ["ph","lbl"])
        >>> round_ph = pdp.ApplyByCols("ph", math.ceil)
        >>> round_ph(df)
           ph  lbl
        1   4  acd
        2   8  alk
        3  13  alk

4   Credits

Created by Shay Palachy (

You can’t perform that action at this time.