Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

let fpgrowth and fpmax work directly with sparse input #622

Merged
merged 3 commits into from
Nov 6, 2019
Merged

let fpgrowth and fpmax work directly with sparse input #622

merged 3 commits into from
Nov 6, 2019

Commits on Nov 6, 2019

  1. let fpgrowth and fpmax work directly with sparse input 

    If transactions are stored in a sparse matrix, it was first converted
into a dense Numpy array in setup_fptree before processing.

This commit allows to build the FPTree directly from sparse input, which
may save memory and processing time.  This is used by both fpgrowth and
fpmax.  There is no change after FPTree is being built.
    @dbarbier
    dbarbier committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    a212806 View commit details
    Browse the repository at this point in the history

  2. in setup_fptree, ensure that sparse DataFrame has no zeroes 

    In a212806 we assumed that all values in sparse DataFrame are non null,
which should always be true.  In order to avoid this corner case, we
now call itemsets.eliminate_zeros().  The alternative would be to replace
   nonnull = itemsets.indices[itemsets.indptr[i]:itemsets.indptr[i+1]]
by
   values = itemsets.data[itemsets.indptr[i]:itemsets.indptr[i+1]]
   nonnull = itemsets.indices[itemsets.indptr[i] + values.nonzero()[0]]
but it is slower.

Add a test case.
    @dbarbier
    dbarbier committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    7948c1f View commit details
    Browse the repository at this point in the history

  3. fix pep8 issue

    @dbarbier
    dbarbier committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    573882f View commit details
    Browse the repository at this point in the history