Addition of cuDF and HoloViews for GPU acceleration

[AdityaR-Bits]

Overview

This implementation of LUX adds the option of utilizing NVIDIA GPUs, with RAPIDS cuDF and HoloViews as the plotting engine. It is capable of a speed up 3-10X compared to the original LUX, and avoids browser memory issues when dealing with datasets in the millions+ rows (measured on the NVIDIA RTX A3000 Laptop GPU).

HoloViews

HoloViews does not require the creation of a JSON file which for larger datasets is both memory and time expensive. It is able to show magnitudes of more data points on its curve without being time consuming, also removing the constraint of having to fall back to heatmaps rather than scatter plots, when the number of rows is too high. In this implementation we have not relied on the LUX widget for displaying the charts, for simplicity in viewing.

To Run

To run the cuDF + HoloViews implementation, simply do the following

#If global_backend is defined as "holoviews" then cuDF and HoloViews will be used, otherwise no need to define backend.set_back
from global_backend import backend
backend.set_back = "holoviews"
import lux
import pandas as pd
if backend.set_back == "holoviews":
    import cudf

To plot the HoloViews curves, run df.maintain_recs() rather than df in a different cell.

Example Output

A brief output is shown below

Next Steps

This implementation is a proof of concept demonstrating the acceleration that RAPIDS can bring to LUX. It also shows the benefits of adding HoloViews as an additional option for plotting. @exactlyallan, @AjayThorve and I would like to discuss if and how an integration like this might proceed, @dorisjlee?

[exactlyallan]

BTW this addresses #478 and shows really good performance potential in either adding HoloViews option OR cuDF option.

[dorisjlee]

Thanks for the contribution @AdityaR-Bits! The cuDF and Holoviews backends is certainly a valuable addition to Lux.
I will be reviewing the PR shortly. In the meanwhile, could you look into addressing the failing tests? Note that the commit messages need to be formatted based on our commit message guidelines for the linter test to pass.

[exactlyallan]

@dorisjlee, @AdityaR-Bits's project has shown that there is value in modularizing lux into components that separate the data structure, interestingness calculations, and visualization frameworks. Are there any plans (or resources) to do so?

[dorisjlee]

Hi @exactlyallan, Definitely agree that it would be a good refactoring change. We have experimented with separating out the LuxDataFrame data structure from the visualization and recommendation modules in a separate branch, but the work has not yet been merged into the main branch yet.