Updating SQL-Engine branch to main branch, Adding Heatmap Functionali…

…ty to SQL Executor (#154)

* Merging Recent SQL Executor changes

* Fix to Validator

Uses unique value metadata to verify if a value is valid

* Fix Bug with Widget Rendering

frame.py was trying to import luxWidget instead of luxwidget

* Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor

Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor

Added tests for SQL executor

* Re-added Licensing Headers

* Adding Recent frame.py changes

* Adjusted SQL Executor Tests

Removed lines that changed Year column type to datetime

* Update Frame with new Action Registering

* Resolving Conflicts in frame.py

* Commenting out local SQL Executor tests

SQL Executor tests interfering with travis build, commenting out for now

* Update correlation.py

* Update frame.py

* bugfix: "number of remaining bars" text overcounts for colored bar charts
* update number of bars calculation to account for when len(data) double counts

* Fixing Code Format

* Cleaning up Pandas Executor imports

* Fix Validation Bug

Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py

* Changed metadata variable name

Renamed num_obs to length, removed ordinal variable from Executor mapping function

* Adding support for setting intent on front end (#112)

* added functionality to delete Vis

* fixed deletion logic

* add observer to automatically update deletions

* able to refresh widget on setting intent

* support for setting intent from frontend

* quick fix to output

* changed variable intentindex name

* Make default_display a global setting (#121)

* remove and register action functions

* update changes inframe.py

* update changes inframe.py

* add documentation and changes

* indentation and comments

* new line

* globally defined default display works with warning

* no examples

* add back space

* new line

* uncomment docstring

Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu>

* Added script to generate Postgresql database

Updated travis.yml file to create postgresql database in test instance.

Added script to populate test database with data.

* Update upload_car_data.py

Updated database credentials

* Updated script name in travis.yml

* Removed unnecessary import from travis.yml

* Added psycopg2 to requirements.txt

* Creating Postgres test database in travis

* Fixed directory issue

* Updated SQL Executor Tests

Added tests for basic SQL Executor functionality.

* Update requirements.txt (#128)

* basic scatterplot experiments

* experiment results with manually binned heatmaps

* experiment result

* incorporated heatmap code into executor and renderer

* additional experiments to evaluate scatter v.s. heatmap performance

* experiment based on real estate and airbnb data

* modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93)

* decrease sampling parameter

* change sampling strategy (above threshold keep 3/4 of data)

* remove experiment dir

* modified performance param

* enforce lux-widget minimum version

* update requirement.txt

* separate dev and install requirements

* replaced _exportedVisIdxs --> _selectedVisIdxs

* bugfix: plot config error when current_vis is None

* Added sql_executor example notebook, minor bug fix

Added an example notebook to showcase how to use the sql-engine.

Fixed variable reference in interestingness.py that was causing issues.

* Add LuxSeries Implementation (#122)

* add preliminary groupby fixes

* preliminary LuxSeries implementation

* add tests for new Series implementation

* clean up the added code

* minor code changes

* fix issues with Vis with index

* small fixes

* remove comments

* bugfix column group display empty Vis involving groupby index

* bugfix Cylinders not showing up as bar charts

Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* add black to travis (#127)

* add black to travis

* reformat all code and adjust test

* remove .idea

* fix contributing doc

* small change in contributing

* update

* reformat, update command to fix version

* remove dev dependencies

* fix doc failing from black format

* Cleaned SQL Executor Example Notebook

restarted kernel and cleared output

* Update custom action reference to executor

Now uses executor tied to the dataframe for execution

* Updated Interestingness Tests (#133)

* add black

* update cars dataset and tests

* Delete old dataset

* Updated Interestingness Tests

Updated tests to use the newly updated cars dataset

* switch to local cars reference

Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu>
Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com>
Co-authored-by: 19thyneb <thyne.boonmark@gmail.com>
Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* fix broken link in docs

* Updated Tests, Added benchmarking for SQL Executor

Updated Compiler and Interestingness tests to work for SQL executor.

Updated SQL Executor to have some benchmarking code for tracking query performance.

* Merge with upstream branch, added preliminary benchmarking code

* Better warning message for Vis and VisList (#135)

* added functionality to delete Vis

* fixed deletion logic

* add observer to automatically update deletions

* able to refresh widget on setting intent

* support for setting intent from frontend

* quick fix to output

* changed variable intentindex name

* added better error msg for > 1 intent for vis

* reverting some changes

* adding warning message for Vis intents being > 1

* passes tests and intent < 3

* minor change to error message, added test

* run black

* accounted for more edge cases and hid traceback

* fixed typo

* added tests

* format w/ black

* ran black again

* Update Vis.py

minor readability changes

Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* Pandas rewrite Performance optimizations  (#136)

* basic scatterplot experiments

* experiment results with manually binned heatmaps

* experiment result

* incorporated heatmap code into executor and renderer

* additional experiments to evaluate scatter v.s. heatmap performance

* experiment based on real estate and airbnb data

* modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93)

* decrease sampling parameter

* change sampling strategy (above threshold keep 3/4 of data)

* remove experiment dir

* modified performance param

* enforce lux-widget minimum version

* update requirement.txt

* testing out modin (Recursion error)

* create modin executor, all else in sync with master changes

* rewrote .loc with column reference, speed up by 100x

* replace agg("count") with .count() --> ~0.1ms speedup

* run black

* Added 2D Binning functionality to SQL Executor

added 2D binning to replace scatterplots when using SQL executor.

* Update README.md

update slack link

* Updated temporal detection and tests (#139)

* Updated temporal detection and tests

* Reformatted code with black

* Update PandasExecutor.py

* added stock date test

Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* Fix Inline comments breaking to new lines (#137)

* add black to travis

* reformat all code and adjust test

* remove .idea

* fix contributing doc

* small change in contributing

* update

* reformat, update command to fix version

* remove dev dependencies

* first pass -- inline comments

* _config/config.py

* delete test notebook

* action

* line length 105

* executor

* interestingness

* processor

* vislib

* tests, travis, CONTRIBUTING

* .format
() changed

* replace tabs with escape chars

* update using black

* more rewrites and merges into single line

Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* Improve warning message when values specified as attributes (#143)

* Improve warning message when values specified as attributes (#142)

* added test, ran black

* bugfix test

* Better warning message for Vis and VisList (#146)

* added functionality to delete Vis

* fixed deletion logic

* add observer to automatically update deletions

* able to refresh widget on setting intent

* support for setting intent from frontend

* quick fix to output

* changed variable intentindex name

* added better error msg for > 1 intent for vis

* reverting some changes

* adding warning message for Vis intents being > 1

* passes tests and intent < 3

* minor change to error message, added test

* run black

* accounted for more edge cases and hid traceback

* fixed typo

* added tests

* format w/ black

* ran black again

* Update Vis.py

minor readability changes

* added check and tests for Vis list and | syntax

* ran black

Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* Updated docs for JupyterLab (#148)

* added functionality to delete Vis

* fixed deletion logic

* add observer to automatically update deletions

* able to refresh widget on setting intent

* support for setting intent from frontend

* quick fix to output

* changed variable intentindex name

* added better error msg for > 1 intent for vis

* reverting some changes

* updated install and faq

* added install_lab.sh script

* Global shared variable in test (#144) (#149)

* using global shared variable in test (#144)

* modified fixture scope as session, resolved dependency test cases

* run black

Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>

* Updated 2D Binning Functionality

Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer.

* Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor

Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor

Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables

* Updated Code Formatting with Black

* Update Requirements to include psycopg2

* Update upload_car_data.py

Updated to use newest car.csv file

* Update Compiler tests to use correct test DB

* Removed Benchmarking Code

* Fixing Black Formatting

Co-authored-by: 19thyneb <thyne.boonmark@gmail.com>
Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com>
Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com>
Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com>
Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu>
Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com>
Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com>
Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu>
Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu>
Co-authored-by: Piyush Gupta <piyushg9794@gmail.com>

.gitignore

@@ -10,5 +10,6 @@ static/
 *.egg-info*
 build/
 .DS_Store
+.idea/
 tests/.coverage
 tests/coverage.xml

.travis.yml

@@ -4,8 +4,8 @@ python:
 services:
   - postgresql
 install:
-  - pip install codecov
   - pip install -r requirements.txt
+  - pip install -r requirements-dev.txt
   - pip install git+https://github.com/lux-org/lux-widget
   #- npm i lux-widget
 before_script:
@@ -15,9 +15,10 @@ before_script:
    - psql -c "DROP schema public cascade;" -U postgres
    - psql -c "CREATE schema public;" -U postgres
    - psql -c "CREATE DATABASE postgres_db;" -U postgres
-# command to run tests
+# command to upload data to test environment SQL database and run tests
 script:
   - python lux/data/upload_car_data.py
+  - black --target-version py37 --line-length 105 --check .
   - python -m pytest tests/*.py
   - pytest --cov-report term --cov=lux tests/
 after_success:

CONTRIBUTING.md

@@ -45,9 +45,12 @@ You can run them locally to make sure that your changes are working and do not b
 python -m pytest tests/*.py
 ```
 
+# Code Formatting
+In order to keep our codebase clean and readible, we are using PEP8 guidelines. To help us maintain and check code style, we are using [black](https://github.com/psf/black). Simply run `black --line-length 105 .` before commiting. Failure to do so may fail the tests run on Travis. This package should have been installed for you. 
+
 # Submitting a Pull Request
 
-You can commit your code and push to your forked repo. Once all of your local changes have been tested and is working, you are ready to submit a PR. For Lux, we use the "Squash and Merge" strategy to merge in PR, which means that even if you make a lot of small commits in your PR, they will all get squashed into a single commit associated with the PR. Please make sure that comments and unnecessary file changes are not committed as part of the PR by looking at the "File Changes" diff view on the pull request page.
+ You can commit your code and push to your forked repo. Once all of your local changes have been tested and formatted, you are ready to submit a PR. For Lux, we use the "Squash and Merge" strategy to merge in PR, which means that even if you make a lot of small commits in your PR, they will all get squashed into a single commit associated with the PR. Please make sure that comments and unnecessary file changes are not committed as part of the PR by looking at the "File Changes" diff view on the pull request page.
 
 Once the pull request is submitted, the maintainer will get notified and review your pull request. They may ask for additional changes or comment on the PR. You can always make updates to your pull request after submitting it.
 

README.md

@@ -149,5 +149,5 @@ Other additional resources:
 
 - Visit [ReadTheDoc](https://lux-api.readthedocs.io/en/latest/) for more detailed documentation.
 - Clone [lux-binder](https://github.com/lux-org/lux-binder) to try out these [hands-on exercises](https://github.com/lux-org/lux-binder/tree/master/exercise) or a more comprehensive [tutorial series](https://github.com/lux-org/lux-binder/tree/master/tutorial) on how to use Lux.
-- Join our [Slack channel](http://lux-project.slack.com/) for support and discussion.
+- Join our [Slack channel](https://lux-project.slack.com/join/shared_invite/zt-iwg84wfb-fBPaGTBBZfkb9arziy3W~g) for support and discussion.
 - Report any bugs, issues, or requests through [Github Issues](https://github.com/lux-org/lux/issues). 

doc/conf.py

@@ -1,5 +1,5 @@
 #  Copyright 2019-2020 The Lux Authors.
-# 
+#
 #  Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 #  You may obtain a copy of the License at
@@ -19,7 +19,8 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html
 
 import subprocess
-subprocess.call(['sh', './docbuild.sh'])
+
+subprocess.call(["sh", "./docbuild.sh"])
 # -- Path setup --------------------------------------------------------------
 
 # If extensions (or modules to document with autodoc) are in another directory,
@@ -28,18 +29,19 @@
 #
 import os
 import sys
-sys.path.insert(0, os.path.abspath('..'))
-sys.path.insert(0, os.path.abspath('..'))
+
+sys.path.insert(0, os.path.abspath(".."))
+sys.path.insert(0, os.path.abspath(".."))
 
 
 # -- Project information -----------------------------------------------------
 
-project = 'Lux'
-copyright = '2020, Doris Jung-Lin Lee'
-author = 'Doris Jung-Lin Lee'
+project = "Lux"
+copyright = "2020, Doris Jung-Lin Lee"
+author = "Doris Jung-Lin Lee"
 
 # The full version, including alpha/beta/rc tags
-release = '0.1.2'
+release = "0.1.2"
 
 
 # -- General configuration ---------------------------------------------------
@@ -48,32 +50,32 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-	'sphinx.ext.autodoc',
-	'sphinx.ext.coverage', 
-	'sphinx.ext.autosummary',
-	'sphinx.ext.doctest',
-	'sphinx.ext.githubpages',
-	'sphinx.ext.intersphinx',
-	'sphinx.ext.viewcode',
-    'sphinx.ext.napoleon',
-	'sphinx.ext.mathjax',
-	'sphinx_automodapi.automodapi', 
-	'sphinx_automodapi.automodsumm'
+    "sphinx.ext.autodoc",
+    "sphinx.ext.coverage",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.doctest",
+    "sphinx.ext.githubpages",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.mathjax",
+    "sphinx_automodapi.automodapi",
+    "sphinx_automodapi.automodsumm",
 ]
 
-autodoc_default_flags = ['members', "inherited-members"]
+autodoc_default_flags = ["members", "inherited-members"]
 autodoc_member_order = "groupwise"
 autosummary_generate = True
 numpydoc_show_class_members = False
 
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
 
 # -- Options for HTML output -------------------------------------------------
@@ -88,7 +90,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ["_static"]
 html_logo = "_static/logo.png"
 
 html_theme_options = {"style_nav_header_background": "#19177c"}
@@ -97,4 +99,4 @@
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-master_doc = 'index'
+master_doc = "index"

doc/source/getting_started/installation.rst

@@ -11,11 +11,19 @@ You can install the Python Lux API through `PyPI <https://pypi.org/project/lux-a
 
     pip install lux-api
 
-You can install the Lux Jupyter widget through `npm <https://www.npmjs.com/package/lux-widget>`_
+To activate the Jupyter notebook extension: 
 
 .. code-block:: bash
 
-    npm i lux-widget
+    jupyter nbextension install --py luxwidget
+    jupyter nbextension enable --py luxwidget
+
+To activate the JupyterLab extension: 
+
+.. code-block:: bash
+
+    jupyter labextension install @jupyter-widgets/jupyterlab-manager
+    jupyter lab build
 
 Additional Requirements
 -----------------------
@@ -56,6 +64,7 @@ Then, we can install the `Lux Jupyter widget <https://github.com/lux-org/lux-wid
     git clone git@github.com:lux-org/lux-widget.git
     cd lux-widget/
     npm install
-    bash install.sh
+    sh install.sh
+    sh install_lab.sh
 
 If you are experiencing issues with installing Lux, please checkout the `Troubleshooting page <https://lux-api.readthedocs.io/en/latest/source/guide/FAQ.html#troubleshooting-tips>`_.

doc/source/getting_started/overview.rst

@@ -95,7 +95,7 @@ When you print out the dataframe again, you should see three different tabs of v
 
     df
 
-.. image:: ../../../../lux-resources/doc_img/overview-3.gif
+.. image:: https://github.com/lux-org/lux-resources/blob/master/doc_img/overview-3.gif?raw=true
   :width: 700
   :align: center
   :alt: scroll through Enhance, click on Filter tab
@@ -129,7 +129,7 @@ Given the updated intent, additional actions (Enhance and Filter) are generated.
     - {MedianEarnings, **AverageCost**}
     - {MedianEarnings, **AverageFacultySalary**}.
 
-.. image:: ../img/overview-4.png
+.. image:: https://github.com/lux-org/lux-resources/blob/master/doc_img/overview-4.png
   :width: 700
   :align: center
   :alt: screenshot of Enhance
@@ -140,7 +140,7 @@ Given the updated intent, additional actions (Enhance and Filter) are generated.
     - {MedianEarnings, **Region=Southeast**}
     - {MedianEarnings, **Region=Great Lakes**}.
 
-.. image:: ../img/overview-5.png
+.. image:: https://github.com/lux-org/lux-resources/blob/master/doc_img/overview-5.png
   :width: 700
   :align: center
   :alt: screenshot of Filter

doc/source/guide/FAQ.rst

@@ -80,7 +80,6 @@ The Lux Jupyter widget does not show up when I print a dataframe.
 
   - If you are able to import lux successfully and you do not see the "Toggle button" when you print the dataframe, it may be possible that Lux is not compatible with your browser. Lux is compatible with Google Chrome, but have not been extensively tested on Safari or Firefox.
   - If you recieve the error message :code:`A Jupyter widget could not be displayed because the widget state could not be found.` This could happen if the kernel storing the widget is no longer available, or if the widget state was not saved in the notebook. You may be able to create the widget by running the appropriate cells.`, you may want to restart the notebook and rerun the cell.
-  - If you recieve the error message :code:`Error displaying widget: model not found`, it is possible that you are using JupyterLab or other frontends (e.g., VSCode) to open up your notebook. Lux does not currently support JupyterLab (`#16 <https://github.com/lux-org/lux-widget/issues/16>`_), please try out Lux with the basic :code:`jupyter notebook`.
   - If you receive the error message :code:`ModuleNotFoundError: No module named 'luxwidget'`, it is possible that your luxwidget and lux-api versions are not in sync. The latest version of lux-api requires luxwidget v0.1 or above. Try running the following code:
 
   .. code-block:: bash

doc/source/reference/gen/lux.core.frame.LuxDataFrame.rst

@@ -155,7 +155,7 @@ lux.core.frame.LuxDataFrame
       ~LuxDataFrame.rec_to_JSON
       ~LuxDataFrame.reindex
       ~LuxDataFrame.reindex_like
-      ~LuxDataFrame.removeDeletedRecs
+      ~LuxDataFrame.remove_deleted_recs
       ~LuxDataFrame.rename
       ~LuxDataFrame.rename_axis
       ~LuxDataFrame.render_widget
@@ -180,6 +180,7 @@ lux.core.frame.LuxDataFrame
       ~LuxDataFrame.set_index
       ~LuxDataFrame.set_intent
       ~LuxDataFrame.set_intent_as_vis
+      ~LuxDataFrame.set_intent_on_click
       ~LuxDataFrame.shift
       ~LuxDataFrame.skew
       ~LuxDataFrame.slice_shift
@@ -246,7 +247,6 @@ lux.core.frame.LuxDataFrame
       ~LuxDataFrame.axes
       ~LuxDataFrame.columns
       ~LuxDataFrame.current_vis
-      ~LuxDataFrame.default_display
       ~LuxDataFrame.dtypes
       ~LuxDataFrame.empty
       ~LuxDataFrame.exported

lux/__init__.py

@@ -1,5 +1,5 @@
 #  Copyright 2019-2020 The Lux Authors.
-# 
+#
 #  Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 #  You may obtain a copy of the License at
@@ -21,5 +21,6 @@
     register_action,
     remove_action,
     actions,
-    update_actions
+    update_actions,
+    config,
 )

lux/_config/__init__.py

@@ -4,4 +4,5 @@
     remove_action,
     actions,
     update_actions,
+    config,
 )