Skip to content

Commit

Permalink
Merge pull request #374 from thyneb19/Capstone
Browse files Browse the repository at this point in the history
Adding in General Database Executor and Code Exporting
  • Loading branch information
thyneb19 committed May 6, 2021
2 parents 3b84282 + 4c5632c commit 18c968e
Show file tree
Hide file tree
Showing 28 changed files with 2,159 additions and 341 deletions.
29 changes: 15 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,16 @@
</a>
</p>

Lux is a Python library that makes data science easier by automating aspects of the data exploration process. Lux facilitate faster experimentation with data, even when the user does not have a clear idea of what they are looking for. Visualizations are displayed via [an interactive widget](https://github.com/lux-org/lux-widget) that allow users to quickly browse through large collections of visualizations directly within their Jupyter notebooks.
Lux is a Python library that facilitate fast and easy data exploration by automating the visualization and data analysis process. By simply printing out a dataframe in a Jupyter notebook, Lux recommends a set of visualizations highlighting interesting trends and patterns in the dataset. Visualizations are displayed via [an interactive widget](https://github.com/lux-org/lux-widget) that enables users to quickly browse through large collections of visualizations and make sense of their data.

<img src="https://github.com/lux-org/lux-resources/blob/master/readme_img/demohighlight.gif?raw=true"
alt="Highlighted Visual Dataframe Workflow with Lux"
style="width:900px" />

Here is a [1-min video](https://www.youtube.com/watch?v=qmnYP-LmbNU) introducing Lux, and [slides](http://dorisjunglinlee.com/files/Zillow_07_2020_Slide.pdf) from a more extended talk.

Try out Lux on your own in a live Jupyter Notebook [here](https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/employee_demo.ipynb)!
Check out our [notebook gallery](https://lux-api.readthedocs.io/en/latest/source/reference/gallery.html) with examples of how Lux can be used with different datasets and analyses.
<br>Or try out Lux on your own in a [live Jupyter Notebook](https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/employee_demo.ipynb)!

# Getting Started

Expand All @@ -43,26 +48,22 @@ import lux
import pandas as pd
```

Then, Lux can be used as-is, without modifying any of your existing Pandas code. Here, we use Pandas's [read_csv](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) command to load in a [dataset of colleges](https://github.com/lux-org/lux-datasets/blob/master/data/college.csv) and their properties.
Lux can be used without modifying any existing Pandas code. Here, we use Pandas's [read_csv](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) command to load in a [dataset of colleges](https://github.com/lux-org/lux-datasets/blob/master/data/college.csv) and their properties.

```python
df = pd.read_csv("https://raw.githubusercontent.com/lux-org/lux-datasets/master/data/college.csv")
df
```

When the dataframe is printed out, Lux automatically recommends a set of visualizations highlighting interesting trends and patterns in the dataset.
<img src="https://github.com/lux-org/lux-resources/blob/master/readme_img/basicDemo.gif?raw=true"
alt="Basic recommendations in Lux"
style="width:900px" />

Voila! Here's a set of visualizations that you can now use to explore your dataset further!

<!-- # Features
Lux provides a suite of capabilities that enables users to effortlessly discover visual insights from their data. -->

<!-- Lux guides users to potential next-steps in their exploration. -->
### Next-step recommendations based on user intent:

In addition to dataframe visualizations at every step in the exploration, you can specify in Lux the attributes and values you're interested in. Based on this intent, Lux guides users towards potential next-steps in their exploration.
In addition to dataframe visualizations at every step in the exploration, you can specify to Lux the attributes and values you're interested in. Based on this intent, Lux guides users towards potential next-steps in their exploration.

For example, we might be interested in the attributes `AverageCost` and `SATAverage`.

Expand Down Expand Up @@ -93,7 +94,7 @@ df

### Easy programmatic access and export of visualizations:

Now that we have found some interesting visualizations through Lux, we might be interested in digging into these visualizations a bit more or sharing it with others. We can save the visualizations generated in Lux as a [static, shareable HTML](https://lux-api.readthedocs.io/en/latest/source/guide/export.html#exporting-widget-visualizations-as-static-html) or programmatically access these visualizations further in Jupyter. Selected `Vis` objects can be translated into [Altair](http://altair-viz.github.io/) or [Vega-Lite](https://vega.github.io/vega-lite/) code, so that they can be further edited.
Now that we have found some interesting visualizations through Lux, we might be interested in digging into these visualizations a bit more or sharing it with others. We can save the visualizations generated in Lux as a [static, shareable HTML](https://lux-api.readthedocs.io/en/latest/source/guide/export.html#exporting-widget-visualizations-as-static-html) or programmatically access these visualizations further in Jupyter. Selected `Vis` objects can be translated into [Altair](http://altair-viz.github.io/), [Matplotlib](https://matplotlib.org/), or [Vega-Lite](https://vega.github.io/vega-lite/) code, so that they can be further edited.

<img src="https://github.com/lux-org/lux-resources/blob/master/readme_img/export.gif?raw=true"
alt="Easily exportable visualization object"
Expand All @@ -117,7 +118,7 @@ Vis(["Region=New England","MedianEarnings"],df)

Lux provides a powerful abstraction for working with collections of visualizations based on a partially specified queries. Users can provide a list or a wildcard to iterate over combinations of filter or attribute values and quickly browse through large numbers of visualizations. The partial specification is inspired by existing work on visualization query languages, including [ZQL](https://github.com/vega/compassql) and [CompassQL](https://github.com/vega/compassql).

For example, we might be interested in looking at how the `AverageCost` distribution differs across different `Region`s.
For example, we are interested in how the `AverageCost` distribution differs across different `Region`s.

```python
from lux.vis.VisList import VisList
Expand All @@ -134,7 +135,7 @@ To find out more about other features in Lux, see the complete documentation on
# Installation & Setup

To get started, please follow both the installation and setup instructions in your command line.
`lux-api` can be installed through [PyPI](https://pypi.org/project/lux-api/).
`lux-api` can be installed through [PyPI](https://pypi.org/project/lux-api/) or [conda-forge](https://github.com/conda-forge/lux-api-feedstock).

```bash
pip install lux-api
Expand Down Expand Up @@ -162,13 +163,13 @@ Note that you may have to restart the Jupyter Notebook server to ensure that the

## Setup in Jupyter Lab

To use Lux in [Jupyter Lab](https://github.com/jupyterlab/jupyterlab), activate the lab extension:
Lux is compatible with both Jupyter Lab version 2 and 3. To use Lux in [Jupyter Lab](https://github.com/jupyterlab/jupyterlab), activate the lab extension:

```bash
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install luxwidget
```
Lux is only compatible with Jupyter Lab version 2.2.9 and below. Support for the recent [JupyterLab 3](https://blog.jupyter.org/jupyterlab-3-0-is-out-4f58385e25bb) will come soon. Note that JupyterLab and VSCode is supported only for lux-widget version >=0.1.2, if you have an earlier version, please upgrade to the latest version of [lux-widget](https://pypi.org/project/lux-widget/). Lux has only been tested with the Chrome browser.
Note that JupyterLab and VSCode is supported only for lux-widget version >=0.1.2, if you have an earlier version, please upgrade to the latest version of [lux-widget](https://pypi.org/project/lux-widget/). Lux has only been tested with the Chrome browser.

If you encounter issues with the installation, please refer to [this page](https://lux-api.readthedocs.io/en/latest/source/guide/FAQ.html#troubleshooting-tips) to troubleshoot the installation. Follow [these instructions](https://lux-api.readthedocs.io/en/latest/source/getting_started/installation.html#manual-installation-dev-setup) to set up Lux for development purposes.

Expand Down
3 changes: 1 addition & 2 deletions doc/source/advanced/executor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To do this, users first need to specify a connection to their SQL database. This
engine = create_engine("postgresql://postgres:lux@localhost:5432")
Note that users will have to install these packages on their own if they want to connect Lux to their databases.
Once this connection is created, users can connect the lux config to the database using the :code:`set_SQL_connection` command.
Once this connection is created, users can connect Lux to their database using the :code:`set_SQL_connection` command.

.. code-block:: python
Expand All @@ -54,7 +54,6 @@ Connecting a LuxSQLTable to a Table/View
----------------------------------------

LuxSQLTables can be connected to individual tables or views created within your Postgresql database. This can be done by specifying the table or view name in the constructor.
.. We are actively working on supporting joins between multiple tables. But as of now, the functionality is limited to one table or view per LuxSQLTable object only.

.. code-block:: python
Expand Down
61 changes: 61 additions & 0 deletions doc/source/reference/gallery.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@

Gallery of Notebook Examples
==============================

Demo Examples
------------------
The following notebooks demonstrates how Lux can be used with a variety of different datasets and analysis

- Basic walkthrough of Lux with the Cars dataset [`Source <https://github.com/lux-org/lux-binder/blob/master/demo/cars_demo.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/cars_demo.ipynb>`_]
- Basic walkthrough of Lux with the College dataset [`Source <https://github.com/lux-org/lux-binder/blob/master/demo/college_demo.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/college_demo.ipynb>`_]
- Understanding Global COVID-19 intervention levels [`Source <https://github.com/lux-org/lux-binder/blob/master/demo/hpi_covid_demo.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/hpi_covid_demo.ipynb>`_]
- Understanding factors leading to Employee attrition [`Source <https://github.com/lux-org/lux-binder/blob/master/demo/employee_demo.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/employee_demo.ipynb>`_]

If you would like to try out Lux on your own, we provide a `notebook environment <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/demo/exercise/4-Data-Playground.ipynb>`_ for you to play around with several provided example datasets.

Tutorials
---------

The following tutorials cover the most basic to advanced features in Lux. The first five tutorials introduces the core features in Lux:

- Overview of Features in Lux [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/0-overview.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/0-overview.ipynb>`_]
- Specifying user analysis intent in Lux [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/1-specifying-intent.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/1-specifying-intent.ipynb>`_]
- Creating quick-and-dirty visualizations with :code:`Vis` and :code:`VisList` [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/2-constructing-vis-and-vislist.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/2-constructing-vis-and-vislist.ipynb>`_]
- Seamless export of :code:`Vis` from notebook to downstream tasks [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/3-widget-vis-export.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/3-widget-vis-export.ipynb>`_]
- Customizing plot styling with :code:`lux.config.plotting_style` [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/4-chart-settings.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/4-chart-settings.ipynb>`_]

The following notebooks covers more advanced topics in Lux:

- Understanding data types in Lux [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/9-datatype.ipynb?>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/9-datatype.ipynb?raw=true>`_]
- Working with datetime columns [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/tutorial/5-datetime.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/5-datetime.ipynb>`_]
- Working with geographic columns [`ReadTheDocs <https://lux-api.readthedocs.io/en/latest/source/advanced/map.html>`_]
- Working with dataframe indexes [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/tutorial/6-index-group.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/6-index-group.ipynb>`_]
- Registering custom recommendation actions [`Source <https://github.com/lux-org/lux-binder/blob/master/tutorial/8-custom-action.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/tutorial/8-custom-action.ipynb>`_]
- Using Lux with a SQL Database [`Source <https://github.com/lux-org/lux-binder-sql/blob/master/notebooks/Using%20Lux%20with%20SQL%20Databases.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder-sql/HEAD>`_]


Exercise
---------

Here are some teaching resources on Lux. The materials are suited for a 1-1.5 hour industry bootcamp or lecture for a data visualization or data science course. Here is a `video <https://www.youtube.com/watch?v=YANIids_Nkk>`_ that walks through these hands-on exercise on Lux. To follow along, check out the instructions `here <https://github.com/lux-org/lux-binder>`_.

1. How to specify your analysis interests as `intents` in Lux? [`Source <https://github.com/lux-org/lux-binder/blob/master/exercise/1-Specify-Intent.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/exercise/1-Specify-Intent.ipynb>`_]

2. How to create visualizations using :code:`Vis` and :code:`VisList`? [`Source <https://github.com/lux-org/lux-binder/blob/master/exercise/2-Quick-Vis.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/exercise/2-Quick-Vis.ipynb>`_]

3. How to export selected visualization(s) out of the notebook for sharing? [`Source <https://github.com/lux-org/lux-binder/blob/master/exercise/3-Export-Widget.ipynb>`_] [`Live Notebook <https://mybinder.org/v2/gh/lux-org/lux-binder/master?urlpath=tree/exercise/3-Export-Widget.ipynb>`_]

Here are the `solutions <https://github.com/lux-org/lux-binder/blob/master/exercise/solution.txt>`_ to the notebook exercises.


Community Contributions
-------------------------
Here are some awesome articles and tutorials written by the Lux community:

- `Exploring the Penguins dataset with Lux by Parul Pandey <https://towardsdatascience.com/intelligent-visual-data-discovery-with-lux-a-python-library-dc36a5742b2f>`_
- `Analysis of the Wine dataset by Domino Data Lab <https://blog.dominodatalab.com/faster-data-exploration-in-jupyter-through-the-lux-widget/>`_
- `Quick Recommendation-Based Data Exploration with Lux by Cornellius Yudha Wijaya <https://towardsdatascience.com/quick-recommendation-based-data-exploration-with-lux-f4d0ccb68133>`_
- `Analyzing the Graduate Admissions dataset with Lux by Pranavi Duvva <https://pub.towardsai.net/speed-up-eda-with-the-intelligent-lux-37f96542527b>`_


If you would like your notebook or articles to be featured here, please submit a pull request `here <https://github.com/lux-org/lux-binder>`_ to let us know!
1 change: 1 addition & 0 deletions doc/source/reference/gen/lux.core.frame.LuxDataFrame.rst
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@
~LuxDataFrame.set_intent_as_vis
~LuxDataFrame.set_intent_on_click
~LuxDataFrame.shift
~LuxDataFrame.show_all_column_vis
~LuxDataFrame.skew
~LuxDataFrame.slice_shift
~LuxDataFrame.sort_index
Expand Down
138 changes: 138 additions & 0 deletions examples/GeneralDatabase_Executor_Example.py.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "expected-facility",
"metadata": {},
"source": [
"This notebook is an example of how to use the General Database Executor in Lux. This execution backend allows users to switch what kind of queries are being used to query their database system. Here we show how to switch from using a SQL template for Postgresql to MySQL."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "helpful-liberty",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "97a93a0b783743fab041362d66d72125",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Button(description='Toggle Table/Lux', layout=Layout(bottom='6px', top='6px', width='200px'), style=ButtonStyl…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e216a8adf9584b6e8a3cc5374ae73209",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import sys\n",
"sys.path.insert(1, 'C:\\\\Users\\\\thyne\\\\Documents\\\\GitHub\\\\lux')\n",
"\n",
"import lux\n",
"import psycopg2\n",
"import pandas as pd\n",
"from lux import LuxSQLTable\n",
"\n",
"connection = psycopg2.connect(\"host=localhost user=postgres password=lux dbname=postgres\")\n",
"lux.config.set_SQL_connection(connection)\n",
"lux.config.read_query_template(\"postgres_query_template.txt\")\n",
"lux.config.quoted_queries = True\n",
"\n",
"sql_tbl = LuxSQLTable(table_name='car')\n",
"sql_tbl.intent = [\"Cylinders\"]\n",
"sql_tbl"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "searching-nancy",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a2c12d8447494178aa6c38fc0a4c59f6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Button(description='Toggle Table/Lux', layout=Layout(bottom='6px', top='6px', width='200px'), style=ButtonStyl…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "26b23f594155417e9fb7ff2b4695477c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import sqlalchemy\n",
"import lux\n",
"from sqlalchemy.ext.declarative import declarative_base\n",
"\n",
"engine = sqlalchemy.create_engine('mysql+mysqlconnector://luxuser:lux@localhost:3306/sys',echo=False)\n",
"lux.config.set_SQL_connection(engine)\n",
"lux.config.read_query_template(\"mysql_query_template.txt\")\n",
"lux.config.quoted_queries = False\n",
"\n",
"sql_df = lux.LuxSQLTable(table_name='car')\n",
"\n",
"sql_df.intent = ['Cylinders']\n",
"sql_df"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading

0 comments on commit 18c968e

Please sign in to comment.