[BUG] Problem with setting intent on a column containing NaNs #179

bilalmussaukfd · 2020-12-14T13:32:44Z

Hiya,
I received the below issue when trying to use Lux in Jupyter notebooks.
My dataframe is a pandas dataframe that i have populated using the BigQuery read_gbq function.

any suggestions?

C:\Users\bmussa\Anaconda3\lib\site-packages\IPython\core\formatters.py:345: UserWarning:
Unexpected error in rendering Lux widget and recommendations. Falling back to Pandas display.
Please report the following issue on Github: https://github.com/lux-org/lux/issues

C:\Users\bmussa\Anaconda3\lib\site-packages\lux\core\frame.py:711: UserWarning:Traceback (most recent call last):
File "C:\Users\bmussa\Anaconda3\lib\site-packages\lux\core\frame.py", line 663, in repr_html
self.maintain_recs()
File "C:\Users\bmussa\Anaconda3\lib\site-packages\lux\core\frame.py", line 494, in maintain_recs
custom_action_collection = custom_actions(rec_df)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\lux\action\custom.py", line 75, in custom_actions
recommendation = lux.actions.getattr(action_name).action(ldf)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\lux\action\enhance.py", line 67, in enhance
vis.score = interestingness(vis, ldf)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\lux\interestingness\interestingness.py", line 76, in interestingness
return unevenness(vis, ldf, measure_lst, dimension_lst)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\lux\interestingness\interestingness.py", line 274, in unevenness
return D * euclidean(v, v_flat)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\scipy\spatial\distance.py", line 614, in euclidean
return minkowski(u, v, p=2, w=w)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\scipy\spatial\distance.py", line 517, in minkowski
dist = norm(u_v, ord=p)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\scipy\linalg\misc.py", line 140, in norm
a = np.asarray_chkfinite(a)
File "C:\Users\bmussa\Anaconda3\lib\site-packages\numpy\lib\function_base.py", line 498, in asarray_chkfinite
raise ValueError(
ValueError: array must not contain infs or NaNs

ValueError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\lux\core\frame.py in repr_html(self)
662 # df_to_display.maintain_recs() # compute the recommendations (TODO: This can be rendered in another thread in the background to populate self._widget)
--> 663 self.maintain_recs()
664

~\Anaconda3\lib\site-packages\lux\core\frame.py in maintain_recs(self)
493 # generate vis from globally registered actions and append to dataframe
--> 494 custom_action_collection = custom_actions(rec_df)
495 for rec in custom_action_collection:

~\Anaconda3\lib\site-packages\lux\action\custom.py in custom_actions(ldf)
74 else:
---> 75 recommendation = lux.actions.getattr(action_name).action(ldf)
76 recommendations.append(recommendation)

~\Anaconda3\lib\site-packages\lux\action\enhance.py in enhance(ldf)
66 for vis in vlist:
---> 67 vis.score = interestingness(vis, ldf)
68

~\Anaconda3\lib\site-packages\lux\interestingness\interestingness.py in interestingness(vis, ldf)
75 if n_filter == 0:
---> 76 return unevenness(vis, ldf, measure_lst, dimension_lst)
77 elif n_filter == 1:

~\Anaconda3\lib\site-packages\lux\interestingness\interestingness.py in unevenness(vis, ldf, measure_lst, dimension_lst)
273 v = v.astype("int")
--> 274 return D * euclidean(v, v_flat)
275

~\Anaconda3\lib\site-packages\scipy\spatial\distance.py in euclidean(u, v, w)
613 """
--> 614 return minkowski(u, v, p=2, w=w)
615

~\Anaconda3\lib\site-packages\scipy\spatial\distance.py in minkowski(u, v, p, w)
516 u_v = root_w * u_v
--> 517 dist = norm(u_v, ord=p)
518 return dist

~\Anaconda3\lib\site-packages\scipy\linalg\misc.py in norm(a, ord, axis, keepdims, check_finite)
139 if check_finite:
--> 140 a = np.asarray_chkfinite(a)
141 else:

~\Anaconda3\lib\site-packages\numpy\lib\function_base.py in asarray_chkfinite(a, dtype, order)
497 if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all():
--> 498 raise ValueError(
499 "array must not contain infs or NaNs")

ValueError: array must not contain infs or NaNs

During handling of the above exception, another exception occurred:

KeyboardInterrupt Traceback (most recent call last)
in
1 data.intent = ['TRAN_INC_ACCESSORIES']
----> 2 data

~\Anaconda3\lib\site-packages\IPython\core\displayhook.py in call(self, result)
260 self.start_displayhook()
261 self.write_output_prompt()
--> 262 format_dict, md_dict = self.compute_format_data(result)
263 self.update_user_ns(result)
264 self.fill_exec_result(result)

~\Anaconda3\lib\site-packages\IPython\core\displayhook.py in compute_format_data(self, result)
149
150 """
--> 151 return self.shell.display_formatter.format(result)
152
153 # This can be set to True by the write_output_prompt method in a subclass

~\Anaconda3\lib\site-packages\IPython\core\formatters.py in format(self, obj, include, exclude)
178 md = None
179 try:
--> 180 data = formatter(obj)
181 except:
182 # FIXME: log the exception

in call(self, obj)

~\Anaconda3\lib\site-packages\IPython\core\formatters.py in catch_format_error(method, self, *args, **kwargs)
222 """show traceback on failed format call"""
223 try:
--> 224 r = method(self, *args, **kwargs)
225 except NotImplementedError:
226 # don't warn on NotImplementedErrors

~\Anaconda3\lib\site-packages\IPython\core\formatters.py in call(self, obj)
343 method = get_real_method(obj, self.print_method)
344 if method is not None:
--> 345 return method()
346 return None
347 else:

~\Anaconda3\lib\site-packages\lux\core\frame.py in repr_html(self)
710 )
711 warnings.warn(traceback.format_exc())
--> 712 display(self.display_pandas())
713
714 def display_pandas(self):

~\Anaconda3\lib\site-packages\IPython\core\display.py in display(include, exclude, metadata, transient, display_id, *objs, **kwargs)
311 publish_display_data(data=obj, metadata=metadata, **kwargs)
312 else:
--> 313 format_dict, md_dict = format(obj, include=include, exclude=exclude)
314 if not format_dict:
315 # nothing to display (e.g. ipython_display took over)

~\Anaconda3\lib\site-packages\IPython\core\formatters.py in format(self, obj, include, exclude)
178 md = None
179 try:
--> 180 data = formatter(obj)
181 except:
182 # FIXME: log the exception

in call(self, obj)

~\Anaconda3\lib\site-packages\IPython\core\formatters.py in catch_format_error(method, self, *args, **kwargs)
222 """show traceback on failed format call"""
223 try:
--> 224 r = method(self, *args, **kwargs)
225 except NotImplementedError:
226 # don't warn on NotImplementedErrors

~\Anaconda3\lib\site-packages\IPython\core\formatters.py in call(self, obj)
700 type_pprinters=self.type_printers,
701 deferred_pprinters=self.deferred_printers)
--> 702 printer.pretty(obj)
703 printer.flush()
704 return stream.getvalue()

~\Anaconda3\lib\site-packages\IPython\lib\pretty.py in pretty(self, obj)
392 if cls is not object
393 and callable(cls.dict.get('repr')):
--> 394 return _repr_pprint(obj, self, cycle)
395
396 return _default_pprint(obj, self, cycle)

~\Anaconda3\lib\site-packages\IPython\lib\pretty.py in repr_pprint(obj, p, cycle)
698 """A pprint that just redirects to the normal repr function."""
699 # Find newlines and replace them with p.break()
--> 700 output = repr(obj)
701 lines = output.splitlines()
702 with p.group():

~\Anaconda3\lib\site-packages\pandas\core\frame.py in repr(self)
742 else:
743 width = None
--> 744 self.to_string(
745 buf=buf,
746 max_rows=max_rows,

~\Anaconda3\lib\site-packages\pandas\core\frame.py in to_string(self, buf, columns, col_space, header, index, na_rep, formatters, float_format, sparsify, index_names, justify, max_rows, min_rows, max_cols, show_dimensions, decimal, line_width, max_colwidth, encoding)
881 line_width=line_width,
882 )
--> 883 return formatter.to_string(buf=buf, encoding=encoding)
884
885 # ----------------------------------------------------------------------

~\Anaconda3\lib\site-packages\pandas\io\formats\format.py in to_string(self, buf, encoding)
919 encoding: Optional[str] = None,
920 ) -> Optional[str]:
--> 921 return self.get_result(buf=buf, encoding=encoding)
922
923 def to_latex(

~\Anaconda3\lib\site-packages\pandas\io\formats\format.py in get_result(self, buf, encoding)
518 """
519 with self.get_buffer(buf, encoding=encoding) as f:
--> 520 self.write_result(buf=f)
521 if buf is None:
522 return f.getvalue()

~\Anaconda3\lib\site-packages\pandas\io\formats\format.py in write_result(self, buf)
832 else:
833
--> 834 strcols = self._to_str_columns()
835 if self.line_width is None: # no need to wrap around just print
836 # the whole frame

~\Anaconda3\lib\site-packages\pandas\io\formats\format.py in _to_str_columns(self)
769 )
770 fmt_values = self._format_col(i)
--> 771 fmt_values = _make_fixed_width(
772 fmt_values, self.justify, minimum=header_colwidth, adj=self.adj
773 )

~\Anaconda3\lib\site-packages\pandas\io\formats\format.py in _make_fixed_width(strings, justify, minimum, adj)
1749 return x
1750
-> 1751 strings = [just(x) for x in strings]
1752 result = adj.justify(strings, max_len, mode=justify)
1753 return result

~\Anaconda3\lib\site-packages\pandas\io\formats\format.py in (.0)
1749 return x
1750
-> 1751 strings = [just(x) for x in strings]
1752 result = adj.justify(strings, max_len, mode=justify)
1753 return result

KeyboardInterrupt:

dorisjlee · 2020-12-14T13:59:03Z

Hi @bilalmussaukfd, Thank you for your interest in Lux. It seems like there are some issues with the way that the visualizations are computed. Could you share a sample of the dataset or a dataset that is similar, so that we can reproduce the error for testing?

bilalmussaukfd · 2020-12-14T14:14:10Z

Hiya,
Do you have an email address I can send the data to please?

Bilal

dorisjlee · 2020-12-14T14:21:46Z

Hi @bilalmussaukfd , You can send it to me at dorisjunglinlee@gmail.com.

* improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test

dorisjlee · 2020-12-21T14:24:40Z

Hi @bilalmussaukfd, I've made various bugfixes to support NaN values and have tested them out on the sample dataset that you sent me. You can access these updated changes by upgrading to the latest version of Lux (v0.2.1.2):

pip install --upgrade lux-api
jupyter nbextension install --py luxwidget
jupyter nbextension enable --py luxwidget

Please let us know if this addresses the issue mentioned above. Thanks!

bilalmussaukfd · 2020-12-21T14:31:23Z

Works a treat – happy days 😊 From: Doris Lee <notifications@github.com> Sent: 21 December 2020 14:25 To: lux-org/lux <lux@noreply.github.com> Cc: Bilal Mussa <bilalmussa@ukflooringdirect.co.uk>; Mention <mention@noreply.github.com> Subject: Re: [lux-org/lux] [BUG] Problem with setting intent on a column containing NaNs (#179) Hi @bilalmussaukfd<https://github.com/bilalmussaukfd>, I've made various bugfixes to support NaN values and have tested them out on the sample dataset that you sent me. You can access these updated changes by upgrading to the latest version of Lux (v0.2.1.2<https://github.com/lux-org/lux/releases/tag/v0.2.1.2>): pip install --upgrade lux-api jupyter nbextension install --py luxwidget jupyter nbextension enable --py luxwidget Please let us know if this addresses the issue mentioned above. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#179 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR7AEORMTTEEV32AIZRQZB3SV5LDRANCNFSM4U2X7DTQ>.

* Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * bump version number and requirements * PATCH v0.2.1.1 * remove __future__ annotations dependency * remove channel inheritance in Enhance * bugfix 3-clause Vis example in 5-datetime.ipynb * update README * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Moved Executor Parameters to Global Config (#157) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * save_as_html feature (#170) * note this requires unpkg depedency from latest npm release (for now download https://github.com/lux-org/lux-widget/blob/master/luxwidget/nbextension/static/index.js, rename it luxwidget.js and place it in the same directory as the exported HTML) * Updating documentation for lux-widget v0.1.2 release (#176) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * Updated readme * updated readthedoc documentation * added config to default display documentation * Update README.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix bug caused by `groupby.agg` on column with many unique values (#174) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update save_as_html docs (#170) * Update README *remove jupyter lab url * Update README.md fix slack link * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Making Lux more robust with missing values and NaN (#179) (#180) * improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test * bump version and requirement * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com>

* Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * bump version number and requirements * PATCH v0.2.1.1 * remove __future__ annotations dependency * remove channel inheritance in Enhance * bugfix 3-clause Vis example in 5-datetime.ipynb * update README * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Moved Executor Parameters to Global Config (#157) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * save_as_html feature (#170) * note this requires unpkg depedency from latest npm release (for now download https://github.com/lux-org/lux-widget/blob/master/luxwidget/nbextension/static/index.js, rename it luxwidget.js and place it in the same directory as the exported HTML) * Updating documentation for lux-widget v0.1.2 release (#176) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * Updated readme * updated readthedoc documentation * added config to default display documentation * Update README.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix bug caused by `groupby.agg` on column with many unique values (#174) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update save_as_html docs (#170) * Update README *remove jupyter lab url * Update README.md fix slack link * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Making Lux more robust with missing values and NaN (#179) (#180) * improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test * bump version and requirement * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement * Update SQL Executor Documentation Updated documentation in SQLExecutor.py. Updated example notebook for SQLExecutor * Updated SQLExecutor Example Notebook Updated example notebook for the SQL Executor and the script to upload the example dataset to a local Postgres database. * Black Formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com>

* Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Moving Current SQL Executor changes to new branch (#119) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Added test environment for Postgresql Executor (#124) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Added example notebook, fixed variable reference (#130) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Updating SQL-Engine branch to main branch, Adding Heatmap Functionality to SQL Executor (#154) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Merged changes from Master branch, Moved Executor Parameters to Config (#163) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement * Update to Sql-Engine (#190) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * bump version number and requirements * PATCH v0.2.1.1 * remove __future__ annotations dependency * remove channel inheritance in Enhance * bugfix 3-clause Vis example in 5-datetime.ipynb * update README * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Moved Executor Parameters to Global Config (#157) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * save_as_html feature (#170) * note this requires unpkg depedency from latest npm release (for now download https://github.com/lux-org/lux-widget/blob/master/luxwidget/nbextension/static/index.js, rename it luxwidget.js and place it in the same directory as the exported HTML) * Updating documentation for lux-widget v0.1.2 release (#176) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * Updated readme * updated readthedoc documentation * added config to default display documentation * Update README.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix bug caused by `groupby.agg` on column with many unique values (#174) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update save_as_html docs (#170) * Update README *remove jupyter lab url * Update README.md fix slack link * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Making Lux more robust with missing values and NaN (#179) (#180) * improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test * bump version and requirement * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Update SQL Executor Documentation Updated documentation in SQLExecutor.py. Updated example notebook for SQLExecutor * Updated SQLExecutor Example Notebook Updated example notebook for the SQL Executor and the script to upload the example dataset to a local Postgres database. * Black Formatting * Update to SQL Executor Example Notebook (#193) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * f…

* Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Moving Current SQL Executor changes to new branch (#119) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Added test environment for Postgresql Executor (#124) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Added example notebook, fixed variable reference (#130) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Updating SQL-Engine branch to main branch, Adding Heatmap Functionality to SQL Executor (#154) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Merged changes from Master branch, Moved Executor Parameters to Config (#163) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement * Update to Sql-Engine (#190) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * bump version number and requirements * PATCH v0.2.1.1 * remove __future__ annotations dependency * remove channel inheritance in Enhance * bugfix 3-clause Vis example in 5-datetime.ipynb * update README * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Moved Executor Parameters to Global Config (#157) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * save_as_html feature (#170) * note this requires unpkg depedency from latest npm release (for now download https://github.com/lux-org/lux-widget/blob/master/luxwidget/nbextension/static/index.js, rename it luxwidget.js and place it in the same directory as the exported HTML) * Updating documentation for lux-widget v0.1.2 release (#176) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * Updated readme * updated readthedoc documentation * added config to default display documentation * Update README.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix bug caused by `groupby.agg` on column with many unique values (#174) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update save_as_html docs (#170) * Update README *remove jupyter lab url * Update README.md fix slack link * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Making Lux more robust with missing values and NaN (#179) (#180) * improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test * bump version and requirement * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Update SQL Executor Documentation Updated documentation in SQLExecutor.py. Updated example notebook for SQLExecutor * Updated SQLExecutor Example Notebook Updated example notebook for the SQL Executor and the script to upload the example dataset to a local Postgres database. * Black Formatting * Update to SQL Executor Example Notebook (#193) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic…

* Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Merged changes from Master branch, Moved Executor Parameters to Config (#163) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement * Update to Sql-Engine (#190) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * bump version number and requirements * PATCH v0.2.1.1 * remove __future__ annotations dependency * remove channel inheritance in Enhance * bugfix 3-clause Vis example in 5-datetime.ipynb * update README * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Moved Executor Parameters to Global Config (#157) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * save_as_html feature (#170) * note this requires unpkg depedency from latest npm release (for now download https://github.com/lux-org/lux-widget/blob/master/luxwidget/nbextension/static/index.js, rename it luxwidget.js and place it in the same directory as the exported HTML) * Updating documentation for lux-widget v0.1.2 release (#176) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * Updated readme * updated readthedoc documentation * added config to default display documentation * Update README.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix bug caused by `groupby.agg` on column with many unique values (#174) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update save_as_html docs (#170) * Update README *remove jupyter lab url * Update README.md fix slack link * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Making Lux more robust with missing values and NaN (#179) (#180) * improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test * bump version and requirement * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Update SQL Executor Documentation Updated documentation in SQLExecutor.py. Updated example notebook for SQLExecutor * Updated SQLExecutor Example Notebook Updated example notebook for the SQL Executor and the script to upload the example dataset to a local Postgres database. * Black Formatting * Update to SQL Executor Example Notebook (#193) * Merging Recent SQL Executor changes * Fix to Validator Uses unique value metadata to verify if a value is valid * Fix Bug with Widget Rendering frame.py was trying to import luxWidget instead of luxwidget * Added Number of Observations to MetaData, Fixed Interestingness issue with SQL Executor Some interestingness functions required the number of observations in the data and visualization, so I added these values to the metadata to make the scoring work when using the SQL executor Added tests for SQL executor * Re-added Licensing Headers * Adding Recent frame.py changes * Adjusted SQL Executor Tests Removed lines that changed Year column type to datetime * Update Frame with new Action Registering * Resolving Conflicts in frame.py * Commenting out local SQL Executor tests SQL Executor tests interfering with travis build, commenting out for now * Update correlation.py * Update frame.py * bugfix: "number of remaining bars" text overcounts for colored bar charts * update number of bars calculation to account for when len(data) double counts * Fixing Code Format * Cleaning up Pandas Executor imports * Fix Validation Bug Issue where validator was relying on metadata which was not yet generated, moved metadata calculation before validation step in frame.py * Changed metadata variable name Renamed num_obs to length, removed ordinal variable from Executor mapping function * Adding support for setting intent on front end (#112) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * Make default_display a global setting (#121) * remove and register action functions * update changes inframe.py * update changes inframe.py * add documentation and changes * indentation and comments * new line * globally defined default display works with warning * no examples * add back space * new line * uncomment docstring Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Added script to generate Postgresql database Updated travis.yml file to create postgresql database in test instance. Added script to populate test database with data. * Update upload_car_data.py Updated database credentials * Updated script name in travis.yml * Removed unnecessary import from travis.yml * Added psycopg2 to requirements.txt * Creating Postgres test database in travis * Fixed directory issue * Updated SQL Executor Tests Added tests for basic SQL Executor functionality. * Update requirements.txt (#128) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * separate dev and install requirements * replaced _exportedVisIdxs --> _selectedVisIdxs * bugfix: plot config error when current_vis is None * Added sql_executor example notebook, minor bug fix Added an example notebook to showcase how to use the sql-engine. Fixed variable reference in interestingness.py that was causing issues. * Add LuxSeries Implementation (#122) * add preliminary groupby fixes * preliminary LuxSeries implementation * add tests for new Series implementation * clean up the added code * minor code changes * fix issues with Vis with index * small fixes * remove comments * bugfix column group display empty Vis involving groupby index * bugfix Cylinders not showing up as bar charts Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * add black to travis (#127) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * fix doc failing from black format * Cleaned SQL Executor Example Notebook restarted kernel and cleared output * Update custom action reference to executor Now uses executor tied to the dataframe for execution * Updated Interestingness Tests (#133) * add black * update cars dataset and tests * Delete old dataset * Updated Interestingness Tests Updated tests to use the newly updated cars dataset * switch to local cars reference Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * fix broken link in docs * Updated Tests, Added benchmarking for SQL Executor Updated Compiler and Interestingness tests to work for SQL executor. Updated SQL Executor to have some benchmarking code for tracking query performance. * Merge with upstream branch, added preliminary benchmarking code * Better warning message for Vis and VisList (#135) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Pandas rewrite Performance optimizations (#136) * basic scatterplot experiments * experiment results with manually binned heatmaps * experiment result * incorporated heatmap code into executor and renderer * additional experiments to evaluate scatter v.s. heatmap performance * experiment based on real estate and airbnb data * modified general sampling criteria, suppress SettingWithCopyWarning stemming from groupby .agg (#93) * decrease sampling parameter * change sampling strategy (above threshold keep 3/4 of data) * remove experiment dir * modified performance param * enforce lux-widget minimum version * update requirement.txt * testing out modin (Recursion error) * create modin executor, all else in sync with master changes * rewrote .loc with column reference, speed up by 100x * replace agg("count") with .count() --> ~0.1ms speedup * run black * Added 2D Binning functionality to SQL Executor added 2D binning to replace scatterplots when using SQL executor. * Update README.md update slack link * Updated temporal detection and tests (#139) * Updated temporal detection and tests * Reformatted code with black * Update PandasExecutor.py * added stock date test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix Inline comments breaking to new lines (#137) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improve warning message when values specified as attributes (#143) * Improve warning message when values specified as attributes (#142) * added test, ran black * bugfix test * Better warning message for Vis and VisList (#146) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * adding warning message for Vis intents being > 1 * passes tests and intent < 3 * minor change to error message, added test * run black * accounted for more edge cases and hid traceback * fixed typo * added tests * format w/ black * ran black again * Update Vis.py minor readability changes * added check and tests for Vis list and | syntax * ran black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated docs for JupyterLab (#148) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * updated install and faq * added install_lab.sh script * Global shared variable in test (#144) (#149) * using global shared variable in test (lux-org#144) * modified fixture scope as session, resolved dependency test cases * run black Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Updated 2D Binning Functionality Interestingness function now compatible with 2D binning with SQL Executor. Fixed issue where some datapoints in heatmap data were of string type instead of integer. * documentation for custom actions (#134) * documentation for custom actions * fix config tests * fix documentaion links * Add default display Config Class documentation * delete hpi * doc resolved changes Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch failing test (#134) * Patch failing test (#134) * Patch failing test (#134) * Added Heatmap generation to SQL Executor, Bug fix in PandasExecutor Added 2D Binning function to collect data for heatmaps in the SQL Executor and updated compiler test to reflect heatmap changes in SQL Executor Fixed issue within PandasExecutor where 2D binning function would not color heatmap using temporal variables * Updated Code Formatting with Black * Update Requirements to include psycopg2 * Update upload_car_data.py Updated to use newest car.csv file * Update Compiler tests to use correct test DB * Removed Benchmarking Code * Fixing Black Formatting * Add Toggle button for LuxSeries (#153) * add black * add groupby to history * add LuxSeries visualization implementation * remove groupby extending * run black * merge master into branch * run black * Reformat Warnings Labels (#151) * add format for warnings * fix small details * globally defined lux * fix formatting * Delete similarity.py * Rename similarity_old.py to similarity.py * fix commit * update master with changes to warning labels * fix formatting Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> * Make plot_config a global variable (#152) * plot_config made global config * docs and tests updates; black formatting * Delete Untitled.ipynb * cleanup * remove data encapsulation * black formatting * Update style.rst Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Patch documentation (#134) * code cleanup * making LuxDataFrame pickle-able * various simplification and rewrite to Compiler and Vis * changing `render_VSpec` to `to_code` * bugfix in filter (list comprehension lazily evaluated) * Moved Executor Parameters to Global Config * Black formatting * Fix issue with read_json and add tests for different file types (#156) * fix read_json bug and add tests * convert tests to reference lux-datasets * run black * remove comments * Update __init__.py * Update test_pandas_coverage.py new_df --> df * fix tests to work with lux-datasets * fix init * remove lxml dependency * remove html test Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Improved warning message stack trace for unexpected error (#145) * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * bug fix patching #146 (#159) * remove sys.tracebacklimit to avoid `ERROR:root:Internal Python error in the inspect module` * changed SyntaxError to TypeError to reserved SyntaxError for Pandas native syntax problems * Ensure that recommendation and current_vis properties are accessible before df first printed (#159) * bugfix erased `name` dtype in LuxSeries (#140) * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update Makefile and pyproject.toml (#160) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version * remove dev dependencies * first pass -- inline comments * _config/config.py * delete test notebook * action * line length 105 * executor * interestingness * processor * vislib * tests, travis, CONTRIBUTING * .format () changed * replace tabs with escape chars * update using black * more rewrites and merges into single line * update pyproject.toml and makefile * updating contributing doc * Update CONTRIBUTING.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update README.md * Update SQLExecutor name parameter * Merging master branch with sql engine. Moving executor parameters to config * Parameter Bug Fix Lux DataFrame current_vis should be an empty list after recs are expired. Executor name for small_df and very_small_df checks should be "PandasExecutor" * Reference Fix in Warning Executor check here should look for name "PandasExecutor" * Black Formatting * Black formatting * bump version number and requirements * PATCH v0.2.1.1 * remove __future__ annotations dependency * remove channel inheritance in Enhance * bugfix 3-clause Vis example in 5-datetime.ipynb * update README * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Cleaned up executor imports, Fixed issue in AltairRenderer Issue where Altair Renderer was not using lux.config.executor * Black formatting * Moved Executor Parameters to Global Config (#157) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * save_as_html feature (#170) * note this requires unpkg depedency from latest npm release (for now download https://github.com/lux-org/lux-widget/blob/master/luxwidget/nbextension/static/index.js, rename it luxwidget.js and place it in the same directory as the exported HTML) * Updating documentation for lux-widget v0.1.2 release (#176) * added functionality to delete Vis * fixed deletion logic * add observer to automatically update deletions * able to refresh widget on setting intent * support for setting intent from frontend * quick fix to output * changed variable intentindex name * added better error msg for > 1 intent for vis * reverting some changes * Updated readme * updated readthedoc documentation * added config to default display documentation * Update README.md Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Fix bug caused by `groupby.agg` on column with many unique values (#174) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Update save_as_html docs (#170) * Update README *remove jupyter lab url * Update README.md fix slack link * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Making Lux more robust with missing values and NaN (#179) (#180) * improve datetime warning message with starter templates * Handling NaN value errors * skipping validator check for NaN filter values * adding special case for PandasExecutor to map filter NaN to isna() * fixing unevenness metric when bar values are NaN * eliminate 1-cardinality filters in Filter action (since equal to overall) * fixed deviation array unequal bug when NaN * Handling NaN filter and data type * fixed data type detection when int coerced to float when containing NaN * added test for applying NaN filter * Ensure that LuxSeries displayed when there is NaN * ensure that NaNs are not dropped in groupbys * exclude NaN values in deviation calculation * fix unnamed series issue * improved debugging message for LuxSeries * Override pd.Series with LuxSeries * Fixes for type checking and line charts with NaNs * exclude NaN for line charts to prevent large axes offsetting * improved type checking for float no-longer NaN columns * fixed and improved deviation calculation test * added float categorical test * bump version and requirement * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Update to Config, and Compiler/Interestingness Tests Removed duplicate set_SQL_connection function from config. Updated tests to reflect these changes. Merged in recent changes in master branch. * Black formatting * Update Requirements.txt Updated to include numpy version requirement * Update SQL Executor Documentation Updated documentation in SQLExecutor.py. Updated example notebook for SQLExecutor * Updated SQLExecutor Example Notebook Updated example notebook for the SQL Executor and the script to upload the example dataset to a local Postgres database. * Black Formatting Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: cjachekang <47467363+cjachekang@users.noreply.github.com> Co-authored-by: Caitlyn Chen <caitlynachen@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Kunal Agarwal <32151899+westernguy2@users.noreply.github.com> Co-authored-by: jinimukh <46768380+jinimukh@users.noreply.github.com> Co-authored-by: Kunal Agarwal <kagarwal2@berkeley.edu> Co-authored-by: Jared Zhao <jaredzhao@berkeley.edu> Co-authored-by: Piyush Gupta <piyushg9794@gmail.com> * Update to SQL Executor Tests Adjusted tests to reference correct PostgreSQL table * Update Travis file and SQL Executor Tests Updated the travis.yml and sql executor test scripts to refer to the correct postgres database * Update .travis.yml * fixed merge conflict issues. vis.data shows None DF. * Merge master into sql-engine + minor mergeconflict fixes * Removing the PYNB * Cleaning up obsolete code * Merging the master branch changes into sql-engine (#208) * Similarity as a default action (#182) * similarity formatting fixed * added another similarity test case; fixed bug where colored heatmap dimension is temporal (invalidate all 2 msr 1 temporal case) * filter and similarity together * filter and similarity together * remove filter * black line length * file reorg and clean; change sim metric Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * bump numpy min version for travis * Special character issue (#184) * rename col * broken * fixed period replacement bug * add tests * refine tests * refine tests * remove cols * fix tests * add agg * fixed tests * clean up PR Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Colored bar interestingness bug (#189) * rewrote chi2 contingency with pd.crosstab * catching KeyError issue with chi2 contingency * padding interestingness with warning instead of error * interestingness now reuses ndim and nmsr computed in Compiler * bug fix for parser with int values * improve Vis repr to better display inferred intent when data is absent but fully compiled intent (all clauses) * Add sampling parameters as a global config (#192) * update export tutorial to add explanation for standalone argument * minor fixes and remove cell output in notebooks * added contributing doc * fix bugs and uncomment some tests * remove raise warning * remove unnecessary import * split up rename test into two parts * fix setting warning, fix data_type bugs and add relevant tests * remove ordinal data type * add test for small dataframe resetting index * add loc and iloc tests * fix attribute access directly to dataframe * add small changes to code * added test for qcut and cut * add check if dtype is Interval * added qcut test * fix Record KeyError * add tests * take care of reset_index case * small edits * add data_model to column_group Clause * small edits for row_group * fixes to row group * add config for start and cap for samples * finish sampling config and tests * black formatting * add documentation for sampling config * remove small added issues * minor changes to docs * implement heatmap flag and add tests * black formatting and documentation edits Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Coalesce all data_type attributes of frame into one (#185) * coalesce data_types into data_type_lookup * black reformat * changed to better variable names * lux not defined error * fixed * black format * Update CONTRIBUTING.md * Bug Fix: User-provided Index causes KeyError in Pandas Execution (#191) * Moved Executor Parameters to Global Config * Black formatting * Moved table_name parameter to frame.py. Removed executor_type parameter executor_type parameter no longer necessary to maintain * Fixed reference to table_name parameter table_name is now a parameter within frame.py * Adjusted Functions to Set SQL Connection Moved set_SQL_connection function to config. Added set_SQL_table function within frame.py to let users specify which database table will be associated with their dataframe * Update SQLExecutor name parameter * Fix Executor Reference Update current_vis() to reference lux.config.executor * Update frame.py * Moved set functions to global config * Fixed Index Issue in Pandas Executor Issue caused when user sets an index. The Pandas Executor was not correctly renaming this new index column to Record in execute_aggregate() * Added tests for set_index functions * Black formatting * Update Pandas Executor to handle NA values Readded missing dropna parameter within execute_aggregate() groupby function call * Updated Pandas Coverage Tests Commented out set_index case which has not been addressed yet * Black Formatting * Update to Pandas Executor Index Handling Cleaned up how execute_aggregrate renames index columns. Now retrieves the index name from vis.data instead of filtering out non-index columns. Created separate test function for when user specifies an index in read_csv. Co-authored-by: 19thyneb <thyne.boonmark@gmail.com> Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> * Initialize Config once only during __init__ (#194) * basic matplotlib chart example * migrate register default action to init * config class * move actions * fixed tests * changes * alright * fix plot_config * black reformat * black reformat Co-authored-by: Doris Lee <dorisjunglinlee@gmail.com> Co-authored-by: Caitlyn Chen <caitlynachen@berkeley.edu> Co-authored-by: Ujjaini Mukhopadhyay <ujjaini@berkeley.edu> * Update README.md * Series Bugfix for describe and convert_dtypes (#197) * bugfix for describe and convert_dtypes * added back metadata series test * black * default to pandas display when df.dtypes printed * Update Lux Docs (#195) * add black to travis * reformat all code and adjust test * remove .idea * fix contributing doc * small change in contributing * update * reformat, update command to fix version…

dorisjlee changed the title ~~Issue: Unexpected error in rendering Lux widget and recommendations.~~ [BUG] Problem with setting intent on a column containing NaNs Dec 14, 2020

dorisjlee added bug Something isn't working priority high priority tasks (for dev) labels Dec 14, 2020

dorisjlee mentioned this issue Dec 19, 2020

Making Lux more robust with missing values and NaN (#179) #180

Merged

dorisjlee closed this as completed Dec 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Problem with setting intent on a column containing NaNs #179

[BUG] Problem with setting intent on a column containing NaNs #179

bilalmussaukfd commented Dec 14, 2020

dorisjlee commented Dec 14, 2020

bilalmussaukfd commented Dec 14, 2020

dorisjlee commented Dec 14, 2020

dorisjlee commented Dec 21, 2020

bilalmussaukfd commented Dec 21, 2020 via email

[BUG] Problem with setting intent on a column containing NaNs #179

[BUG] Problem with setting intent on a column containing NaNs #179

Comments

bilalmussaukfd commented Dec 14, 2020

dorisjlee commented Dec 14, 2020

bilalmussaukfd commented Dec 14, 2020

dorisjlee commented Dec 14, 2020

dorisjlee commented Dec 21, 2020

bilalmussaukfd commented Dec 21, 2020 via email