Merge pull request #3 from twosixlabs/v2.0.0

Version 2.0.0 commit

CHANGELOG.md

@@ -1,3 +1,14 @@
+# 2.0.0
+
+- Various bug fixes and performance improvements, including major backend refactor. This breaks some graphic config jsom layout rules, so bumping major version
+- Includes documentation in-line in wizard
+- Allows upload of multiple files simultaneously
+- Added search box to wizard to better find graph config options, and hid more options in dropdowns by default
+- Added datetime parsing functionality to numerical filters
+- Includes Seaborn graphics plotting functionality, for static figures that use Python API rules. Selectors still work for filtering, but no JS tooltips and such. This does not include all Seaborn plots- some catplot-type multi figure plots not yet supported
+- Added Cytoscape graphics plotting for network graph visualization
+
+
 # 1.0.0
 
 - Removes local csv handler option, tests, and documentation

README.md

@@ -31,15 +31,12 @@ Yes. Escalation has a few advantages:
 ## What do you need for the app to work?
 
 Note, because of Docker issues with Windows, there may be substantial hiccups running on that operating system.
-The instructions below have been tested on Mac and Linux.
+The instructions below have been tested on Mac and Linux. Linux users will need sudo permissions to run the following setup commands. 
 
 - [Configure the app](#2.-Configuring-the-app)
     - Escalation uses configuration files (json) to build the dashboard organizational structure, link the data in visualizations, and construct the visualizations themselves.
     - These configuration files can be built by hand, using the Configuration Wizard, or any combination of the two
 - [Data](#2.2-Load-your-data) 
-    - When setting up Escalation, you choose to use either a CSV or SQL backend.
-    - Depending on the backend, you'll either link the app to a database (new or existing) or a file system path containing your data files. 
-     A file backend may be easier for those unfamiliar with SQL, but SQL is more performant, and storing data in a database offers advantages beyond the database's use in Escalation
     - Escalation includes tooling to ingest CSV files into SQL, automatically building the necessary SQL data tables and the code necessary to integrate them with Escalation.
     - ToDo: Data Migration helpers- what happens when the format of your data changes over time?
 - [Python environment to run the app](#3.-Running-the-configured-app)
@@ -57,9 +54,13 @@ From the root level of the code repository, run:
 
 We recognize that Docker is less common in academic settings, but highly recommend using it. 
 Here are [instructions](https://docs.docker.com/get-started/) on getting started using Docker.
+Additionally, we use Docker Compose for running our multi-container application, which does depend on the Docker Engine. 
+Once you have Docker successfully installed, your next step is to install Docker Compose. 
+Here are [instructions](https://docs.docker.com/compose/) for the installation process. 
 We use the Docker containers to run our configuration wizard, as well as the scripts to ingest csv data into a SQL database.
 Once we set up a configuration and your data, we'll also use these containers to run the web app.
 
+
 ## 2. Configuring the app
 
 ### 2.1 Run the Configuration Wizard
@@ -75,24 +76,8 @@ Refresh your browser to update the contents to match your saved configuration.
 
 Some notes on [creating your first config files with the UI wizard](documentation/wizard_guide/creating_first_graphic_with_wizard.md).  
 
-This includes a gallery of various chart types and how they're configured.
-
-![config_gallery_thumbnail](documentation/gallery/images/gallery_thumbnail.png)
-
-
-#### Build a config from scratch (advanced, optional)
-Run `python build_app_config_json_template.py` to build a base config file. 
-Everything blank or in `<>` should be changed.
-
-#### Debugging config files manually (advanced, optional)
-
-An example of a [main config file](documentation/main_config_example/main_config_example.md).  
-Examples of [different plots and graphic config files](documentation/plotly_examples/plotly_config_info.md).  
-Examples of [different selectors](documentation/selector_examples/selector_config_info.md). 
-
 ### 2.2 Load your data
-
-#### SQL database backend (recommended)
+#### SQL database backend
 
 Escalation provides functionality to parse csv data files, determine the relevant sql schema,
  create tables in the SQL database from your file, and create the necessary `models.py` file for the app to interact with the database. 
@@ -104,7 +89,7 @@ Use this web form to upload each file you'd like to use for your visualizations
 Note, it may take a little while to run.
 
 If you'd like to add more than one csv to the same table, you have two options: 
-combine them before uploading, or submit the additional CSVs using the  "Append to existing table" option.
+Load them at the same time (by holding shift and selecting multiple files), or submit the additional CSVs using the  "Append to existing table" option.
 
 Todo: If you have an existing SQL database, how do you copy it into Escalation?
  We require each table to have an `upload_id` column, and a key that is unique within an `upload_id` value
@@ -195,22 +180,30 @@ ToDo: More detailed instructions on virtual env setup, requirements install,  an
 
 ### Developing for Escalation
 
+- optional, but recommended: run in a virtual environment of your choice. If you use conda and not pyenv or similar, use the appropriate package installer in the next step
 - `pip install -r requirements-dev.txt`
 - `pre-commit install` sets up the pre-commit hooks to auto-format the code. This is optional, the repo is formatted with Flake and Black. 
-
+- From the `escalation/` directory (at the same level as `app,py`, run `export FLASK_ENV=development && python -m flask run`
 
 ### How to add a new type of plot
-Development for Escalation has focused on Plotly, but the codebase should be compatible with other libraries or custom graphics. If you want to use something other than Plotly, your code should:
-* Needs to inherit from graphic_class.py
+Development for Escalation has focused on Plotly, Cytoscape, and Seaborn, but the codebase should be compatible with other libraries or custom graphics. If you want to use something other than the provided libraries, your code should:
+* Inherit from graphic_class.py
 * Be added to available_graphics.py
 * Include an html file with javascript code required to plot
-
+* In addition, if you would like to use the wizard with the new graphic, you should define a schema with a class that inherits from graphic_schema.py. The schema should follow [JSON Schema](https://json-schema.org/). 
 ### How to add a new option feature
 * add it to available_selectors.py
 * create a html document input elements need name "\<id>|\<type>|<column_name>"
-* add to create_data_subselect_info and reformat_filter_form_dict in controller.py
-* build in functionality graphics_class or data_storer class
+* add to `create_data_subselect_info`, `modify_graphic_dict_based_on_addendum_dict`, and `add_operations_to_the_data_from_addendum` as appropriate in graphic_class.py or one of its inheritors.
+* build in functionality in graphic_class or data_storer inheritor
 
+# Using and citing Escalation
+
+
+Code submitted to [Zenodo](https://zenodo.org/) for DOI registration and version tracking.
+Bibtex and similar citation formatting available here:
+
+[![DOI](https://zenodo.org/badge/267139951.svg)](https://zenodo.org/badge/latestdoi/267139951)
 
 # License
 

documentation/droplet_directions.md

@@ -32,5 +32,24 @@
 - At this point you can log out of the ssh session and the app will continue to run
 
 - If you would like to stop the app, run `docker-compose down` in the ssh terminal to your droplet
+# Pointing a domain to Escalation (Nginx)
+One method to point a domain or subdomain to a port:
+- Follow  [the directions for setting up a server block](https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-server-blocks-virtual-hosts-on-ubuntu-16-04)
+- In the directions, replace the file /etc/nginx/sites-available/example.com with
+
+
+```
+server {
+        listen 80;
+        listen [::]:80;
+                
+        server_name your_domain.com;
+
+        location / {
+                proxy_pass http://127.0.0.1:8000;
+        }
+}
+```
+
 
 todo: set Flask SECRET_KEY on server, not in code, set postgres db password, set password for admin actions

documentation/wizard_guide/creating_first_graphic_with_wizard.md

@@ -4,31 +4,35 @@ Before starting this tutorial, you should have your wizard app launched.
 
 Ready? Let's begin.
 
-- First open the url where wizard app is being hosted (e.g. http://localhost:8000/). Click on "Configuration Wizard" on the navbar at the top. You should see
+- First, open the URL where the wizard app is being hosted (e.g. http://localhost:8000/). Click on `Configuration Wizard` on the navbar at the top. You should see
 
 ![](images/first_look.png)
-- First, fill out the title, a description for your dashboard, select the data backend you're using (SQL recommended) and click save.
-- Next, to add data to the app, click on "Upload" on the navigation bar on the top. Name your data table (Data File Type),
- pick a csv file to upload (the csv file must have a header with column names), give a username for the uploader, and click submit. 
+- First, fill out the title, a description for your dashboard, and click save.
+- Next, to add data to the app, click on `Upload` on the navigation bar on the top. Name your data table (Data File Type),
+ pick a CSV file to upload (the CSV file must have a header with column names), give a username for the uploader, and click submit. 
 
 ![](images/add_csv.png)
 
-- Click on "Configuration Wizard".
+- Click on `Configuration Wizard`.
 - Now let's add our first page.
-Enter a name for the new dashboard panel page and click "Add a Dashboard Panel."
+Enter a name for the new dashboard panel page and click `Add a Dashboard Panel`.
 
 ![](images/add_page.png)
 
-- Now let's add a graphic to the page. Click "Add a Graphic"
-
+- Now let's add a graphic to the page. Click `Add a Graphic`. It will take you to the graphic config landing page.
+![](images/landing_page.png)
+- Choose what uploaded data you would like to use and what kind of graphic you would like then click `Make Graph`.
 ![](images/graphic_config.png)
 
-- Fill out the form. The html example here
- [gallery/index.html](../gallery/index.html) (best opened and viewed in a browser) contains graphics and the forms that generated them. Once you are done hit submit.
-Now you will have a graphic in your page. 
-
-![](images/add_graphic.png)
+- Fill out a title and a description.
+- Then specify the data on the axis (axes) and other customization options.
+- To access advanced customization options click the properties buttons.
+- You can use the search bar on the top right side to search for options in the config. The box below the search bar will give you the path in the config to options with matching words in the title or description. Clicking on the path will describe the option in the second box.
+- You can click `Preview Graph` to view the current graph as long as the schema is valid.
+- At the end bottom you can choose data selectors to allow users to interact with the graphs on a page.
+- Once you are done, hit submit.
+- Now you will have a graphic on your page. 
 
-- You can see the graphic by clicking on "Dashboards" then on the title of your page.
+![](images/navbar_newpage.png)
 
-![](images/navbar_newpage.png)
+- You can see the graphic by clicking on `Dashboards` then on the title of your page.

escalation/app.py

@@ -7,6 +7,8 @@
 from app_setup import create_app, configure_app
 from utility.constants import APP_DEPLOY_DATA
 
+# from utility.constants import TEST_APP_DEPLOY_DATA as APP_DEPLOY_DATA
+
 if os.environ.get("CONFIG_FILE_PATH"):
     config_file_path = os.environ["CONFIG_FILE_PATH"]
 else:
@@ -19,8 +21,7 @@
 
 
 config_file_path = os.path.join("app_deploy_data", "main_config.json")
-# config_file_path = "test_app_deploy_data/test_sql_app_config.json"
-# config_file_path = "test_app_deploy_data/test_app_local_config.json"
+# config_file_path = "test_app_deploy_data/main_config.json"
 
 with open(config_file_path, "r") as config_file:
     config_dict = json.load(config_file)

escalation/app_deploy_data/app_settings.py

@@ -26,7 +26,6 @@
         "database": url.database,
     }
 
-
 # password required for endpoints that use the @auth.login_required decorator
 # NB: real versions of these passwords should be passed in as environment variables to the app
 # Don't check in secrets to version control!

escalation/app_setup.py

@@ -89,22 +89,21 @@ def configure_app(app, config_dict, config_file_folder):
     return app
 
 
-def configure_backend(app, models_path="app_deploy_data.models"):
+def configure_backend(app):
     # setup steps unique to SQL-backended apps
     from database.sql_handler import SqlHandler, SqlDataInventory
 
     app.db = SQLAlchemy()
-    engine = create_engine(
-        app.config[SQLALCHEMY_DATABASE_URI], convert_unicode=True
-    )
+    engine = create_engine(app.config[SQLALCHEMY_DATABASE_URI], convert_unicode=True)
     app.db_session = scoped_session(
         sessionmaker(autocommit=False, autoflush=False, bind=engine)
     )
     app.db.init_app(app)
 
     data_backend_class = SqlHandler
     data_backend_writer = SqlDataInventory
-    models_imports = importlib.import_module(models_path)
+    models_file_path = ".".join([app.config[CONFIG_FILE_FOLDER], "models"])
+    models_imports = importlib.import_module(models_file_path)
     app.Base = getattr(models_imports, "Base")
 
     @app.teardown_appcontext