Dyns

Declarative and dynamic web based dashboards in python, integrated with pandas.

It is in early alpha, active development state. APIs are subject to change.

Intro

Dyns (for dynnamic dashboards) easily integrates into arbitrary data pipelines to declaratively create interactive, responsive, and dynamic web based charts and dashboards. Create forms for dynamic execution of python code, and dynamic tables and charts. A dashboard and all of its components are defined in python, and all data and information used to generated charts, tables, and other visualizations come directly from python.

The end-to-end python for dashboard creating is perspective shift from how most of the popular front end business intelligence tools function (microstrategy, lookr, tableau, caravel, etc.). They mostly sit on top of a database/datastore (SQL, spark, druid) and do any manipulation, wrangling, and slicing via the front end gui. Dyns connects directly to python and pandas, and allows you to do the analysis, slicing, and dicing in python – though you could also set this up to behave like the popular front end BI tools with something like sqlalchemy.

Dyns makes no assumptions about data source, structure, or how it should be analyzed. Ultimately you just need to be able to to supply pandas series’ and dataframes to visualization objects defined in the library, then declaratively layout how you want them to be rendered / interacted with, all in python. The aim is to abstract away all front end and back end web development by default (though it’s also possible to customize any layer of the webapp if you want). Focus on the data, and let the library turn it into a responsive sufficient-looking webapp by default for you.

Dyns uses Bootstrap on the front end so dashboards should be responsive and looking good on computers, phones, and tablets.

Getting Set Up

Dyns works with python 2 and 3.

Clone the repo with

git clone https://github.com/tdrobbin/dyns.git

Then cd to it, and install via pip with

pip install -e .

If you are upgrading an already installed version, use:

pip install -U -e .

Alternatively, you can manually install dependancies via pip, and import dyns in python after you have cd'd into the cloned git repo.

Current dependancies are:

• pandas
• numpy
• plotly
• cufflinks
• flask
• wtforms
• ipython (for cufflinks)
• waitress (optional)

Usage

Quickstart

Create the following file titled example1.py:

import numpy as np
import pandas as pd
from dyns import Dashboard, Page, Table

first_page = Page(
    title='First Page',
    renderers=[
        Table(f=pd.DataFrame, f_kwargs={'data': np.random.randn(10, 4)}, title='Data')
    ]
)

dbaord = Dashboard(
    title='My Dashboard',
    pages=[
        first_page
    ]
)

if __name__ == '__main__':
    dbaord.run_app('dev')

Then run it at the command line:

python example1.py

After a few moments for initialization, a new windows in your default web browser should open at http://localhost:5000 showing the dashboard. You can also navigate there in your web browser manually.

Going from the bottom of example1.py up we see that we are declaring and running a Dashboard object to contain one Page, namely first_page, with a dashboard title of My Dashboard. Above that, a Page object is declared to contain just one Table object and to be called First Page.

The Table object is a subclass of PandasRenderer. PandasRenderers know how to take pandas DataFrames and Seriess and turn them into tables and charts using plotly and cufflinks. Here we are only creating a table, but see below for charts, and other PandasRenderers.

All PandasRenderers have the same API for working with DataFrames and Seriess. They take some function f (f can actually be any python object that supports __call__), and a dictionary f_kwargs. f is some function (or python callable) that takes parameters, and returns a DataFrame or a Series. f_kwargs is a dictionary of parameters to be passed to f.

In example1.py, our callable is pd.DataFrame, as this will return a DataFrame, and we are passing np.random.randn(10, 4), a 10 x 4 numpy array of random numbers, to the DataFrame data parameter.

Dashboard Basics

At the highest level, there is the Dashboard object. The Dashboard object contains pages and each page contains renderers. Renderers are the basic visualization objects (charts, tables, custom defined renderers). The most useful renderers are CufflinksRenderer, DataTableRenderer, DynsForm

All code below assumes the following:

import pandas as pd
import numpy as np
from dyns import * # obv dont do this * nonsense IRL

Dashboard: holds everything, configures all the app routing, and provides a wsgi app you can use to run however you want. Can pass custom css and js files (so you can add javascript libraries pretty easily – see below). At bare minimum takes a list of Page objects and A title

dboard = Dashboard(
    title='Dashboard',
    pages=[
        page1,
        page2,
        page3
    ]
)

Once you have your dashboard object you can run it directly using flask's dev server or using the pure python waitress library server. You can also access the WSGI app object by calling get_app

# run the dev server
dboard.run_app('dev')

# run using waitress
dboard.run_app('prod')

# get the wsgi app
dboard.app

Page: pass these in a list to a Dashboard. At a minimum, takes a title and list of renderers:

welcome_page = Page(
    title='Welcome to dyns',
    renderers=[
        HTML(title='Hello World!', markup='<p>This is some HTML!</p>'),
        Table(f=pd.DataFrame, f_kwargs={'data': np.random.randn(10,4)}, title='Some Random Data'),
        CFLineChart(f=pd.DataFrame, f_kwargs={'data': np.random.randn(10,4)}, title='Some More Random Data'),
    ]
)

Renderers

Renderers are passed to page objects, and sometimes to other renderers. Within a page renderers are used to display content in dashboards. The Renderer class is abstract, and every subclass needs to implement an _inner_html method that must return valid html. The html from this function call will be used to render the objects in the dashboard. It is easy to subclass Renderer to implement your own custom visualizations and content. dyns provides the following useful implemented renderers:

HTML: takes an argument markup which can be any valid html string, and will display in dashboard:

HTML(title='Hello World!', text='<p>This is some HTML!</p>')

PandasRenderer: Base class for all renderers that want to display pandas objects. All PandasRenderers have the same API for working with DataFrames and Seriess. They take some function f (f can actually be any python object that supports __call__), and a dictionary f_kwargs. f is some function (or python callable) that takes parameters, and returns a DataFrame or a Series. f_kwargs is a dictionary of parameters to be passed to f

Table: subclass of PandasRenderer that just turns a dataframe or series into a table. Can apply various boostrap-based formatting options. See signature for available args.:

Table(f=pd.DataFrame, f_kwargs={'data': np.random.randn(10,4)}, title='Some Random Data')

CFRenderer: uses the plotly and cufflinks packages to turn pandas Series and DataFrames into visualizions in the dashboard. Cufflinks is a package that binds plotly to to DataFrames, so you can call iplot on DataFrames and generate interactive plotly graphs. You can see how plotly and cufflinks work together here: https://plot.ly/ipython-notebooks/cufflinks/. The CFRenderer object takes a dataframe or series, a title, a chart type, and any keyword args, as cf_kwargs that a the cufflinks plot function takes (look at the output of In[5] in this notebook: http://nbviewer.jupyter.org/gist/santosjorge/f3b07b2be8094deea8c6).

For your convenience there are a few subclasses of CFRenderer for common visualization types, so that you don't need to pass anything to plotly and cufflinks via cf_kwargs. We currently support the following:

• CFLineChart
• CFBarChart
• CFScatterChart
• CFPieChart
• CFHeatMap

They all have similar APIs, and this is an example of creating a Line Chart:

CFLineChart(f=pd.DataFrame, f_kwargs={'data': np.random.randn(10,4)}, title='Some More Random Data')

DataTable: uses the jquery datatables library to view dataframes (see here: https://datatables.net/). Enhances basic html tables to make them searchable and paginated:

DataTable(f=pd.DataFrame, f_kwargs={'data': np.random.randn(10,4)}, title='Some Random Data'),

Note that DataTables cannot currently handle more than a few hundred rows in a table

All renderers mentioned above display single content items within the page, whether the content is a chart showing dataframes and series or custom markup. Additionally, there are renderers that can group the single content renderers into visually grouped structural components in the page. These renderers are Group Renderers and all share a similar API, whereby they need a title, and a list of renderers that will be grouped. Currently the following groups are supported: HorizontalGroup, VerticalGroup, TabbedGroup. The TabbedGroup can take a list, in which case the titles of the renderers will be the titles of the tabs, or it can take a dictionary where keys are tab titles, and values are renderers. If passing a dictionary, the values can also be lists in which case there will be multiple renderers per tab. HorizontalGroup and VerticalGroup only accept a list to the renderers arg.

Forms and Dynamic Renderers.

dyns provides the means to create interavtive and dynamic dashboards. DynsForms declare fields in a form of various types (e.g. boolean, int, string, multiple options), and Renderers that will take user submitted input from the forms. The wtforms library is used to define the fields of the form. When defining renderers to be dynamic to the responses of a form, you request the submitted form value with a FormAccessRequest:

def rand_df(rows=10, cols=4):
    return pd.DataFrame(np.random.randn(rows, cols))

DynsForm(
    title='Some example Renderers',
    fields=dict(
        columns=FloatField('Columns', default=10),
        rows=FloatField('Rows', default=4),
        bmk=SelectField('Compare Against Benchmark?: ', choices=[(None, 'None'),('^GSPC', 'S&P 500')]),
        # multi=SelectMultipleField('Compare Against Benchmark?: ', choices=[(1,1),(2,2)])

    ),
    renderers=[
        CFLineChart(
            f=rand_df,
            f_kwargs=dict(
                cols=FormAccessRequest('columns')
            )
        ),
        CFBarChart(
            f=rand_df,
            f_kwargs=dict(
                rows=FormAccessRequest('rows'),
                cols=FormAccessRequest('columns')
            ),
            css_id='custom-id',
            css_class='cf-bar-custom',
            title='hey'
        ),
    ]
)

The Form 'Some example Renderers' will have 2 fields. One will be displayed in the dashboard as 'Columns', but will be available to other renderers as columns. It is expecting a float and has default value of 10. The row field is similarly defined. The third argument this form takes is a list of renderers. The renderers in here can by any valid renderer. Here we are using CF renderers and they will display data that is varied based on the input to the forms.

The first renderer, a CFLineChart, will call rand_df - which will return a dataframe of row rows and col columns. We want the cols argument to be dynamic based on the user input, so we pass the result of the colums field in the form it belongs to. Recal that columns is the corresponding key to the FloatField that will have a title Columns in the Dashboard, within the dict that defines the fields in the form. We Similary define a dynamic Bar Chart that will have the number of rows and columns as specified by the user in the dashboard.

When this form is rendered in the dashboard, it will show only the fields defined in the fields argument to the form. When the form is submitted by the user, an ajax call is made and the forms passed in the renderers parameter will be computed and displayed. It is at this point (when the form is actually submitted) that the Renderers will call their f using f_kwargs with any possible parameters passed in via the Form on the front end. This delay in calling f untill the renderer actually needs to be displayed occurs with all PandasRenderers regardless of whether or not they are a part of forms.

Customization

It is very easy to extend dyns to add your own renderes. You just need to subclass the desired renderer (most likely PandasRenderer) then include your library in header_js_paths, or footer_js_paths when inititalizing the dashboard. At the minimum you need to implelment a _inner_html function that just needs to return valid html which will become the html for that renderer. In this way it is very easy to add js libraries and write the derived renderer to render them.

Take a look at CustomHTMLRenderer for an example of creating your own general renderers (though this doesn’t subclass PandasRenderer). Also take a look at the source for the Table class to see how to subclass a PandasRenderer.

Help

The examples folder shows dashboard examples to varying degrees of complexity and interactivity.

For any questions or help, feel free to reach out at tylerdrobbin@gmail.com