API proposal for create_dashboard() and update_dashboard()

[jackparmer]

Starting a discussion for a wrapper proposal for V2 dashboard creating and updating:

https://api.plot.ly/v2/dashboards#create

I'm not aware of any formal dashboard JSON spec documentation, but this dashboard is almost exhaustive in its options usage:

https://plot.ly/dashboard/jackluo:430/view
JSON: https://plot.ly/dashboard/jackluo:430/view.json

The only thing that it is missing is a table (an embedded plotly grid), which this dashboard has:

https://plot.ly/dashboard/jackp:16818
https://plot.ly/dashboard/jackp:16818.json

All of the options for styling embedded plotly grids (through URL query parameters) are here:
http://help.plot.ly/add-a-table-to-spectacle-editor/

Creating a dashboard

create_dashboard()

This call to create_dashboard would create the first 2 rows of this dashboard, plus a 3rd row with a table and a markdown cell:

https://plot.ly/dashboard/jackluo:430/view

# First row has 1 graph
row1 = list( dict( type='plot', url='https://plot.ly/~jackluo/400' ) )

# Second row has 2 graphs
# These dashboard cells have titles
row2 = list(
    dict( type='plot', url='https://plot.ly/~jackluo/402',
        title='Average rotor diameter by manufacturer' ), 
    dict( type='plot', url='https://plot.ly/~jackluo/404', 
        title='Number of turbines by manufacturer' ) )

markdown_text = ""## Jack Luo | Plotly\n\nDownload original dataset here:     \nhttp://www.nature.com/articles/sdata201560\n"

table = dict(
    type = 'table', 
    url = 'https://plot.ly/~datasets/2798',
    show_row_numbers = False,
    text_transform = 'uppercase',
    header_font_weight = 300,
    header_background = '%23ab63fa'
)

# Third row has 1 table and 1 text box
# All table styling options are here: http://help.plot.ly/add-a-table-to-spectacle-editor/
row3 = list(
    table,
    dict( type='text', text=markdown_text ), 
)

dashboard = dict(
    rows = list( row1, row2, row3 ),
    foreground_color = "#cccccc", 
    box_background_color = "#020202", 
    links = [ { "url": "http://www.nature.com/articles/sdata201560", 
                "title": "| Download dataset here" } ], 
    title = "US Wind Turbine Dataset", 
    box_border_color = "#020202", 
    header_foreground_color = "#cccccc", 
    header_background_color = "#151515", 
    background_color = "#151515", 
    logo_url = "https://astrogeology.usgs.gov/images/usgs_logo_main_2x.png", 
    box_header_background_color = "#020202"
)

dashboard_url = create_dashboard( dashboard )

Updating a dashboard's top-level attributes

update_dashboard()

Rewrite a top-level attribute of the dashboard (ie something in the "settings" key of https://plot.ly/dashboard/jackluo:430/view.json)

new_links = list( 
    dict( url = "http://www.nature.com/articles/sdata201560", 
            title = "| Download dataset here" ),
    dict( url = "#", title = "Last updated " + datetime_string ) )

update = dict( links = new_links )

update_dashboard( dashboard_url, update )

Appending or updating a dashboard row

update_dashboard_row()

Update or append one row at a time by passing the row index and

new_row = list(
    dict( type='plot', url='https://plot.ly/~jackluo/423',
        title='Blade Length' ), 
    dict( type='plot', url='https://plot.ly/~jackluo/425', 
        title='Rotor Density' ) )

# Rewrite the first row of the dashboard
update_dashboard_row( new_row, 1 )

# Append a new row to the dashboard
update_dashboard_row( new_row )

cc @charleyferrari @cldougl @theengineear @Kully @thejackluo @chriddyp

This was the simplest, no-frills API that I could think of, but there are probably better ways. 👍 if you think this looks good or chime in below to suggest some alternate proposals.

📉 📊 📈
🔢 📈 📊
📉 📊 🔢

[theengineear]

I'm not aware of any formal dashboard JSON spec documentation, but this dashboard is almost exhaustive in its options usage:

"first": {
    "boxType": "plot", 
    "shareKey": null, 
    "fileId": "jackluo:402"
}

Looks like this is versioned version: 2, which is great, let's make sure that we can bake that into the Python handling so that we future updates don't need to become backwards incompatible.

create_dashboard()

So far, we've taken the approach of object-based manipulation. The idea would be do something more like:

# Instantiate a new dashboard from scratch
dashboard = Dashboard()

# Instantiate a dashboard from url
dashboard = Dashboard(https://plot.ly/dashboard/jackluo:430)

# Methods to update a dashboard (just examples, not sure on the what methods we really need)
dashboard.add_row()
dashboard.add_column()
dashboard.update_cell(row, col, update)

Dev cycles

General question, thoughts on development cycles? Dashboards take quite a while to load if you've got a few plots, I think that we should have an offline integration here, right? I don't mean that we need to have an offline-mode for dashboards, but perhaps just a way to make layout updates and see a super simple preview of the expected layout when you do make the call to Plotly to update?

It'd be sweet if you could have a dashboard.manual_sync = True flag on instantiation that allows you to sync to plotly still, but it would only happen when you call dashboard.sync() or something. I just imaging having an annoying time making small tweaks to the layout on a dashboard if I have to wait for a dashboard page to load each time.

Second thought, we could also include a ?no-content flag or something on dashboards so that you can quickly iterate loading dashboard pages in your browser (or in the notebook) without needing to load Plotly and retrieve all the plots. Not sure how hard that would be, but I really question how on board folks will be if the dev cycles for pixel pushing layout boxes is super long.

Official dashboard JSON schema

We almost got there with plot-schema, perhaps we can just take the initiative and define an official JSON-schema for these dashboards? http://json-schema.org/ If we did that in streambed, we get all these amazing validation, generation tools for free in any language we choose (yay standardization --> https://pypi.python.org/pypi/jsonschema). This also assists us with versioning down the line since we just need to match versions to schemas and go from there.

^^ Note, this isn't super hard, the schema for the dashboards is fairly simple. I feel pretty strongly about this and I'd be happy to whip one up.

^^ Also note that I'm pretty sure there is about 0 validation on what goes into a dashboard:

POST https://api.plot.ly/v2/dashboards
{
	"content": "{\"rows\": [], \"banner\": {}, \"foobar\": \"anything i want...\"}"
}

Just works for example. We can do the same thing on the backend with jsonschema as we would in the python api library...

[theengineear]

@tarzzz , you wrote the api for dashboards, right? Why do we require the double-encoded content field there? Is there a reason? This is an inconsistency of ours that I don't really understand. You have to do it for /v2/grids/[fid]/cols as well, it'd be great if we could either accept something more like:

POST https://api.plot.ly/v2/dashboards
{
	"content": {"rows": [], "banner": {}}
}

Or, if we accepted both the above and:

POST https://api.plot.ly/v2/dashboards
{
	"content": "{\"rows\": [], \"banner\": {}}"
}

[tarzzz]

@theengineear : in DRF, JSONField accepts a binary argument. When set to True, it saves (and retrieves) the data as a JSON string. If set to False, it saves(and retrieves) data as a python primitive (i.e. dict in this case). It defaults to False, and we cannot have both (obviously).

That said,

• We can override default validator to validate for both dicts, and JSON strings. I can add an issue for this if you agree..
• How should we save the data to DB? Should it be as a python primitive (i.e. binary=False) or as a coded JSON string(i.e. binary=True). I am not sure if it matters much how we save it to DB. Let me know your thoughts on same..

[theengineear]

How should we save the data to DB?

It should get saved to the DB in the same way it's getting saved right now.

We can override default validator to validate for both dicts, and JSON strings. I can add an issue for this if you agree..

👍 Let's definitely open up an issue, I do find it fairly confusing that there are just a couple endpoints that require this different upload format.

[charleyferrari]

Chris put together a walkthrough that has been very useful: https://plot.ly/~chris/17759/

Rather than being arranged in rows, dashboards are arranged in parent-child relationships. This is probably a more explanatory example of what's going on: https://plot.ly/dashboard/charleyferrari:1281/present

Check the titles to see how they're nesting into each other. F and S refer to first and second, so the simplified json structure is something like this:

{
    'first': {
        'first': 'graph F-F',
        'second': {
            'first': 'graph F-S-F',
            'second': {
                'first': {
                    'first': {
                        'first': 'graph F-S-S-F-F-F',
                        'second': 'graph F-S-S-F-F-S'
                    },
                    'second': 'graph F-S-S-F-S'
                },
                'second': 'graph F-S-S-S'
            }
        }
    },
    'second': 'empty' // there's always a second empty node here
}

Are you guys planning on abstracting away this structure with this API? Could be a good idea, but I think it's complicated...

With drag and drop this structure "just works", but if someone were building a dashboard programmatically from scratch, I don't think they'd be imagining these sorts of nodes. They'd be thinking in rows or columns, perhaps with horizontal or vertical (respectively) splits in individual cells.

I could see two ways of doing this:

1: We keep the structure as is, encourage users to create a template dashboard using our drag and drop interface, and make it easy for them to swap in new graphs. So, make it easy to do things like update_dashboard_row() (perhaps making this update_dashboard_cell()), or create an analog to something like py.get_figure() for dashboards that allows users to easily bring the dashboard in and just swap out the plot URLs.

2: We abstract away this structure in the API, focusing on building something with rows and columns. IMO, if a user is building a dashboard programmatically from scratch, this is the way they're thinking about it.

I think this is a tradeoff unfortunately. If we abstract the API away it makes it easier to work with from scratch, but harder to use an existing dashboard as a template.

[theengineear]

I'm thinking the JSON Schema should look something like:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Plotly Dashboard JSON Content",
    "description": "A JSON representation of the content in a Plotly Dashboard",
    "type": "object",
    "properties": {
        "layout": {"$ref": "#/definitions/container"},
        "settings": {
            "type": "object",
            "properties": {
                "backgroundColor": {"$ref": "#/definitions/color"},
                "boxBackgroundColor": {"$ref": "#/definitions/color"},
                "boxBorderColor": {"$ref": "#/definitions/color"},
                "boxHeaderBackgroundColor": {"$ref": "#/definitions/color"},
                "foregroundColor": {"$ref": "#/definitions/color"},
                "headerBackgroundColor": {"$ref": "#/definitions/color"},
                "headerForegroundColor": {"$ref": "#/definitions/color"},
                "links": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "title": {"$ref": "#/definitions/shortText"},
                            "url": {"$ref": "#/definitions/url"}
                        },
                        "additionalProperties": false
                    }
                },
                "logoUrl": {"$ref": "#/definitions/url"},
                "title": {"$ref": "#/definitions/shortText"}
            },
            "additionalProperties": false
        },
        "version": {"type": "integer", "minimum": 0, "maximum": 1000}
    },
    "additionalProperties": false,
    "definitions": {
        "color": {
            "oneOf": [
                {"$ref": "#/definitions/colorHex"}
            ]
        },
        "colorHex": {
            "type": "string",
            "pattern": "^#([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$"
        },
        "container": {
            "type": "object",
            "properties": {
                "size": {"$ref": "#/definitions/size"},
                "sizeUnit": {"$ref": "#/definitions/sizeUnit"},
                "direction": {
                    "type": "string",
                    "pattern": "^(vertical|horizontal)$"
                },
                "first": {
                    "oneOf": [
                        {"$ref": "#/definitions/container"},
                        {"$ref": "#/definitions/leaf"}
                    ]
                },
                "second": {
                    "oneOf": [
                        {"$ref": "#/definitions/container"},
                        {"$ref": "#/definitions/leaf"}
                    ]
                },
                "type": {
                    "type": "string",
                    "pattern": "^split$"
                }
            },
            "additionalProperties": false
        },
        "leaf": {
            "type": "object",
            "properties": {
                "size": {"$ref": "#/definitions/size"},
                "sizeUnit": {"$ref": "#/definitions/sizeUnit"},
                "title": {"$ref": "#/definitions/shortText"},
                "text": {"$ref": "#/definitions/shortText"},
                "fileId": {
                    "type": "string",
                    "pattern": "^[.a-zA-Z0-9_-]+:(?:[0-9]|[1-9]\\d+)$"
                },
                "boxType": {
                    "type": "string",
                    "pattern": "^(empty|plot|text)$"
                },
                "shareKey": {
                    "oneOf": [
                        {"type": "null"},
                        {"$ref": "#/definitions/shortText"}
                    ]
                },
                "type": {
                    "type": "string",
                    "pattern": "^box$"
                }
            },
            "additionalProperties": false
        },
        "shortText": {
            "type": "string",
            "maxLength": 200
        },
        "size": {
            "type": "integer",
            "minimum": 0
        },
        "sizeUnit": {
            "type": "string",
            "pattern": "^(px|%)$"
        },
        "url": {
            "type": "string",
            "maxLength": 2000
        }
    }
}

Needs cleaning up, but the recursive functionality of the container was the thing I wanted to show. definitions actually work really well in JSON Schema :)

[theengineear]

Looks like there may be motivation to do JSON schema versioning with 1-0-0-style formats:

Interesting read on self-describing schemas:

http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/

Not terribly worried about it, but something to consider.

[theengineear]

We could try to share definitions amongst all of our api endpoints (and perhaps that's the right way to go), but I'd prefer just creating a GET /v2/dashboards/schema endpoint to expose the schema we'd be validating with. For a first pass (and maybe all we need to do), is to just go ahead and return exactly what our backend says to the caller if the validation fails on our backend.

[theengineear]

@Kully Here's a first pass at a full spec for the dashboard wrapper. What do ya think?

# plotly/dashboard_objs/dashboard_objs.py

class Container(object):

    def __init__(self, box0, box1, split_direction):
        pass


class Box(object):
    pass


class PlotBox(Box):
    pass


class EmptyBox(Box):
    pass


class TextBox(Box):
    pass


class WebpageBox(Box):
    pass


class Dashboard(object):

    fid = None

    def __init__(self, content):
        self.content = content

    def _get_box(self, box_id):
        box_id_map = self._get_box_id_map()
        path = box_id_map[box_id]
        return {}  # Use the path to get a box and return it.

    def _get_box_id_map(self):
        # This should map machine-readable ids based on splits to an integer
        # box number that a user can easily understand.
        #

        #
        # You should traverse the dashboard content using
        # `utils.node_generator` or something (as long as it's consistent) and
        # just create a little list as you go and return it.
        # return
        pass

    def get_preview(self, preview_type='html'):
        # Should be able to copy ViewLayoutNode:ViewLayoutSplit logic?
        # Importantly an ID should be shown in each box.
        # Say you have the following dashboard:
        # {
        #     "layout": {
        #         "direction": "vertical",
        #         "first": {
        #             "direction": "vertical",
        #             "first": {"boxType": "text", "text": "Text.",
        #                       "title": "text title", "type": "box"},
        #             "second": {
        #                 "direction": "horizontal",
        #                 "first": {"boxType": "plot", "fileId": "local:126",
        #                           "shareKey": null, "title": "plot title",
        #                           "type": "box"},
        #                 "second": {"boxType": "plot", "fileId": "local:108",
        #                            "shareKey": null, "title": "plot title",
        #                            "type": "box"},
        #                 "size": 50,
        #                 "sizeUnit": "%",
        #                 "type": "split"
        #             },
        #             "size": 150,
        #             "sizeUnit": "px",
        #             "type": "split"
        #         },
        #         "second": {"boxType": "empty", "type": "box"},
        #         "size": 1150,
        #         "sizeUnit": "px",
        #         "type": "split"
        #     },
        #     "settings": { .. },
        #     "version": 2
        # }
        #
        # For reference, the layout looks like this:
        #
        #  ------------------------------------------------------------------
        # |                                                                  |
        # |                               0                                  |
        # |                                                                  |
        # |------------------------------------------------------------------|
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        # |               1               |                  2               |
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        #  ------------------------------------------------------------------
        #
        # I'd suggest relating the numbers above to an array of paths locating
        # the "leaf" boxes:
        # [
        #     ['first', 'first'],            # 0
        #     ['first', 'second', 'first'],  # 1
        #     ['first', 'second', 'second']  # 2
        # ]
        #
        # Note that I'd rather we *hide* the notion of a container box from
        # users. I've thought a fair amount on this, and I ultimately think
        # that containers are simply an implementation detail. It's like a
        # `div`, just organizational, but it's not what we really want to force
        # users to interact with.
        pass

    def insert(self, box_id, box, side):
        # This do something like...
        original_box = self._get_box(box_id)
        if side == 'right':
            container = Container(original_box, box, 'vertical')
        elif side == 'left':
            container = Container(box, original_box, 'vertical')
        elif side == 'below':
            container = Container(original_box, box, 'horizontal')
        elif side == 'above':
            container = Container(box, original_box, 'horizontal')
        else:
            raise Exception('Boom')
        self.replace(box_id, container)
        pass

    def insert_right(self, box_id, box):
        self.insert(box_id, box, 'right')

    def insert_left(self, box_id, box):
        self.insert(box_id, box, 'left')

    def insert_above(self, box_id, box):
        self.insert(box_id, box, 'above')

    def insert_below(self, box_id, box):
        self.insert(box_id, box, 'below')

    def move_insert_right(self, source_box_id, destination_box_id):
        box = self._get_box(source_box_id)
        self.remove(source_box_id)
        self.insert_right(destination_box_id, box)

    def move_insert_left(self, source_box_id, destination_box_id):
        box = self._get_box(source_box_id)
        self.remove(source_box_id)
        self.insert_left(destination_box_id, box)

    def move_insert_above(self, source_box_id, destination_box_id):
        box = self._get_box(source_box_id)
        self.remove(source_box_id)
        self.insert_above(destination_box_id, box)

    def move_insert_below(self, source_box_id, destination_box_id):
        box = self._get_box(source_box_id)
        self.remove(source_box_id)
        self.insert_below(destination_box_id, box)

    def replace(self, box_id, box):
        # Simply replace the box at box_id with the new box.
        pass

    def remove(self, box_id):
        # Remove the box at box_id, this will likely need to cleanup an empty
        # container box as well.
        pass

    def rotate(self, rotation_direction, box_id):
        # This should change the split in the parent container of box_id.
        pass

    def rotate_clockwise(self, box_id):
        self.rotate('clockwise', box_id)

    def rotate_counterclockwise(self, box_id):
        self.rotate('counterclockwise', box_id)

    def swap(self, box_id):
        # switch which box comes first in the pair.
        pass

    def resize(self, box_id, direction, size, size_unit):
        # This should alter props on the parent container.
        pass

    def create(self):
        # Not sure on the call signature for this, it needs privacy/filename
        # opts.
        pass

    def update(self):
        # Not sure on the call signature for this, it needs privacy/filename
        # opts.
        pass

^^ The underlying format for the dashboard is super clean, it's just a little hard to manage. I've attempted to boil things down into a set a base actions that bind closely to the underlying format and then construct some more user-friendly actions insert_move_right, etc, etc, that are just compositions.

@charleyferrari, I don't want to abstract away the underlying implementation too much. We'll have fewer headaches the thinner we make this. If folks want something schmancier down the line, we can just build a later ontop of this implementation that does more magic. The current format is perfectly reasonable, it's just a little confusing at first pass. I'd rather we just use names like insert, remove, swap, etc, etc, so that folks don't need to grok the long/confusing ['first', 'second', 'first', 'first, ... ].

@jackparmer @Kully @charleyferrari , I think a very important part of this to get right is the visual feedback. This will all feel pretty natural (imo) as long as the flow for users is some thing like:

# create a dashboard
# look at the current layout...
# add a box to the dashboard
# look at the layout *now*...
# add another box
# look at the layout *now*...
# move a box
# look at the layout *now*...

As long as the visual feedback is good and fast, I think this will be a pretty fun experience.

Also, let's hold off on offline mode for now, as a first pass let's just force folks to pass in a url or a fid for plots that are to be added.

Also, also, I spaced on the settings portion, that will be required somewhere as well. I just want to get this out before I head to bed.

[theengineear]

@Kully let's iterate on this, I'd love to hear your thoughts. Note that I'm adding that JSON schema so that you won't need to worry about any sort of validation since the backend will take care of that. Dashboards are relatively small objects that can be passed over the network pretty fast, so there's not really a gain to try and validate locally (plus, we should be controlling the actual object and won't get it wrong). Here's the PR for that https://github.com/plotly/streambed/pull/8989

[Kully]

@theengineear Okay, all this sounds really good. Chris wrote a script for using the API to make dashboards so I'll refer to that as an aid as I go through this.

I'm confused about the paths you mentioned for tracking a specific box in a dashboard layout:

['first', 'first'],            # 0
['first', 'second', 'first'],  # 1
['first', 'second', 'second']  # 2

What is it based on? Was this in place before we opened up this thread?

[theengineear]

^^ Those are literally the paths to the boxes in question:

        # {
        #     "layout": {
        #         "direction": "vertical",
        #         "first": {
        #             "direction": "vertical",
        #             "first": {"boxType": "text", "text": "Text.",
        #                       "title": "text title", "type": "box"},
        #             "second": {
        #                 "direction": "horizontal",
        #                 "first": {"boxType": "plot", "fileId": "local:126",
        #                           "shareKey": null, "title": "plot title",
        #                           "type": "box"},
        #                 "second": {"boxType": "plot", "fileId": "local:108",
        #                            "shareKey": null, "title": "plot title",
        #                            "type": "box"},
        #                 "size": 50,
        #                 "sizeUnit": "%",
        #                 "type": "split"
        #             },
        #             "size": 150,
        #             "sizeUnit": "px",
        #             "type": "split"
        #         },
        #         "second": {"boxType": "empty", "type": "box"},
        #         "size": 1150,
        #         "sizeUnit": "px",
        #         "type": "split"
        #     },
        #     "settings": { .. },
        #     "version": 2
        # }
        #
        # For reference, the layout looks like this:
        #
        #  ------------------------------------------------------------------
        # |                                                                  |
        # |                               0                                  |
        # |                                                                  |
        # |------------------------------------------------------------------|
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        # |               1               |                  2               |
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        # |                               |                                  |
        #  ------------------------------------------------------------------
        #
        # I'd suggest relating the numbers above to an array of paths locating
        # the "leaf" boxes:
        # [
        #     ['first', 'first'],            # 0
        #     ['first', 'second', 'first'],  # 1
        #     ['first', 'second', 'second']  # 2
        # ]

See how the zeroth box is the object that can be located at dashboard['layout']['first']['first'] and the first box can be located at dashboard['layout']['first']['second']['first'] and the second at dashboard['layout']['first']['second']['second']?

[theengineear]

I'm just saying that users shouldn't need to know about the first/second/second/first path and whatnot, they should just have to interact with a number.

That said, do you think it'd be confusing that the numbers will change? It's by far simpler than trying to assign and track permanent ids, but users will have to get used to the fact that when you split a box, the numbers are going to move around.

[Kully]

@theengineear
Every box is then put into 'first' in order to be put in the layout right?

i.e. dashboard['layout']['first'] is just a statement that box appears in the full frame

[theengineear]

In theory layout is a split, which means it can contain two the objects first and second, either of which may be a split or a box.

The dashboard schema may be useful here.

In the schema split === container and box === leaf. I.e., a box cannot have first and second, but a split can. Likewise, a split only acts as a container for other containers and/or boxes.

TL;DR...

Every box is then put into 'first' in order to be put in the layout right?

Is this to ask if layout['second'] must always be an empty box?

i.e. dashboard['layout']['first'] is just a statement that box appears in the full frame

I don't really understand this comment since there's no way to know if first is a box/leaf or a split/container.