Create a public dashboard for e-mission data

[shankari]

We've been thinking about creating a simple public dashboard for e-mission data. We are optimistic (although maybe we shouldn't be!) that multiple applicants will choose to use e-mission for their evaluation.

It would be great to display the status of the projects as they proceed and maybe even allow us to compare them!

Alas, I have no time to work on this project given that I will be making all other changes to all tiers of the systems.
We explored the idea of an intern to build a public dashboard in R/Shiny, similar to Amarin's deployer dashboard.

However, @asiripanich reveals that R/Shiny has a steep learning curve.

So I came up with a dead-simple workaround for now.

Then last night, I figured out the generalization of that approach.

• We will support only static graphs, not interactive
• The graphs will be represented by image files (e.g. purpose_pie__.png) and will be generated at pre-defined intervals (e.g. once a week)
• We will have a simple website where users can select the week that they are interested in, which will load all the images for that week.
• The image files can be generated by any program in any language, which allows us to get over arguments around which language to use. Python/R/node.js are all fine. The interns can write this code using ipython notebook or RStudio and export the resulting scripts.
• The only requirements are that the program can: (i) be automated, (ii) run on linux, and (iii) generate png outputs.

The static dashboard would be something like:
https://designrevision.com/demo/shards-dashboard-lite-react/blog-overview
or
https://nadavshaar.github.io/react-dashboard/

except with static images instead of charts

The scripts can run periodically on the server and upload data to GitHub or store on local disk or ???

@asiripanich @PatGendre @jf87 @xubowenhaoren:

• Does this approach make sense?
• Is it something you would find relatively easy to extend?

This time, I want to spend more time on choosing the technology so we are not stuck with a codebase that is difficult for contributors to work with.

After we see usage patterns, we can come up with a solution to add interactivity. But I want to get this done by the end of the week.

[shankari]

One other dashboard options:
https://github.com/plouc/mozaik

The problem with a lot of the options is that they are very complex, and use redux for storage etc.

Since we only want to display images, and don't need to store any data for now, I want to pick something super light weight.
I really like the idea of grid cards that users can add and resize because it allows for some interactivity even with our super simple approach. For example, users can show two images for the same metric side by side.

It looks like
https://strml.github.io/react-grid-layout/examples/6-dynamic-add-remove.html

Is a super simple example that gives us exactly that (one HTML + one JS). if we can figure out a way to make the edit button prettier, maybe by adding a header bar, we would be done.

[shankari]

It even has a "Drag from outside" option.
https://strml.github.io/react-grid-layout/examples/15-drag-from-outside.html

So as long as I can get a header bar of some sort to put the "Drag from outside button", and to give some choices around metric, week and org, we are fine.

[shankari]

looks like there are some fairly simple built-in? components we can use
https://reactnativeelements.com/docs/header/

[shankari]

@asiripanich makes another plug for R markdown as the dashboard
Docs: https://bookdown.org/yihui/rmarkdown-cookbook/html-tabs.html
Example: https://ivtmobis.ethz.ch/mobis/covid19/reports/mobis_covid19_report_en_2021-01-11.html

Estimated time for him to whip up those metrics is half a day. But you would need to use R :(

[PatGendre]

@shankari if I understand, e.g. for the la Rochelle pilot, we would just have to run a script each week/day and you would add our pilot metrics on the global e-mission dashboard, correct? That's a nice idea and seems very feasible:-)

As for which library to use, I am non competent, but I agree it should be as simple and limited as possible ; mozaik looks really nice but if used only for very basic stuff and not reused elsewhere in the project, it's maybe not worth investing.

[shankari]

@PatGendre I was thinking of having the metrics to compare groups for the eBike project, not necessarily a global dashboard.
You could run the scripts each week/day and update a dashboard for your deployment, similar to the old e-mission "metrics" page but with metrics that make sense for you. That should help the City of La Rochelle and its citizens keep track of how they are doing, and maybe encourage more participation.

wrt simplicity, my final comments focused on this
https://strml.github.io/react-grid-layout/examples/15-drag-from-outside.html

where the "drag from outside" button would have a day/week selector and a metric selector. users would select and then drag and drop so they can compare. Super easy; single webpage, no complications.

[shankari]

Recording some design decisions:

• using gridster instead of react-grid-layout since it is simpler to use. It uses jquery, so no need for JSX
• adapting it to a bootstrap layout/components so it looks prettier
• we will only support monthly stats initially since:
  • that is all that the CEO requires
  • it makes the selector static and super easy instead of fiddling around with date ranges for weeks in javascript.

[shankari]

wrt the viz scripts:

• we can support classic python, but we can also support ipython notebooks using the nbconvert option to automatically re-generate all graphs
• we need to decide if we are going to query the database directly or use requests to query a web server
  • using the database directly:
    • both existing modules (webserver and analysis server) talk directly to the database
    • plenty of nice interface functions that make this easier. In particular, we might be able to reuse large chunks of the existing metrics code (after refactoring) since it already handles queries and grouping
    • CON: hard to adapt to other languages without our specialized python drivers
  • connecting to the webserver using requests
    • easier to work with other languages since they don't have our specialized python drivers
    • more in line with our long-term goals of UPC since the data will need to be decrypted and then made available to the services
    • CON: current web interface does not support retrieving data without a user token. It would be trivial to remove that limitation, but that would be unwise since even the aggregated data will have times associated with each entry. If the times are removed, we won't be able to do "per day" metrics on the client, which seems like a serious limitation.
    • to fix this, we would need to write an internally visible API layer that does support retrieving aggregate data without a token

Given the short time frame, I am inclined to go with (1) and upgrade to (2) later when we get our first non-python user.

[shankari]

Simple and stupid dashboard prototype is done (@PatGendre @asiripanich)

code: https://github.com/e-mission/em-public-dashboard
example: https://dashboard.canbikeco.org/

As simple as possible:

• jQuery + bootstrap for the frontend
• raw ipython notebooks for the analysis

will record some other design decisions here tomorrow

[shankari]

the week specification (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/week)
defined in ISO 8601 (https://en.wikipedia.org/wiki/ISO_8601#Week_dates) is a good option to go for weeks, but it is not clear how well it is supported in the underlying date libraries.

[shankari]

moment supports it

moment("2021-W01")
Object { _isAMomentObject: true, _i: "2021-W01", _f: "GGGG-[W]WW", _isUTC: false, _pf: {…}, _locale: {…}, _a: (7) […], _d: Date Mon Jan 04 2021 00:00:00 GMT-0800 (Pacific Standard Time), _isValid: true, _z: null }

moment("2021-W02")
Object { _isAMomentObject: true, _i: "2021-W02", _f: "GGGG-[W]WW", _isUTC: false, _pf: {…}, _locale: {…}, _a: (7) […], _d: Date Mon Jan 11 2021 00:00:00 GMT-0800 (Pacific Standard Time), _isValid: true, _z: null }

arrow does not

>>> import arrow
>>> arrow.get("2021-W01")
<Arrow [2021-01-01T00:00:00+00:00]>
>>> arrow.get("2021-W02")
<Arrow [2021-01-01T00:00:00+00:00]>
>>> arrow.get("2021-W03")
<Arrow [2021-01-01T00:00:00+00:00]>

[shankari]

Apparently, the pendulum library does support it
https://stackoverflow.com/a/46369287/4040267

Yup, confirmed

>>> import pendulum
>>> pendulum.parse("2021-W01")
DateTime(2021, 1, 4, 0, 0, 0, tzinfo=Timezone('UTC'))
>>> pendulum.parse("2021-W02")
DateTime(2021, 1, 11, 0, 0, 0, tzinfo=Timezone('UTC'))
>>> pendulum.parse("2021-W03")
DateTime(2021, 1, 18, 0, 0, 0, tzinfo=Timezone('UTC'))

Maybe we should switch from arrow to pendulum over the long term...

[xubowenhaoren]

Sorry for joining the conversation late. I built a dashboard for AWARE (another mobile data collection platform). Some of its highlights:

• This is an interactive dashboard that provides real-time data quality visualization. It currently supports location and activity data for AWARE data collection.
• It contains a series of interactive graphs that visualizes the relationship between the update interval of the sampled data and the physical motion of the participant. This helps differentiate real motion data from GPS drifting.
• The Pandas powered analysis scripts take raw sensor data from the MySQL database and perform data cleaning, formatting, and generate location and activity histograms for participant data to verify data quality.
• The Plot.ly powered graphs interact with multiple user actions such as mouse & trackpad zooming, panning, and hovering.
• The Flask powered dashboard is responsive to user inputs such as UUIDs, the start and the end dates of the search query.
• It provides RESTful APIs to allow easy integrations to existing visualization dashboards.
• https://github.com/xubowenhaoren/AWARE-visualized

It's interactive in the sense that Plot.ly graphs themselves respond to user input. You can totally ignore the start & end date query/update to create pseudo-static representations.

[shankari]

@xubowenhaoren thanks for the update! I believe this would be closer to what we are calling the "deployer dashboard", similar to @asiripanich's emdash (https://github.com/asiripanich/emdash).

The public dashboard is different. It focuses on coarse, aggregate metrics that can be displayed without any privacy concerns. By design, it will not have any filters related to UUID, and will not visualize locations or individual-level histograms.