Home

Welcome to the Chart Builder development wiki!

The purpose of this wiki is to provide a narrative of the development work carried out to create the chart builder. It documents the development process, design considerations, technology choices, other contextual information and links to resources. The wiki is intended as a technical reference for future developers and other project stakeholders.

Chart Builder - Objectives

Chart builder is a proof-of-concept (POC) tool intended to allow users to publish statistical charts from a variety of data sources. The project is in an innovation phase and will go through a series of design iterations layering in additional features and capability.

Once complete the POC will be integrated into a Content Management System (Plone) via its front-end publishing environment (Volto). This integration will be either as a Volto Add-On or as a direct React component integration.

A specific list of objectives for the POC are:

User Interface and User Experience

The UI/UX design is specified here:

https://www.figma.com/file/84XELgGlZTrF83oxXrZEX1/Chart_Builder

and here:

https://5b8wy6.axshare.com/#id=nbwhwx&p=chart_builder

We have created a space optimised layout for the chart builder reducing as much unnecessary content as possible. Chart builder can be characterised very much as a web application vs a website or webpage.

The main window has has two key features:

A chart configurator (left) which allows a user to specify a data source, data series, chart type and other chart preferences/stylistic aspects
A chart preview (right) which shows the user a preview of the chart they have created

Online demo

An online demo of chart builder is hosted on Google Cloud Platform:

https://chart-builder-no4vxskx7a-nw.a.run.app/

Development log

Our starting point was to understand the objectives for the Chart Builder project, who the intended users are, what are some of the basic constraints and how we could separate the work into distinct phases delivering useful iterations of the product at each phase.

We quickly understood that there are potentially a very large number of factors to consider in designing and implementing the chart builder.

Key ways we can manage the complexity are:

Fast iteration and feedback cycles between design and development
Start with a small set of chart types and properties
Consider how we can keep the chart configurator aspect of the UI clean by limiting the overall number of visible choices shown at a time
Identify natural sequences of user actions (e.g. select data first, select chart type second, select chart subtype next, etc.)
Attempt to constrain the set of chart configurations to a useful subset of the total possible configurations
Explore a diverse set of example data to understand how the shape and extents of different datasets vary
Identify design principles and statistical best practice we can encapsulate into the published charts and thereby constrain the chart configuration choices

Chart design and best practice

Our primary source of guidance on chart design best practice is the following ONS guides:

https://style.ons.gov.uk/category/data-visualisation/

https://style.ons.gov.uk/category/data-visualisation/interactives-chart-animations-dashboards/

We are also drawing directly on data visualisation expertise within our team and from experts within ONS.

Extensible design of the chart properties configurator

We have implemented an extensible design for the chart properties configurator following the open for extension, closed for modification principle. This uses a declarative JSON schema to define which chart properties the user may configure and then maps the users selections through to chart layout specification that is passed to the charting library.

Publish to image feature

As an initial end-to-end proof of concept we have implemented a publish to image feature for the chart builder. This uses the dom-to-image library and renders all content within the chart-preview element into a .png image file and automatically saves the output.

https://github.com/tsayen/dom-to-image

Responsive chart vs user specified dimensions chart

During the discovery phase we have explored options for implementing both user specified dimension charts and also responsive charts. With responsive charts we found that plotly.js doesn't always render these properly, for example the chart legend will overlay the x axis tick labels when the chart is sized less than 250px in height.

We are working with the UI/UX team to define a set of size constraints for charts that keeps the rendered chart to a high standard of presentation. Plotly also offers chart properties that allow x and y offsets of chart elements to be specified which may be useful to provide to the user to override issues such as the legend intersecting with the x axis.

Support for mobile devices

There are several aspects to support for mobile users:

We have to consider the publisher type user who will view the chart builder and the chart preview
We also have to consider the end user (consumer) of the published charts
A further dimension to consider is on the accessibility of inputs to touch screen devices (size of the inputs and touch specific event handling).

Our initial implementation is focussing on proving out the core functionality of the chart builder, as we progress then progressively greater support will be added for mobile devices.

CSV file loading

We have implemented support for loading data from a CSV file into the chart builder. This was a natural first step in getting data ingested into the application because of the relative ease of implementation. CSV is also still a popular format for holding data amongst the GSS community and is a supported download format from data.gov.uk

For convenient UX we have implemented a drag-drop zone for csv files that occupies the empty state space of the empty chart preview before it has data loaded.

Tidy data support

We have implemented support for tidy data which is the IDS standard for storing datasets. As a part of this support we have also implemented a UI feature for the user to specify which columns in the data are the category axis, measure of interest and dimension of interest. The user may then select Y series from those available within the dimension of interest.

Title, statistical summary, source and footnotes

Chart title, summary, source and footnotes have special treatment within chart builder in that they are not included in the properties configurator. These textual aspects of the chart will likely be input by text 'blocks' within the CMS. The exact relationship between these elements and the main chart have yet to be explored as we push forward with the CMS integration.

In the initial standalone chart builder we have included these elements as disconnected components (functionally separate from the chart builder) that show as blocks of editable text much as they will do in the CMS.

Evaluation of plotly.js as the chart rendering technology

In awareness of the limited time resource for the initial development phase of chart builder we have opted to use plotly.js as the charting technology to render the charts. Plotly was also included in a previous code-base within the organisation (the climate change v1 dashboard) and so formed a useful reference resource.

Integration with Plone-Volto Data Driven CMS

Objective

Our objective is to integrate the Chart Builder into the Plone-Volto Data Driven Content Management System (DD-CMS) to allow users to easily publish charts along with other content types into pages, articles, and other document types.

The integration encompasses several dimensions:

Understanding the technical requirements and constraints of integrating with Volto
Defining the responsibilities and separation of concerns of data sources and publication outputs
Adapting the Chart Builder source code to work with Volto

Plone-Volto overview

Plone is a content management system written in Python.

Volto is an extensible React front-end for Plone.

Reference implementations

We looked to two reference implementations to guide us:

volto-plotlycharts Add-On - a similar chart building tool but many more features than our requirements
Climate change dashboard v1 - a reference site we aim to replicate in the DD-CMS to prove out it's ability to present charts and related content to publication standard

Volto Add-Ons

A common approach to extending Volto is to create Volto add-ons.

The following tutorials are useful:

Key concepts and considerations of volto add-on development

Volto provides Server Side Rendering (SSR) - as such any code or dependencies that are only able to run in the browser should be code split to prevent them running on the server
SSR is managed by Razzle. - creates 2 configs, 2 ports, 2 webpack instances, both watching and hot reloading the same filesystem, in parallel during development
The Volto project plus add-ons act collectively as a monorepo
yarn is used to manage dependencies for the main volto project and its add-ons
Common dependencies with matching versions will be hoisted up to the main volto project
Volto add-ons are configured as yarn workspaces so that volto knows to install their dependencies

How we created the volto-chart-builder add-on

Notable points on the implementation are:

We created the volto-chart-builder add-on using the procedure outlined in the Volto add-ons tutorial
The location of the dd-cms repo is https://github.com/GSS-Cogs/dd-cms
The add-on was created using a code generator tool called Yeoman (yo)
The add-on is named volto-chart-builder and is located in volto/src/addons
The add-on wraps the chart-builder allowing chart builder to continue to be developed as a standalone project in its own repository
We use mrs-developer to sync the chart-builder source code code into volto/src/addons on build
The volto-chart-builder add-on is registered in the addons section of volto's package.json
The volto-chart-builder add-on is also registered in the workspaces section of volto's package.json
A separate branch of chart-builder, volto-integration, is being maintained for the volto add-on. This is required because some of the styling in the standalone chart builder clashes with styling in volto

Tips on working with the solution:

Docker can be used to conveniently run and debug both Plone and Volto (both must be running for the solution to load)
Hot Module Reloading (HMR) is available on both client and server
The compile cycle after code changes can be quite lengthy
It is possible to restart the Volto container without restarting Plone using the restart button in Docker Desktop
Use the React dev tools to inspect the state of volto components
Use yarn develop to sync the chart-builder source code into the volto monorepo
A development environment for Windows devs can be set up using WSL2 connected with visual studio code and Docker Desktop

View and Edit components

Volto has the concept of View and Edit components.

View components show a block's content when the CMS is not in edit mode.
Edit components are shown when the block is being edited.

The View and Edit components for the chart builder add-on are located in volto-chart-builder/src/components

ChartbuilderView is the View component of the add-on and provides a preview of the plotly chart.
ChartBuilderEdit is the Edit component of the add-on and includes a preview of the plotly chart that shows in the block and a chart properties configurator that shows in the Volto SidebarPortal. This component also saves changes to the block state (e.g. configuration of the chart) to the backend.

Both of these components import and wrap components from the standalone chart-builder source code in volto/src/addons/chart-builder

Issues encountered and their solutions

1. Server Side Rendering of client libraries

Webpack threw the error ReferenceError: window is not defined when running Volto with our chart builder add-on. The error was caused by the SSR attempting render react-plotly.js in the node-express environment which does not have access to the window object.

The solution to the issue was to conditionally load the react-plotly dependency only when the code is run client side, using the following statement:

const Plot = typeof window !== "undefined" ? require("react-plotly.js").default : null;

2. Dependency management issues

We removed conflicting versions of dependencies that existed in both the Volto and chart-builder projects. We retained the dependencies in Volto and removed them from chart-builder. A longer term solution is to package chart builder up as a dependency of the volto-chart-builder add-on.

3. Block state update performance issues

Our initial implementation of block state updates to the plone back-end showed some lag when updating text fields. We resolved this issue by debouncing updates to the chart config state.

SPARQL data sources

Using https://beta.gss-data.org.uk/sparql as the endpoint for these queries

Example SPARQL queries

Carbon dioxide (CO2) emissions (territorial basis) per capita by local authority, UK, 2019, (tonnes of carbon dioxide)

PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX geo: <http://statistics.data.gov.uk/def/statistical-geography#>
PREFIX data: <http://gss-data.org.uk/data/gss_data/climate-change/beis-2005-to-2019-local-authority-carbon-dioxide-co2-emissions#>
PREFIX dim: <http://gss-data.org.uk/data/gss_data/climate-change/beis-2005-to-2019-local-authority-carbon-dioxide-co2-emissions#dimension/>
PREFIX meas: <http://gss-data.org.uk/def/climate-change/measure/>

SELECT ?geography_uri ?label  (ROUND((SUM(xsd:float(?val)) * 100) / 100) AS ?values)
WHERE {
  ?obs qb:dataSet data:dataset ;
       dim:year <http://reference.data.gov.uk/id/year/2019> ;
       dim:local-authority-code ?geography_uri ;
       meas:territorial-emissions ?val .

       FILTER(?geography_uri != <http://statistics.data.gov.uk/id/statistical-geography/N92000002>)

    OPTIONAL { ?geography_uri geo:officialname ?label }
}
GROUP BY ?geography_uri ?label

Woodland area by UK Local Authority, 2019

PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX geo: <http://statistics.data.gov.uk/def/statistical-geography#>
PREFIX data: <http://gss-data.org.uk/data/gss_data/climate-change/forestry-research-woodland-area-local-authority#>
PREFIX dim: <http://gss-data.org.uk/data/gss_data/climate-change/forestry-research-woodland-area-local-authority#dimension/>
PREFIX meas: <http://gss-data.org.uk/def/climate-change/measure/>

SELECT ?geography_uri ?label (SUM(xsd:float(?val)) as ?values)
WHERE {
  ?obs qb:dataSet data:dataset ;
       dim:year <http://reference.data.gov.uk/id/year/2019> ;
       dim:local-authority-area ?wrong_la_uri ;
       meas:woodland ?val ;
       <http://purl.org/linked-data/sdmx/2009/attribute#unitMeasure> <http://gss-data.org.uk/def/climate-change/concept/measurement-unit/percentage> ;
       .

   BIND (
     IRI(
       REPLACE(
         STR(?wrong_la_uri),
         "^.*/",
         "http://statistics.data.gov.uk/id/statistical-geography/"
       )
     ) AS ?geography_uri )
   FILTER(?geography_uri != <http://statistics.data.gov.uk/id/statistical-geography/N92000002>)

OPTIONAL { ?geography_uri geo:officialname ?label }

}
GROUP BY ?geography_uri ?label

Chart configuration constraints - some initial considerations

As a starting point we considered a list of rules and statements that might be relevant to the design of the chart configurator. The purpose of this list is to stimulate the design process by creating an initial set of constraints to be extended, questioned, changed or removed

A data source should always be selected before other chart options are presented to the user
X and Y data dimensions should be selected by the user before the chart type options are presented to the user
Chart 'main type' should be selected before chart sub-types are presented to the user
A limited set of predefined colour palettes to indicate data series will be available to the user
The chart background should always be a standard light blue-grey
Chart titles should always be at the top
Some charts don't need a title if their meaning is self explanatory or clearly indicated in surrounding context
Line chart series plot lines should always be 2px wide, irrespective of chart dimensions
Bar chart bar padding should always be 5% of bar width
Responsive charts should never be smaller than 400px / 25rem
A chart's dimensions should never exceed 1920px × 1080px
Axis labels for the start and end of the domain should always be shown
Gridlines should always have an opacity of less than 0.7
Gridlines should always be dashed
For continuous variables axis tick labels should never exceed a tick density of more than 1 tick per 40px
The user should be offered a set of predefined chart sizes such as big, medium, small as well as a responsive chart option
No text on the chart other than the title should be less than 11px or greater than 14px
There should never be more than 10 data series on a single line chart
There should never be more than 10 data series on a stacked bar chart
A chart with over 500 visual elements (e.g. svg elements) should trigger a warning before publication

ONS Chart guide (draft) late 2022

https://ons-design.notion.site/ONS-chart-style-guide-abc7605a97624dc2bc7f2a3e16379d82

Provide feedback

Saved searches

Use saved searches to filter your results more quickly