Home

Data Hub is an open source CRM platform developed at the Department for International Trade to track activities related to assisting UK Companies to export their goods & services and track and record activities related to projects investing in the UK.

The platform is split into 2 major components: the API back-end and the web front end. The back-end for Data Hub provides an HTTP Restful API, which provides calls for searching, filtering and maintaining data.

This Project provides the web front end which 'drives' the API to provide a web-based interface that can be used by both desktop and mobile web browsers. This document and related WIKI articles focus on the web frontend service.

The frontend is written as a server that communicates with the API and renders HTML pages to send to the client. The browser, in turn, renders the pages and is responsible for sending data back to the server in the form of regular HTTP GET requests for pages or HTML POST requests to upload data.

Service Elements

The frontend is a Node/Express/Nunjucks server application. Requests to the server are routed through one or more middleware components, which may communicate with the back-end API, and eventually transform data into structures which are passed onto the Nunjucks layer to render to HTML before being sent back to the client.

A request is made to the server which follows a number of predefined steps:

• Routing: The Express app has a large amount of routing configured to define the path a request will take through the application, including which middlewares to call, which parameter handlers to use and which controllers.
• Middlewares: The service is made up of a number of middlewares, some common to all requests, such as authentication, session retrieval, compression, and some unique to the request, such as retrieval of a company record or attempting to save a form posted to the server. The majority of the application is implemented in this middleware layer. If there is a job to be done then having a dedicated middleware for that makes it testable and reusable.
• Controllers: Controllers are kept lightweight. In some cases, a request is relatively simple and the controller can fetch the data and call the template. Anything more complex is generally split out. The primary job of a controller, call the page render function.
• Nunjucks: Nunjucks accepts the data and renders the HTML. Nunjucks consists of pages, layouts and macros (components). The Nunjucks element of the application is relatively large but contains a large number of generalised macros that will turn data in a predefined content into a common chunk of HTML on screen. Nujucks macros form a large part of how forms and collections work.
• Back-end(Leeloo): The back-end API that takes care of search, database storage, filtering. The frontend talks to the back-end via a documented API.
• Redis. The frontend keeps a small amount of session data (the user OAuth token) within Redis currently.

Things the frontend does do

When creating the initial architecture the decision was made to split into frontend and back-end. Within these two domains, there is a very clear definition of which part of the architecture does what. The ultimate ideology is that all code or tasks related to storing information or implementing business logic or rules should be put into the back-end and it should be possible to create another frontend application, such as a mobile phone app, whilst keeping the duplication of code to a minimum. So, with this in mind, what does the frontend do?

• Call the API to get data
• Transform data into web pages
• Provide navigation
• Present forms for editing data
• Ask the API to save data
• Present errors back to the user for things such as validation

Things the frontend doesn't do

So if that is what the frontend does do, what things does it not do? Anything that involves making a decision or enforcing logic is what it doesn't do, this includes:

• Data validation
• Marking elements of a work-flow complete or incomplete
• Deciding what can and cannot be shown according to the role

There is still some flow and process logic that is kept in the frontend, such as the flow to add a company needing to know what sort of company is being added. Pragmatism always wins.

Non-Spa

And this is not a Single Page Application (SPA), the application in its current form can function 100% without client-side javascript enabled.

The rationale behind the current architecture, primarily doing all the work on the server and working without Javascript, is due to a few things:

• Wide range of hardware/software capabilities, some unknown
• Varying network capabilities, in some cases very limited.
• Robustness
• Using the right tool for the job

The service is used by a wide range of users, both government and 3rd party agencies and is used in places with varying levels of hardware/software and in some cases very limited network bandwidth. We now have a better idea of the client software users have thanks to analytics's but the service must cater for all users and consider their access. Moving a lot of code or using large frameworks on the client can put too much stress on the network and limited processing power available.

Also, relying on SPA frameworks can mean the application can become unusable if there is a single fault in the source code or if the browsers are unable to load javascript due to network or security considerations.

Finally, in the majority of the application, the SPA approach simply adds no real benefit to the user, though there are some areas that client-side JS can definitely make a massive difference.

Overall the approach has to be 'progressive enhancement' but still be able to do the job without client-side JS.