kromcuich edited this page May 15, 2014 · 1 revision

System Overview

1. Overview

There are 4 Applications

1.1 Main / SearchApp:

This is the main app where entities are listed and edited.

entry point:main.js

modules: CRUD, Data, elements, Navigation and SolrSearch

**dependency declaration: **loader.js

1.2 data-entry:

This is an input form for users in the data entry group. Currently it just allows it’s users to create entities.

**entry-point: **lib/data-entry/main.js

dependency declaration: data-entry-loader.js

**modules: **data-entry, elements, Data

1.3 Reporting

This creates graphs using d3/nvd3. The three entities pull their data from solr, and user data comes from the django backend.

**entry-point: **lib/reporting/main.js

dependency declaration: reporting-loader.js

**modules: **reporting, elements, SolrSearch

1.4 Monitor

The monitor app is used to configure and monitor data imports

**entry-point: **lib/monitor/main.js

dependency declaration: monitoring-loader.js

**modules: **monitor, elements

2. Design decisions

The system uses an event driven architecture, this was chosen to promote decoupling of modules and to work with the asynchronous nature of javascript applications and ajax requests.

To aid with this the Bacon library was used to provide a more fully featured event system. It provides Functional programming tools for events.

RequireJS is used to provide a module system.

2.1 Conventions:

Models and Collections are found in folders called data

templates, surprisingly enough are found in templates!

views in views

tests are in tests on a per module basis

  • This is with a view that modules could be reused across projects and therefore have their test folders brought with them.

Container views store their subviews in a variable called childViews, destroy is invoked on each of these subviews to clean up after they are removed from the display

3. Dev Environment

This outlines the requirements for getting a dev environment set up for the front end. There are two main build tools used, guard from ruby for compiling sass/compass and grunt for managing the js build requirements.

It would probably be nice to move the sass/compass stuff to grunt too.

3.1 Sass/Compass

You will need to have ruby installed on your machine, with the bundle gem. Try using rvm https://rvm.io/ to manage this.

bundle (0.0.1)

bundler (1.3.5, 1.0.21)

then from /static run

bundle install

to install the sass/compass/guard dependencies

and once installed

bundle exec guard

to watch for changes to the sass files, it builds them with sourcemaps which is good for dev on chrome

Details here on how to use sourcemaps in chrome, https://developers.google.com/chrome-developer-tools/docs/css-preprocessors

Combined with livereload it’s great, but you’ll have to set that up yourself.

Output should be optimized for production.

This can be done in config.rb, instructions are contained within that file.

The stylesheets are broken up into smallish components.

The app also uses an icon font generated on http://icomoon.io/app/#/select

You can add new ones by uploading the corroborator.dev.svg file from /static/fonts/corroborator

3.2 Js Dependencies

Node dependencies

Node deps are installed using npm install. you must also have the grunt cli installed globally and karma, so

npm install -g grunt-cli karma

Frontend dependencies are managed using bower.

jquery.ui and leaflet don’t come with builds.

If you find yourself in a situation where you need to re-install them, go to the root folder of the project, run npm install and then grunt build, these projects usually have the dist folder in the .gitignore

There are a few libraries in there that shouldn’t be, but looking at the loader files will tell you what the real deps are.

Not used:

backbone.paginator, q, socket.io, spinjs

3.3 Templating

Handlebars is used to generate the main html templates. These are all precompiled. To compile the templates run grunt handlebars. Template file declarations have been moved into the grunt folder, and are referenced at the top of Gruntfile.js.

3.4 Testing

karma is the test runner and jasmine is the assertion library, run karma start from static/js to have a watcher watch your folders for changes to files.

test files should be in the test folder of modules and end in Spec.js

html fixtures are created via the jasmine jquery library which also provides jquery like assertions

In the grunt file there is also a karma:ci task that can be run to have the tests run once

Karma config is in karma.conf.js and the test entry point is in test/test-main.js. This should pretty much mirror your loader.js file with regard to dependencies.

3.5 Documentation

Generate the documentation/annotated source by running grunt docco. The docs will appear in docs/annotated-source

4. Backbone Applications

4.1 Search/Main Application

The app is divided into 5 modules Navigation, SolrSearch, CRUD, Data and elements.

4.1.1 Data

Data provides the models and collections that are shared with the other modules. It also prepares the data before sending it to the django backend.

There’s nothing too special here, the main entity collections listen for new content on the searchBus defined in lib/streams.js

There are two models for each entity, one for normal save and display and one for persisting via the multisave

collections.js provide the collection instances

actor.js, bulletin.js and incident.js provide the model and collection definitions.

collection-mixins provides code that is shared between the collections using the underscore extend function.

These are populated by solr.

The initial intention had been to use the data from solr for the main display of entities, but this proved impractical, because the data from solr could easily be stale, even if it were not when downloaded initially, it could become so if another user edited the entity.

Saved searches are stored in saved-search-collection.js

Locations in LocationCollection

these are populated via the Bootstrap variable

4.1.2 SolrSearch

SolrSearch is responsible for triggering solr searches and displaying the results.

Init for the application occurs in lib/SolrSearch/main.js

4.1.2.1 Result Display:

Results.init starts a listener in results.js that watches for navigation between tabs and updates the list on show based on the entity selected.

The result views themselves are fairly normal, they only real special thing they have is a kind of infinite scroll functionality, whereby results are rendered in chunks rather than 500 at a time.

4.1.2.2 Filters(Facets):

This provides an interface for the filtering of search results.

Searches return facets(filters) the user may select a facet and further refine the search. All facets returned by the search will still be displayed to allow for ORing of facets within the same group

If a user has selected a facet(s) and then instigates a search, the system then checks the selected facet(s) and reapplies them if they are available in the search result.

The data processing occurs in lib/SolrSearch/data

Each entity has a filter collection which in turn has FilterGroupCollections for each facet group

These hold the available filters

Each entity also has a SelectedFilterCollection which stores the selected filters for each collection.

4.1.2.3 Search Building

The search is assembled in QueryBuilder...needs documenting

4.1.2.3 Saved search

Searches are saved as json objects in the database, in the PredefinedSearch model. This is not really cool.

the logic is in lib/SolrSearch/persist

search-saver sends an event requesting the current filters, which is picked up by the SelectedFilterCollections of the 3 entities, who send them back. these are then persisted to the database as json

apply-saved-search.js is used to send the filters out to the filter collections, this happens again via the event stream.

4.1.2.4 Improvement

This would probably be better and more simply done by allowing direct access to the SelectedFilterCollections. A schema should be designed to save filters and search terms to the database.

4.1.2.5 Display Logic

this is all in lib/SolrSearch/views/filters/

filter manager decides which set of filters should be displayed.

actor, bulletin, incident -filters.js show the filters for each entity.

filter-mixins provides shared functionality

filter-elements provide the label and autocomplete widgets

date-range.js - the date range fitler

map filter - display the map with a list of locations

4.1.3 CRUD

So this kind of does what it says, it allows users to create read update and delete stuff. It is responsible for the display of all our entities, and their edit forms.

It’s mainly views, divided into display-views(well named) and search-views. Search views were intended to be only the results of embedded search, which for the main part they are, but there are a few extras in there, like the media form and revision.

4.1.3.1 Forms:

Like most sections there is a manager class that is responsible for opening and closing views based on the route. It is also responsible for managing whether the entity is displayed full screen or as a section on the right.

Conventions

  • All form inputs have their name attribute set to match the backend model. This allows us not to worry about mapping between form html names and backend names.

  • All form inputs have -field as a class name. This is used to label form fields as belonging to a certain entity and facilitates nested forms events for example

Widgets

There are a number of configurable widgets for forms, the fields are defined in the -form.js files, and implementations are in form-mixins.js

labelFields:

example: Sources - shows a drop down auto complete list and displays selected items below

params:

containerid: dom id of the element

collection: collection instance containing the values for the auto complete

multiple: can one or more be selected

field_name: name of the model field on the backend

field_label: label to be shown

eventIdentifier: label selection triggers an event which has what is here + ‘_label’

bus: the bus to send the events on

mapFields:

Show a map

locationSource: event identifier for where the locations that will be displayed on the map are coming from

bus: event bus to listen on

sliderFields:

show a jquery ui slider

sliderDiv: jquery selector

display: where the score value is displayed

formfield: model field name

dateFields:

these are jquery ui date picker fields

dateTimeFields:

use the date time picker from jquery_time plugin

comboIds

id’s for fields that will use the drop down boxes, values are currently hard coded on forms

4.1.3.2 Nested Forms:

Nested forms like events comments, have a hidden multi select object that stores the resource uris for child elements.

4.1.3.3 Templates:

display templates contains all the templates for viewing the elements, search views have all the data entry and form templates.

4.1.3.4 Embedded search

Each entitiy under search-views has a an -search-field.js this is used to show the embedded search field and it’s results

4.1.4 Navigation

Navigation contains the Router, the search box and the home button. Again it’s initialisation occurs in the main.js file

4.1.4.1 Router

This is the main router for the application. it dispatches events based on uri’s clicked. There is a view in Tabrouter.js also that handles the class toggling for the tabs.

4.1.4.2 activity

Listens for events that indicate the search has started and finished. Displays a spinner

4.1.4.3 NavCombo

Show the search drop down that has the search button, predefined searches and the save search option. Extends the Combo View from elements

It’s pretty dumb and delegates responsibility for saving and running searches to SolrSearch modules

4.1.5 elements

Elements is a module used to store re-usable elements.

helpers:

cookie.js

get the csrf cookie for forms

date-helper.js - poorly named

these are all the handlebars helpers for our templates

form-validate.js

helpers for forms, validation for fields and tranformation from jquery serializeArray format to one more digestible by backbone models. Set’s error fields on forms also.

Improvement

It would be nice to abstract form validation and widgets into a library that could be used for creating rich backbone forms

ie-fixes

polyfills

view-close.js - again poorly named, this adds a destroy method to all views, to help prevent zombie views and a build up of detached DOM elements.

views

CollectionViews are used to display lists of things, those defined by foreign key fields on models

**ScrollViewMixin **

Model render paging for list views( in SolrSearch ) and CRUD (embedded search)

combo.js

Combo box view

date-time-range.js

used on forms for showing a date time range

dialog.js

used for saving searches

input.js

View for the search text field

This could probably be simplified,

label-widget.js

This is used generally with a combo box in form views. It displays an autocomplete text field, the selected fields, and manages a hidden multiple select form field to store the selections to be saved.

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.