Skip to content

News NRM

Mikko Heikkinen edited this page Jan 2, 2019 · 9 revisions

DINA Web development news from Swedish Museum of Natural History, Stockholm.

2019-01-02

The last three months have been spent on improving the DINA Collections system and preparing it for production testing, by

  • Doing heavy improvements in user interface
  • Migrating all relevant data from the pilot system
  • Doing testing and data verification

The aim is now to publish a fully capable testing version of DINA Collections in February. This includes all features that are needed to manage the pilot collection (Mammals at NRM) with the same service level as using their current system. The difference is that DINA Collections is not just a system for one collection, but a generic database and platform to be which can flexibly be customized for other collections also.

The project is now in a transition phase and waiting a new project manager, new project plan and new decisions on the long-term future, so development news will probably be less frequent.

2018-09-21

Happened during previous month

The previous two sprints have both focused on same kind of issues:

  • Making the user interface better and easier to use
  • Working on infrastructure in preparation of putting the system into full testing and then into production

The record form has been divided into sequential sections, which are shown to the user one at a time. Sections can be navigated using left sidebar and previous/next buttons. This is to make the form less cluttered and easier to understand to the user. Feedback about this has been positive.

Date fields have been redesigned to allow handling of differenct kinds of dates (single dates, ranges and end dates).

Tree data navigation and entry/edit forms have also been improved.

Tree data editor currently

Search tool is now roughly ready for production use. Search results can be exported into a CSV file, with columns selected by the user. (Later we will work on the format of the CSV, e.g. how to handle fields which may have multiple values.)

Next we are going to

  • Continue polishing the user interface, to make it more intuitive and efficient to use.
  • Working on the infrastructure, preparing for putting the system in production.

The goal is to have the first collection (mammal collection from Stockholm) using the system by the end of the year. Before that we have to test the system with the end users, fix any bugs found, and set up and test e.g. deployment and backup processes.

2018-08-28

Happened during previous two weeks

We have mostly been working on the user interface, including

  • Finishing search tools
  • Improving specimen entry form
  • Improving tee data forms (for localities, taxa, agents and storage locations)

Next we are going to

  • Continue polishing the user interface, to make it more intuitive and efficient to use.
  • Adding few smaller features, e.g. for handling catalogue numbers and help texts and
  • Build a tool for exporting specimen data in spreadsheet format.

We are also increasingly working on the infrastructure, preparing for putting the system in production. The goal is to have the first collection (mammal collection from Stockholm) using the system by the end of the year. Before that we have to test the system with the end users, fix any bugs found, and set up e.g. deployment and backup processes.

2018-08-14

Happened during the summer

New search interface is finished. It allows

  • Searching specimens using c.10 different filters (more to be added soon)
  • Viewing a list of results, using infinite scroll (ie. more specimens are loaded as you scroll)
  • Customizing what fields are shown on the list of results
  • Switching between search results, viewing single specimen and adding new specimens

See videos of the UI:

We also designed how to improve and simplify the user interface, especially the entry form. This work will continue before launching the first production version.

Most of the data from the mammal collection has been migratyed into the system (though this will not be publicly shown in for the time being). Migration will happen in steps, so that potential difficulties with some of the data will not prevent migrating rest of the data.

Next we are going to

  • Improve the user interface:
    • Entry form
    • Tree data manager (e.g. taxonomy, localities)
  • Work on infrastructure, preparing for putting the system into production

2018-06-12

Happened during last sprint

Work has been focused on designing and developing search.

Next we are going to

Continue work with search. First version should be ready in June.

During the summer vacation period work will be done on

  • User interface improvements
  • Data migration
  • Infrastructure that is needed for putting the system into production (e.g. production server, backups and restores)

2018-05-28

Happened during last two sprints

New and refined features built:

  • Tools to add, edit and use agents (person & organization), and to connect them to users (persons who have username to the system).
  • Tool to add taxon names to storage locations. Eventually the system can suggest storage locations for new specimens, so that the user doesn’t need to remember wehre to put store particular kinds of specimens (e.g. skins of species X).

Design work done for:

  • Designed user & agent handling.
  • Designing new, more powerful search. The aim is to provide both easy search tool for commonly done searches, and flexible and powerful (but more difficult) tool that can be used to search almost anything.

Large part of the work was focused on infrastructure, i.e. things that make the system easier to extend and more reliable, but are not visible for the users directly. This included working on the database, help system,

Refining data import tools.

Next we are going to

Major focus is on designing first version of an advanced search tool, which should be powerful (allowing multitude of search use cases), easy enough to use, and feasible to build in one sprint.

2018-05-02

Happened during last sprint

Much of the sprint was spent on infrastructure work: making the system more robust and easy to expand further. Also support for editing taxonomy data was added, but this is not yet visible for users.

New fields were added to the registration form for:

  • Type specimen data
  • Events (death, observation and origin event data)
  • Catalogue card creation data
  • Acquisitions

With these in place, the registration form and underlying database have nearly all the fields needed for mammal catalogue card registration. A bit later these will be organized and improved to make registration process easier.

We have also been working on migrating mammal data to the system for testing purposes. Most of the data has been imported, partly in the correct fields and partly as read-only data (though not currently visible on the public testing version), which allows us to go deeper into connecting it to the DINA data model.

Next we are going to

Next sprint will not focus on developing new features for users, but on

  • Further improving infrastructure, data model and database
  • Designing how users and agents (people and organizations with names) are handled.

After this, we will develop user/agent handling, and start to design and develop more advanced search features.

2018-04-16

Happened during last sprint

New features added on the registration form:

  • Adding additional identifiers, e.g. from different data management systems.
  • Better storage location management, using locality tree management tool mentioned in previous newsletter

Search feature now uses Elasticsearch for searching and getting the data. This allows fast and powerful search features to be built. First version of an advanced search tool is being built.

Data model was being developed further and documented. Documentation is needed for understanding the data by both developers and users, and to work with the international DINA-partners.

Next we are going to

Develop new features according to the designs during the previous sprint, including

  • Type specimens
  • Specimen events, e.g. specimen origin and death
  • Record history events (e.g. when data was entered in previous systems or catalogue cards)
  • Basic tools for using taxonomy (not editing yet)
  • Acquisitions

With these in place, the registration form and underlying database will have nearly all the fields needed for mammal catalogue card registration. After this, we will focus on agent handling (users, collectors etc...) and advanced search.

We also plan to import real data into the system soon for testing purposes.

2018-04-03

Happened during last sprint

We we mostly adding new fields and features to the specimen registration form:

Ability to add several physical objects and curatorial assessments about them. A curatorial assessment means information about the object’s existence and condition at a given time.

Physical Object

This shows an example of a specimen consisting of a skeleton and a skin that was previously missing.

Field for the locally accepted taxon name used for a specimen. This means the name which is used e.g. for storage, without being a “real” determination.

Read-only data field. This is a temporary storage place for any data that has not yet been put into proper fields on DINA-Web, or is still waiting for data cleaning. This makes data migration faster, as a specimen can be migrated into the system even if contains some special fields.

There is also an early prototype of a curated locality management tool. (Accessible form the left sidebar.) This can be used to browser, add and edit hierarchical localities, e.g. continents, countries and provinces. It can also be customized to work with other hierarchical data, e.g. taxonomy and storage locations (e.g. rooms and cabinets). This is work-in-progress, and contains some known bugs.

Curated

The tool offers also a side-by-side view which allows editing locality data while seeing the related localities.

Next we are going to do

Develop new features according to the designs made, including

  • Adding additional identifiers, e.g. from different data management systems
  • “Recording events”: information on when and by who was the record created or edited in previous systems (sensu lato), e.g. old database or catalogue cards.
  • Better storage location management, using aforementioned locality management tool

Infrastructure work on database and user interface, and testing how the system can handle large amounts of data. Also a new system (Elasticsearch) for creating better search tools is being tested.

New features being planned for handling:

  • Type specimen
  • Specimen origin
  • Death circumstances

2018-03-26

Happened during last sprint

The entry form now includes refined feature observation handling. This allows recording multiple features (measurements, bone counts, sex and age), and additional information about these features and how they were recorded (method/unit, date, person and remarks). This can be be later customized to allow more efficient handling of the most important features in a given collection.

FeatureObservations v 0.2

Infrastructure work in the background continues (data model, database and testing), as well as preparing for data migration from current system to DINA by documenting the data.

We have new designs ready for:

  • Physical objects
  • Handling of different types of dates
  • Record events, i.e. information about when the data was previously saved and where
  • Curatorial assessments, e.g. status and condition of specimens

Latest features can be viewed at https://alpha-cm.dina-web.net (username "john doe", ask password from DINA team at NRM)

The mock data version is not actively updated anymore, since the new Javascript-based stack allows us develop both back- and front-end synchronously within the main version.

Next we are going to do

  • Develop new features according to the designs made, including
    • Curated locality handling: adding and editing curated localities and connecting them to specimens.
    • Several physical objects
  • Design new features
    • Handling storage locations
    • Adding additional identifiers, e.g. from different data management systems

2018-03-13

Happened during last month

During the last month most of the work has been done on the database and API, instead of creating new features visible for users.

Data model has been restructured in significant ways. The aim here is to provide more flexible basis for future development, to accommodate new collections and allow fast development of new features. Data model and the API built on it has been split into several logical parts, e.g. separating storage handling from actual specimen handling.

API is now built using Javascript and Node. This allows using same technology and code on both frontend (user interface) and backend (API and database), and using a "single source of truth" to specify the data model and data validation rules. This will reduce complexity and room for errors.

Database was switched to PostgreSQL, which allows more powerful and flexible document-oriented functionality than previously used MySQL. The user interface has also been adjusted to use the new database and API.

DINA International Technical Committee held a meeting in Berlin in late February, sharing progress and discussing how to proceed. NRM will continue with building first version of the collection management system for NRM's Mammal collection, AAFC will focus on SeqDB and a taxonomy service, and MfN will work on label and data migration tools. The partners will together continue on high-level long-term planning for generic cross-collection system.

User interface has one new feature: better catalogue number handling. The system can create a catalogue number automatically, or allow user to set it manually.

Next we are going to do

  • Finish setting up new data model with the database and API
  • Better feature observation handling on the specimen entry form (age, sex, measurements, bone counts etc.)
  • Design
    • More advanced curated locality management
    • Storage locations and determinations
    • Handling several physical objects (e.g. skin and bones from same individual)
    • Inventory status (is the specimen found in collection)

2018-01-30

Happened during last two weeks

We mostly worked on the mammal record form, adding fields and improving functionality.

Now there is possibility to have multiple determinations in a single specimen. These are displayed as an “accordion”, meaning that each determination can be "expanded" to reveal its content, while others are “collapsed” and display only a summary of the determination.

Determination element also contains few new fields and a button to copy the taxon name to verbatim taxon name, which can be used to reduce typing in the case these are similar. We already have user feedback on these, and the determination element will be improved in upcoming sprints.

There are many new locality fields. Some of these (continent & country) use test data from the current mammal collection system. More restructuring of these fields will follow, to create tools for adding, editing and using “curated localities” - a database of shared and standardized localities with coordinates.

We created a platform for contextual help messages: by clicking on the ? icon shown next to (most) fields, a help sidebar is opened.

In addition we worked on better login/logout system, improved code testing.

Try these out in the Alpha mock version at: https://alpha-cm-mock.dina-web.net/ (use random username and password)

There are also new and improved tools for supporting the development of the system:

Next we are going to do

  • Develop better catalogue number handling: the ability to choose between automatically assigned (preferred) or user-defined catalogue numbers.
  • Develop curated locality management (see above)
  • Design data model and user interfaces for new features, adding fields on the form and then improving design and user interface
    • Feature observations (e.g. age, sex, measurements)
    • Storage locations and determinations
    • Handling several physical objects (e.g. skin and bones from same individual)
    • Inventory status (is the specimen found in collection)
    • Additional identifiers (e.g. “old skin numbers”, Rovbase numbers, old loan numbers)
  • Plan mammal data migration
  • Focus more on medium-term (few months … end of year) planning
  • Prepare for annual DINA International meeting in Berlin, at the end of February

2018-01-16

Happened during last month

We had a five-week sprint due to Christmas vacations. Things done during the sprint:

Landing page: The front page https://alpha-cm.dina-web.net/ now shows some information about the DINA-Web.

Simple search form can be used to find specimens by a catalogue number (try 12345) and a taxon name (try "Mus musculus"). The result list links to the edit form, which can be then used to edit the specimens. The form has only few new fields compared to last sprint, but can be used for testing with the fields it has. New fields are added in the next sprint.

Extending the database (and the API used to access it) to handle mammal specimen data. This is something that is continuosly worked on, whenever new fields are added to specimen record forms.

We also created a basic platform for developing and deploying the user interface. Technical list of new features can be seen at Github, and the lastest UI is available at

So-called notification platform: A set of tools to show users error messages, help texts and other kind of notifications, which help using the system. The goal is to have help texts and other information presented to the users in understandable format and at the point where they are needed, instead of having completely separate manual somewhere else.

Planning data migration. New tools for transforming, validating and migrating the mammal collection data to DINA were planned and partly done.

Next we are going to do

During the next sprint (2018-01-16/23) we will design and add more features with the aim of strarting testing the system with real specimen data soon.

  • Better locality management (partly done already)
  • Better catalogue number handling
  • More fields for determination, and allowing multiple determinations in one specimen
  • Planning for
    • How to handle curated localities and connect specimens to them. A curated locality is a pre-defined geographical area, for example a country or a trap site.
    • Feature observations: how to handle e.g. different kinds of measurements
    • How dates are handled

2017-12-07

Happened during last month

New front-end developers Anton and Fredrik started working at NRM in early November. We started building Collections manager module, which will become the tool for museum employees to do core collection management and curatorial work. The biggest task has been setting up a new system for developing the module. In technical terms, we are using React and Redux on the frontend and using Java-based API and MySQL database on the backend. There's also an automated process for testing the system (to make sure there are no serious programming errors) and for deploying the new version to NRM's servers.

We have set up an "Alpha version" of the collection manager - this is meant for testing the system by anyone. New features and other changes we are making are published on the alpha version almost every day. Alpha is accessible at two addresses:

  1. Mock version: https://alpha-cm-mock.dina-web.net/ - This uses mockup data and doesn't need username for logging in
  2. Data version: https://alpha-cm.dina-web.net/ - This will allow saving test data to the database. Use username NNN and password YYY to log in.

NRM's mammal collection was selected as the pilot collection, meaning we will concentrate first on building tools for them in particular, then expanding to other collections later.

The Alpha version contains a basic form for recording mammal specimen data, specifically data from an old card catalogue. It includes basic fields and some validation, and connects to the database. The design of the form is still preliminary, and will change considerably in the near future.

There's also a very basic first version of a search form. Users are also able to change the language of the user interface between English and Swedish, though many Swedish translations are still missing.

Next we are going to do

The goal of the next sprint is to build the mammal specimen record form to such a stage that meaningful testing by end users from the mammal collection is possible. This includes working on:

  • Finishing basic platform for developing and deploying.
  • Creating a simple form that can be tested by mammal collection users. Save specimen data to the database, using the collections API. Add necessary fields. Design the form for better usability.
  • Adding ability to record basic taxon information and to show storage location.
  • Enabling search by a few key fields (catalogue number and taxon name) from data from database, provide results as a list, and link to the entry form for editing.
  • Adding a start page for non-logged in users, describing what the system is and what has been changed

The next sprint will continue until 2018-01-09 due to Christmas and New Year. After that we will get back to two-week sprints.

You can’t perform that action at this time.