Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Upgrading from v1.x to 2.0.0
Upgrading from v1.x to 2.0.0
The 2.0.0 release represents a major upgrade to the Roadmap codebase. One important aspect of these changes involves an emphasis on data validation. Prior iterations of the Roadmap system did not have robust data validation and it is likely that data imported from DMPonline_v3 or other systems introduced invalid records.
These new validations will cause invalid records to fail when accessed or updated by your users so it is imperative that you follow the instructions below to ensure that your site functions properly after the upgrade.
We highly recommend that you backup your existing database before running through these steps to prepare your system for Roadmap 2.0.0!
Overview of Database changes
- Added new table
statsto facilitate the Usage Statistics page available to Admins
- Added new
versionable_idfield to the following tables to help ensure accurate transfer of template customizations when a funder template changes:
default: falseto boolean fields:
- Dropped unused tables:
- Removed unused fields:
- Various index changes in support of the above changes
Overview of model validations
Many new validations were introduced to enforce referential integrity, data types, etc. The following list contains highlights of some of these validations. Please review the validation section of each of the models for a complete list of changes.
- Annotation type must be unique per question (currently only 2 options: example answer or guidance)
- Guidance records must now have at least one associated theme if they are published.
- Language abbreviations must follow the ISO standard using dashes instead of underscores (e.g. pt-BR not pt_BR)
- Org now requires a contact_name, contact_email, feedback_msg and feedback_subject when feedback_enabled is true
- Guidance and Templates can no longer be published if they belong to the 'Other' Org,
orgs.is_other = true
- User must be associated with an Org
- Templates must now have a family_id
- User must have a firstname and surname
- Question, Annotation and Note text cannot be blank
- Org names must now be unique and they must have an abbreviation
- Template, Phase, Section, Plan and Theme title cannot be blank
- Phase, Section and Question numbers must be unique for their given context (e.g. Section #23 cannot have 2 question #1s)
You will need to run through the following steps to upgrade your system to v2.0.0 and ensure that your existing database records are valid:
The Part one script runs 3 separate steps to prepare your system for Roadmap v2.0.0:
- Updates existing records in your database to prepare them for some new database validations that prevent NULL values in boolean fields:
guidance_groups.optional_subset- any NULL values will be set to 'false'
guidance_groups.published- any NULL values will be set to 'false'
notes.archived- any NULL values will be set to 'false'
orgs.is_other- any NULL values will be set to 'false'
- The standard Rails database migrations will then be run
- A script that checks your existing data to determine if any records would become invalid due to the new validations added to each of the models. This script runs SQL queries to do a quick scan of your data. It is intended to give you an idea of the extent of any issues you may have before running the more in depth scripts in part two.
The final script will output a report to your console. That looks something like this:
In the example below, we can see that there are 36 orphaned records in my answers table and 3 user records that do not have a unique email address:
Checking Annotation records 0 records with a empty text field 0 orphaned records due to nil or missing org 0 orphaned records due to nil or missing question 0 records with a empty type field 0 records that are not unique per (type, question_id, org_id) Checking Answer records 0 orphaned records due to nil or missing plan 0 orphaned records due to nil or missing user 36 orphaned records due to nil or missing question 0 records that are not unique per (question_id, plan_id) ... Checking User records 0 records with a empty email field 3 records that are not unique per (email)
If you were fortunate enough to only find a few invalid records in the report from part one, you can go ahead and clean them up manually. We advise documenting any queries you run though in the event that you need to run them again on another instance of your database.
Finding invalid records
Please run the following script to run a thorough investigation of every record in your database. (Note that this can take in excess of one hour depending on the size of your database):
This script will output another report detailing any additional records that are still invalid
Correct any data issues
Review the report from the prior rake task and then either correct the issues manually or create your own data cleanup rules that can then be run through the
rake data_cleanup:clean_invalid_records task. This task will pick up any rules you have added to the
Please watch the following video for a detailed overview of this process: DMPRoadmap v2.0.0 DataCleanup
Iterate through the find/clean steps until you can run
rake data_cleanup:find_invalid_records and verify that all of your existing data is valid.
The final script will perform a few other tasks to initialize some of the new tables and fields.
- Runs a script to initialize all of the new versionable_id fields that were added to the
- Normalizes your
- Populates the new usage statistics tables
Please note that the task that updates the
versionable_id fields can take in excess of 15 minutes depending on the size of your database.
Updating your deployment scripts
This release also involved some changes to the basic infrastructure of the system. Please check and update your deployment scripts as needed for your individual installations.
- Tests are now run via RSpec instead of MiniTest. Use
rpec spec/to run the tests
- Modifications were made to the Webpack configuration. To compile you assets for production (or any situation where you do not want Webpack to watch your assets for changes) you should now use
npm run bundle -- -p --no-watch
- We have updated the Gemfile to segregate deployment specific gems so that you can exclude them as needed when running
bundle install --without [groupA] [groupB]. For example:
bundle install --without mysql puma thin:
- You will need to setup a job (e.g. via CRON) that runs the
rake stat:build_monthlytask at the beginning of each month