Trade Tariff Backend
The API back-end for:
If using the GOV.UK development Vagrant VM
Ensure that you have pulled the latest version of the puppet repo.
Run the bootstrap command
Dependencies (OS X using Homebrew)
ElasticSearch & MySQL & Redis
brew install elasticsearch brew install redis brew install mysql
brew install chruby brew install ruby-install
Check out wiki article on the subject, or get a recent database snapshot.
Load database seeds for development API user
bundle exec rake db:seed
bundle exec ./startup.sh
Performing daily updates
Create config/trade_tariff_backend_secrets.yml file with correct values.
sync_username: sync_password: sync_host: sync_email:
Run the sync rake task
bundle exec rake tariff:sync:apply
- checking failures (check tariff_synchronizer.rb) - if any of updates failed in the past, sync process will not proceed
- downloading missing files up to Date.today (check base_update.rb and download methods in taric_update.rb and chief_update.rb)
- applying downloaded files (applying measures, etc. TARIC first, then CHIEF)
Applying downloaded CSV files is the most confusing and buggy part. Updates are performed in portions and protected by redis lock (see TariffSynchronizer#apply).
BaseUpdate#apply is responsible for most of the logging/checking job and running
import! methods located in Taric/ChiefUpdate classes. Then it runs TaricImporter
and ChiefImporter to parse and store xml/csv files.
Whole process is quite similar for both TARIC and CHIEF, but CHIEF updates also does transformation process at the end. Check ChiefTransformer class for more info (and ChiefUpdate#import!).
In case of any errors, changes (per single update) are roll-backed and record itself is marked as failed.
Keep in mind that there are two ways of rolling-back, one with keeping the intermediary updates stored in db, and another one without. The default option is to remove TARIC/CHIEF updates and data transformations.
DATE='2014-01-30' bundle exec rake tariff:sync:rollback
Project does not contain schema.rb, do not use rake db:schema:load. Sequel does not yet support view creation via schema file.
When writing validators in
app/validatorsplease run the rake task
audit:verifywhich runs the validator against existing data.
Please check out the Contributing guide
Trade Tariff is licenced under the MIT licence