Enabling the population and distribution via API of UK Customs tariffs and duties
Ruby Other
Permalink
Failed to load latest commit information.
app using actual scope for footnotes api Mar 16, 2017
bin
config
data Restore missing chief records (#45) Jul 1, 2016
db fix data migration Mar 16, 2017
lib
log Need to have log directory present, as libraries expect it. Aug 15, 2012
public no need for index.html May 21, 2012
spec Add spec for the auditable plugin Jan 18, 2017
vendor/plugins Remove assets dependency (#70) Jul 21, 2016
.cfignore Auto deployment from circle ci, using deployment branches Aug 9, 2016
.codeclimate.yml
.env.test
.foreman Update Readme, using forman locally Apr 27, 2016
.gitignore
.rspec Add .rspec global config to git Mar 31, 2016
.rubocop.yml remove outdated cop Jul 19, 2016
.ruby-version Upgrade ruby version to 2.3.3 Jan 31, 2017
.sample.env
CHANGELOG.md Update CHANGELOG (#84) Aug 3, 2016
CONTRIBUTING.md get rid of travis Mar 3, 2016
Gemfile Upgrade ruby version to 2.3.3 Jan 31, 2017
Gemfile.lock Upgrade ruby version to 2.3.3 Jan 31, 2017
LICENCE.txt
Procfile
README.md Update README.md Aug 10, 2016
Rakefile Disabling external requests Mar 16, 2016
circle.yml Upgrade ruby version to 2.3.3 Jan 31, 2017
config.ru
prod2dev.sed Add sed commands for database import Apr 11, 2014

README.md

CircleCI Code Climate

Trade Tariff Backend

The API back-end for:

Other related projects:

Development

Dependencies

  • Ruby
  • Postgresql
  • ElasticSearch
  • Redis
  • Memcached (production)

Setup

  1. Setup your environment.

    bin/setup

  2. Update .env file with valid data.

  3. Start Foreman.

    foreman start

  4. Verify that the app is up and running.

    open http://localhost:3018/healthcheck

Load database

Check out wiki article on the subject, or get a recent database snapshot.

Performing daily updates

These are run hourly by a background worker UpdatesSynchronizerWorker.

Sync process

  • checking failures (check tariff_synchronizer.rb) - if any of updates failed in the past, sync process will not proceed
  • downloading missing files up to Date.today (check base_update.rb and download methods in taric_update.rb and chief_update.rb)
  • applying downloaded files (applying measures, etc. TARIC first, then CHIEF)

Updates are performed in portions and protected by redis lock (see TariffSynchronizer#apply).

BaseUpdate#apply is responsible for most of the logging/checking job and running import! methods located in Taric/ChiefUpdate classes. Then it runs TaricImporter and ChiefImporter to parse and store xml/csv files.

Whole process is quite similar for both TARIC and CHIEF, but CHIEF updates undergo a tranformation transformation process to convert them into a TARIC format. Check ChiefTransformer class for more info (and ChiefUpdate#import!).

In case of any errors, changes (per single update) are roll-backed and record itself is marked as failed. The sync would need to be rerun after a rollback.

Deployment

We deploy to cloud foundry, so you need to have the CLI installed, and the following cf plugin installed:

Download the plugin for your os: https://github.com/contraband/autopilot/releases

chmod +x autopilot-(YOUR_OS)
cf install-plugin autopilot-(YOUR_OS)

Set the following ENV variables:

  • CF_USER
  • CF_PASSWORD
  • CF_ORG
  • CF_SPACE
  • CF_APP
  • CF_APP_WORKER
  • HEALTHCHECK_URL
  • SLACK_CHANNEL
  • SLACK_WEBHOOK

Then run

./bin/deploy

NB: In the newer Diego architecture from CloudFoundry, no-route skips creating and binding a route for the app, but does not specify which type of health check to perform. If your app does not listen on a port, for example the sidekiq worker, then it does not satisfy the port-based health check and Cloud Foundry marks it as crashed. To prevent this, disable the port-based health check with cf set-health-check APP_NAME none.

Notes

  • When writing validators in app/validators please run the rake task audit:verify which runs the validator against existing data.

Contributing

Please check out the Contributing guide

Licence

Trade Tariff is licenced under the MIT licence