Snapshot updates #40

Merged
merged 3 commits into from Dec 12, 2012

2 participants

@jabley

Initial draft of archiving updates.

This is brute force stupid. It will download all items from a given date, not checking
the database or filesystem to see if it already has the file locally.

It should only need to be run a single time, to grab all the items from a particular date.
After that, we should just get the daily updates stored in the same location.

https://www.pivotaltracker.com/story/show/40611041

I've punted on tests - assuming that the existing tests cover most of the functionality
already, and this is intended to be a one-off task to slurp the files for now. We know
the download and store code works from having a working system in production.

jabley added some commits Dec 4, 2012
@jabley jabley Handle HMRC server being temperamental
Unable to download for some reason? Don't just loop until we hit VM
stack limits.
9b081d8
@jabley jabley Initial draft of archive functionality 1226605
@jabley jabley Add test for preventing infinite loop
Smug lisp weenies might wax about a decent condition system and
restarts...
a943147
@saulius

This looks good. Btw, there is no point in going beyond https://github.com/alphagov/trade-tariff-backend/blob/master/lib/tariff_synchronizer.rb#L59 these two days for Taric and CHIEF respectively, as we have the initial seeds from those dates.

Btw added https://github.com/alphagov/trade-tariff-backend/wiki/Importing-CHIEF-initial-seed-from-CSV just for reference in case it will be needed.

@saulius saulius merged commit e96c539 into master Dec 12, 2012
@jabley

Thanks for the documentation. How do we get the CSV files? Do HMRC provide an Oracle .dmp file and we have to then import that into an Oracle instance and generate the CSV ourselves?

@saulius

@jabley you generated those CSVs for us :-)

I still got Oracle binary files (not .dmp, no extension), I can send them to you, because I don't know how the magic happens.

@matthewford matthewford added a commit that referenced this pull request Jul 11, 2016
@theharq theharq Remove logstasher (#40)
We are going to use papertrail in the CF environment. this can be
configured to read the info from the log stream in cloud foundry, no
need to add code in the app.
f2714ea
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment