Initial draft of archiving updates.
This is brute force stupid. It will download all items from a given date, not checking
the database or filesystem to see if it already has the file locally.
It should only need to be run a single time, to grab all the items from a particular date.
After that, we should just get the daily updates stored in the same location.
I've punted on tests - assuming that the existing tests cover most of the functionality
already, and this is intended to be a one-off task to slurp the files for now. We know
the download and store code works from having a working system in production.
Handle HMRC server being temperamental
Unable to download for some reason? Don't just loop until we hit VM
Initial draft of archive functionality
Add test for preventing infinite loop
Smug lisp weenies might wax about a decent condition system and
This looks good. Btw, there is no point in going beyond https://github.com/alphagov/trade-tariff-backend/blob/master/lib/tariff_synchronizer.rb#L59 these two days for Taric and CHIEF respectively, as we have the initial seeds from those dates.
Btw added https://github.com/alphagov/trade-tariff-backend/wiki/Importing-CHIEF-initial-seed-from-CSV just for reference in case it will be needed.
Thanks for the documentation. How do we get the CSV files? Do HMRC provide an Oracle .dmp file and we have to then import that into an Oracle instance and generate the CSV ourselves?
@jabley you generated those CSVs for us :-)
I still got Oracle binary files (not .dmp, no extension), I can send them to you, because I don't know how the magic happens.
Remove logstasher (#40)
We are going to use papertrail in the CF environment. this can be
configured to read the info from the log stream in cloud foundry, no
need to add code in the app.