Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Snapshot updates #40

Merged
merged 3 commits into from Dec 12, 2012

Conversation

Projects
None yet
2 participants
Contributor

jabley commented Dec 11, 2012

Initial draft of archiving updates.

This is brute force stupid. It will download all items from a given date, not checking
the database or filesystem to see if it already has the file locally.

It should only need to be run a single time, to grab all the items from a particular date.
After that, we should just get the daily updates stored in the same location.

https://www.pivotaltracker.com/story/show/40611041

I've punted on tests - assuming that the existing tests cover most of the functionality
already, and this is intended to be a one-off task to slurp the files for now. We know
the download and store code works from having a working system in production.

jabley added some commits Dec 4, 2012

Handle HMRC server being temperamental
Unable to download for some reason? Don't just loop until we hit VM
stack limits.
Add test for preventing infinite loop
Smug lisp weenies might wax about a decent condition system and
restarts...
Contributor

saulius commented Dec 12, 2012

This looks good. Btw, there is no point in going beyond https://github.com/alphagov/trade-tariff-backend/blob/master/lib/tariff_synchronizer.rb#L59 these two days for Taric and CHIEF respectively, as we have the initial seeds from those dates.

Btw added https://github.com/alphagov/trade-tariff-backend/wiki/Importing-CHIEF-initial-seed-from-CSV just for reference in case it will be needed.

saulius added a commit that referenced this pull request Dec 12, 2012

@saulius saulius merged commit e96c539 into master Dec 12, 2012

Contributor

jabley commented Dec 12, 2012

Thanks for the documentation. How do we get the CSV files? Do HMRC provide an Oracle .dmp file and we have to then import that into an Oracle instance and generate the CSV ourselves?

Contributor

saulius commented Dec 12, 2012

@jabley you generated those CSVs for us :-)

I still got Oracle binary files (not .dmp, no extension), I can send them to you, because I don't know how the magic happens.

matthewford added a commit that referenced this pull request Jul 11, 2016

Remove logstasher (#40)
We are going to use papertrail in the CF environment. this can be
configured to read the info from the log stream in cloud foundry, no
need to add code in the app.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment