Skip to content

codeforIATI/iati-data-dump

main
Switch branches/tags
Code
This branch is 63 commits ahead, 2 commits behind IATI:master.
Contribute

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

IATI Data Dump

A daily snapshot of all IATI data (and metadata) on the IATI registry.

Download all IATI data

Rationale

Lots of IATI products do a daily pull of all IATI XML data. That involves downloading gigabytes of data, via thousands of HTTP requests. So it’s quite slow.

Downloading a single archive file is significantly faster. That’s available here!

How does this work?

We perform a daily fetch of all IATI data, compress it, and uploads the archive. There’s a metadata.json file inside the zip, that lets you know how fresh the data is.

This github gist is updated with the list of erroring datasets.

Acknowledgements

The original code relied heavily on IATI Registry Refresher, made by @caprenter and @Bjwebb.

The new, much faster code was contributed by @notshi and @xriss.

About

📷 A daily snapshot of all IATI data on the IATI Registry

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Languages

  • Shell 56.0%
  • Python 43.3%
  • Ruby 0.7%