Pull external codes automatically including withdrawn (historic) codes

[andylolz]

This is an update to #51, with a bunch of improvements:

• it pulls currency, language, country and file formats from various secondary sources on https://github.com/datasets
• it pulls DAC CRS codes from https://github.com/datasets/dac-crs-codes
• it uses the status, activation-date and withdrawal-date codelist-item attributes (rather than just status)
• it handles withdrawn codes, and newly activated codes, by comparing the previous version of data with the new version of the data (NB as part of this, a system call to git is made, but this isn’t reeeeally necessary)

This PR replaces #51, and addresses #52.

[andylolz]

Ooooh – I’ve realised I could update this to include the OrganisationRegistrationAgency codelist! 😻 😻 😻

[andylolz]

Somewhat related pull request in open contracting: open-contracting/standard#607

[markbrough]

@dalepotter @amy-silcock @allthatilk @PetyaKangalova @bill-anderson

Following the recent discussions on Discuss, this PR looks like a good way to reduce a lot of the burden on the IATI tech team in terms of keeping codelists up to date (which I think should be one of the highest priorities for the IATI Standard). Is there any plan to proceed or at least review it?

What is the reason for the delay here -- does the tech team not think that these codelists should be kept up to date? If that's the case, it would probably be sensible to start adding disclaimers across the IATIStandard.org site to make users aware that a lot of content is deliberately out of date, and that users should refer to original sources for accurate codelists. (I think it would be far preferable to have up to date and versioned codelists maintained by IATI, it would be much more convenient and useful to retain a record of withdrawn codes.)

Just leaving this PR in limbo seems like a waste to me, and is a shame given that @andylolz has done so much work for free here (and elsewhere). This issue is also over 3 years old now, if you count its predecessor.

How can we move this forward?

[bill-anderson]

it would probably be sensible to start adding disclaimers across the IATIStandard.org site to make users aware that a lot of content is deliberately out of date

Could I point you to Github's Code of Conduct - https://help.github.com/articles/github-community-forum-code-of-conduct/

[markbrough]

Sorry @bill-anderson - I could have been more clear. I mean to write that “a lot of codelists are
intentionally (or known to be) out of date”.

I vaguely recall some discussion previously about whether IATI should be mirroring third-party codelists at all, and also that the DAC should publish their own (though that still doesn't appear to have happened) so I wondered whether the lack of action on this PR reflected a deliberate policy decision?

[andylolz]

This pull request turned one year old yesterday 🎂 🎈

@bill-anderson is there a way to move forward with this?

The code in this pull request is of a quality consistent with the rest of the repo. However, it doesn’t do anything to improve the overall code quality of the repo. I could refactor it so that it does, but it would be good to get a sense of whether that would make it more likely to be merged.

The last we heard on this was from Dale back in April ’17:

Just to update on this issue - we are in discussions with the OECD regarding the publication of machine-readable codelists and are meeting to explore this further in early May. From the outcomes of these conversations, we will have a better view on how to take this work forward and we will update here accordingly.

Is there an update since then? Tagging @PetyaKangalova too.

[hayfield]

A comment to remove this from my Review Requests queue.

[andylolz]

@hayfield I tried to unassign you but it turns out I do not have the Requisite Powers™.

Would you mind unassigning yourself as Dale has done, so it’s clear that this currently has no reviewers / no reviews? TIA.

[hayfield]

I no longer have admin on this repo, so aren't able to do so - would have to be one of the current IATI peeps.

[amy-silcock]

@hayfield I can't remove you :( I'll ask the guys today.

[samuele-mattiuzzo]

I can't seem to be able to remove Hayden either, but I've assigned the issue to myself so that, as correctly pointed out, it doesn't get lost and a decision can be taken. Thank you all!

[andylolz]

From @bill-anderson’s comments elsewhere, it sounds like the intended solution for DAC codelists is to rely on the XML rather than the XLS. In which case, much of this pull request is no longer valid.

I think lots of this might still be useful for country / currency etc, in case someone rediscovers this later and it can be of some use.

Closing.

[andylolz]

The following response from the DAC suggests it still isn’t possible to use their XML directly – it requires some processing in order to get it into a non-embedded codelist:

For the time being we do not have longer these removed codes in our database so it’s not possible for us to provide this information in the xml file. We are tracking changes in purposecodes from 2016 only (new codes starting in 2016 and code removed in 2015).