New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull external codes automatically including withdrawn (historic) codes #172

Closed
wants to merge 30 commits into
base: master
from

Conversation

Projects
None yet
8 participants
@andylolz
Copy link
Contributor

andylolz commented Aug 30, 2017

This is an update to #51, with a bunch of improvements:

  • it pulls currency, language, country and file formats from various secondary sources on https://github.com/datasets
  • it pulls DAC CRS codes from https://github.com/datasets/dac-crs-codes
  • it uses the status, activation-date and withdrawal-date codelist-item attributes (rather than just status)
  • it handles withdrawn codes, and newly activated codes, by comparing the previous version of data with the new version of the data (NB as part of this, a system call to git is made, but this isn’t reeeeally necessary)

This PR replaces #51, and addresses #52.

Bjwebb and others added some commits Feb 3, 2015

[#8] Update IANA Media Types import code for CLv3
Also, pull the latest list of codes.
[#9] Remove historic 2 letter codes from Country list
4 letter codes should be used instead.
Rewrite of the convert.py code
This mostly handles withdrawals. It works for all DAC CRS codelists, but
some of the others still need a bit of work.
@andylolz

This comment has been minimized.

Copy link
Contributor

andylolz commented Oct 27, 2017

Ooooh – I’ve realised I could update this to include the OrganisationRegistrationAgency codelist! 😻 😻 😻

@andylolz

This comment has been minimized.

Copy link
Contributor

andylolz commented Nov 1, 2017

Somewhat related pull request in open contracting: open-contracting/standard#607

@markbrough

This comment has been minimized.

Copy link

markbrough commented Jun 26, 2018

@dalepotter @amy-silcock @allthatilk @PetyaKangalova @bill-anderson

Following the recent discussions on Discuss, this PR looks like a good way to reduce a lot of the burden on the IATI tech team in terms of keeping codelists up to date (which I think should be one of the highest priorities for the IATI Standard). Is there any plan to proceed or at least review it?

What is the reason for the delay here -- does the tech team not think that these codelists should be kept up to date? If that's the case, it would probably be sensible to start adding disclaimers across the IATIStandard.org site to make users aware that a lot of content is deliberately out of date, and that users should refer to original sources for accurate codelists. (I think it would be far preferable to have up to date and versioned codelists maintained by IATI, it would be much more convenient and useful to retain a record of withdrawn codes.)

Just leaving this PR in limbo seems like a waste to me, and is a shame given that @andylolz has done so much work for free here (and elsewhere). This issue is also over 3 years old now, if you count its predecessor.

How can we move this forward?

@bill-anderson

This comment has been minimized.

Copy link

bill-anderson commented Jun 27, 2018

it would probably be sensible to start adding disclaimers across the IATIStandard.org site to make users aware that a lot of content is deliberately out of date

Could I point you to Github's Code of Conduct - https://help.github.com/articles/github-community-forum-code-of-conduct/

@markbrough

This comment has been minimized.

Copy link

markbrough commented Jun 27, 2018

Sorry @bill-anderson - I could have been more clear. I mean to write that “a lot of codelists are
intentionally (or known to be) out of date”.

I vaguely recall some discussion previously about whether IATI should be mirroring third-party codelists at all, and also that the DAC should publish their own (though that still doesn't appear to have happened) so I wondered whether the lack of action on this PR reflected a deliberate policy decision?

@andylolz

This comment has been minimized.

Copy link
Contributor

andylolz commented Aug 31, 2018

This pull request turned one year old yesterday 🎂 🎈

@bill-anderson is there a way to move forward with this?

The code in this pull request is of a quality consistent with the rest of the repo. However, it doesn’t do anything to improve the overall code quality of the repo. I could refactor it so that it does, but it would be good to get a sense of whether that would make it more likely to be merged.

The last we heard on this was from Dale back in April ’17:

Just to update on this issue - we are in discussions with the OECD regarding the publication of machine-readable codelists and are meeting to explore this further in early May. From the outcomes of these conversations, we will have a better view on how to take this work forward and we will update here accordingly.

Is there an update since then? Tagging @PetyaKangalova too.

@hayfield
Copy link
Contributor

hayfield left a comment

A comment to remove this from my Review Requests queue.

@andylolz andylolz force-pushed the andylolz:9-historical-codes branch 3 times, most recently from d47a515 to 48a857b Sep 10, 2018

@dalepotter dalepotter removed their request for review Sep 10, 2018

@dalepotter dalepotter removed their assignment Sep 10, 2018

@andylolz

This comment has been minimized.

Copy link
Contributor

andylolz commented Sep 10, 2018

@hayfield I tried to unassign you but it turns out I do not have the Requisite Powers™.

Would you mind unassigning yourself as Dale has done, so it’s clear that this currently has no reviewers / no reviews? TIA.

@hayfield

This comment has been minimized.

Copy link
Contributor

hayfield commented Sep 11, 2018

I no longer have admin on this repo, so aren't able to do so - would have to be one of the current IATI peeps.

@amy-silcock amy-silcock requested review from samuele-mattiuzzo and Ocre42 Sep 11, 2018

@amy-silcock

This comment has been minimized.

Copy link
Contributor

amy-silcock commented Sep 11, 2018

@hayfield I can't remove you :( I'll ask the guys today.

@samuele-mattiuzzo samuele-mattiuzzo self-assigned this Sep 11, 2018

@samuele-mattiuzzo

This comment has been minimized.

Copy link
Contributor

samuele-mattiuzzo commented Sep 11, 2018

I can't seem to be able to remove Hayden either, but I've assigned the issue to myself so that, as correctly pointed out, it doesn't get lost and a decision can be taken. Thank you all!

@andylolz

This comment has been minimized.

Copy link
Contributor

andylolz commented Nov 17, 2018

From @bill-anderson’s comments elsewhere, it sounds like the intended solution for DAC codelists is to rely on the XML rather than the XLS. In which case, much of this pull request is no longer valid.

I think lots of this might still be useful for country / currency etc, in case someone rediscovers this later and it can be of some use.

Closing.

@andylolz andylolz closed this Nov 17, 2018

@andylolz andylolz deleted the andylolz:9-historical-codes branch Nov 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment