New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frequency of publication #2

Open
markbrough opened this Issue Jan 22, 2014 · 8 comments

Comments

Projects
None yet
7 participants
@markbrough
Member

markbrough commented Jan 22, 2014

Description
The Index provides a higher score for more frequent publication.

2013 Index test
In 2013, the maximum score for each indicator was:

  • 100% if overall publication was at least quarterly
  • 95% if overall publication was less than quarterly

This was calculated by looking at the frequency of updates recorded on the IATI Registry. If the average number of days between updates was less than 93 days, it was considered quarterly. Otherwise, it was considered less than quarterly.

Issues

  1. The multiplier of 0.95 made only a small difference to scores; but frequency of publication makes a big difference at country level.
  2. Donors publishing more frequently than quarterly (e.g. monthly) were not rewarded for this important improvement in their publication.
  3. Any update in the data by a publisher (for any file) was counted as an update. Even a small change in metadata on the Registry would count as an update.
  4. Donors publishing for the first time were considered to be at least quarterly, even if they eventually published less frequently.
  5. Timeliness of information was not considered (as distinct from frequency).

Questions

  1. Should there be a more graduated multiplier based on frequency of publication? What should the multipliers be?
  2. Should timeliness be captured separately? If so, how? (Understanding when data is captured in donor systems is hard to know if it isn't published.)
  3. What should count as an "update"? Should it a be a quorate number of files? What would that number be? Should we rely on IATI Registry update logs?
  4. Should donors publishing for the first time be considered to publish at the frequency that they specify in their implementation schedule (or alternatively, less than quarterly by default)?

2014 Index test
How should we adjust the frequency tests this year to address these issues?

@ErinCoppin

This comment has been minimized.

Show comment
Hide comment
@ErinCoppin

ErinCoppin Feb 11, 2014

  1. Yes, there should be a more graduated score on the basis of frequency of publication. I suggest multipliers of 0.75 for annual or more, 0.85 for six monthly or more, 0.95 for quarterly or more, and 1.00 for monthly or more.
  2. Although ideally timeliness would be captured, that would require donors to tag the publication date of a document (published outside of IATI) as a separate field, or alternatively the date that it is captured in the donor system (which is nebulous). Unless this already happens, it would seem unlikely to be a priority in negotiations about the standard.
  3. Although it would make sense to have a quorate number of files, this test will become more difficult as the existing data set grows larger. Once donors have 5 years of data, with recent years being (or reasonably expected to be ) larger sets than earlier years, it would be necessary to set a quorate number of files as being considerably less than that which might reasonably constitute one quarter’s worth of data, or even one month. Otherwise, frequent small updates might be lost. It might be worth setting a quorate as 1% for now but revisiting in no more than three years to check assumptions.
  4. Yes, donors should be assumed at first to publish as frequently as their implementation schedules state. But this should then be tested after the appropriate interval. So if a donor states that they will publish monthly, this should be assumed for the first month but then tested. If they state they will publish quarterly, this should be assumed for the first quarter but then tested.

Index: We should adjust the Index in line with (1) and (4) but leave (2) for now. For (3), it would be good to check what setting a quorate number of files at 1%, 5% or 10% would mean before making a decision.

ErinCoppin commented Feb 11, 2014

  1. Yes, there should be a more graduated score on the basis of frequency of publication. I suggest multipliers of 0.75 for annual or more, 0.85 for six monthly or more, 0.95 for quarterly or more, and 1.00 for monthly or more.
  2. Although ideally timeliness would be captured, that would require donors to tag the publication date of a document (published outside of IATI) as a separate field, or alternatively the date that it is captured in the donor system (which is nebulous). Unless this already happens, it would seem unlikely to be a priority in negotiations about the standard.
  3. Although it would make sense to have a quorate number of files, this test will become more difficult as the existing data set grows larger. Once donors have 5 years of data, with recent years being (or reasonably expected to be ) larger sets than earlier years, it would be necessary to set a quorate number of files as being considerably less than that which might reasonably constitute one quarter’s worth of data, or even one month. Otherwise, frequent small updates might be lost. It might be worth setting a quorate as 1% for now but revisiting in no more than three years to check assumptions.
  4. Yes, donors should be assumed at first to publish as frequently as their implementation schedules state. But this should then be tested after the appropriate interval. So if a donor states that they will publish monthly, this should be assumed for the first month but then tested. If they state they will publish quarterly, this should be assumed for the first quarter but then tested.

Index: We should adjust the Index in line with (1) and (4) but leave (2) for now. For (3), it would be good to check what setting a quorate number of files at 1%, 5% or 10% would mean before making a decision.

@theosande

This comment has been minimized.

Show comment
Hide comment
@theosande

theosande Feb 19, 2014

agree with increased granularity, hesitations about rewarding publication for annual or more at all. Not in the spirit of IATI to publish with that frequency

theosande commented Feb 19, 2014

agree with increased granularity, hesitations about rewarding publication for annual or more at all. Not in the spirit of IATI to publish with that frequency

@bill-anderson

This comment has been minimized.

Show comment
Hide comment
@bill-anderson

bill-anderson Feb 20, 2014

The miniscule weighting of this item is the biggest weakness in the index and (together with the low weighting of forward-looking data) is about to cause a lot of confusion when the Global Partnership transparency indicator is made public. The Indicator gives a third of the total score to timeliness and frequency, and another third to forward-looking data. In my opinion frequency and timeliness should account for a fifth of the index score.

bill-anderson commented Feb 20, 2014

The miniscule weighting of this item is the biggest weakness in the index and (together with the low weighting of forward-looking data) is about to cause a lot of confusion when the Global Partnership transparency indicator is made public. The Indicator gives a third of the total score to timeliness and frequency, and another third to forward-looking data. In my opinion frequency and timeliness should account for a fifth of the index score.

@johnadamsDFID

This comment has been minimized.

Show comment
Hide comment
@johnadamsDFID

johnadamsDFID Feb 20, 2014

Q1) We agree with Erin and Bill, we need timeliness to be weighted much higher. Should we even consider a greater range of weighting for key elements (e.g. Busan)?

Q2) Timeliness could be captured by comparing publication dates (from the Dashboard?) with the last-updated-date on each activity. What you want to look for is the smallest gap as there will be a large % of projects that aren't updated because nothing has happened to them. That might address the "what is an update" issue.

Q3) Do IATI Registry update logs allow you to tell if a file contents have been changed even if the URL is the same?

Q4) You either need 2 data points - 2 publications - for a donor, or assume that they are quarterly.

johnadamsDFID commented Feb 20, 2014

Q1) We agree with Erin and Bill, we need timeliness to be weighted much higher. Should we even consider a greater range of weighting for key elements (e.g. Busan)?

Q2) Timeliness could be captured by comparing publication dates (from the Dashboard?) with the last-updated-date on each activity. What you want to look for is the smallest gap as there will be a large % of projects that aren't updated because nothing has happened to them. That might address the "what is an update" issue.

Q3) Do IATI Registry update logs allow you to tell if a file contents have been changed even if the URL is the same?

Q4) You either need 2 data points - 2 publications - for a donor, or assume that they are quarterly.

@bill-anderson

This comment has been minimized.

Show comment
Hide comment
@bill-anderson

bill-anderson Feb 20, 2014

A3) registry logs provide information on actual updates - this is more reliable than last-updated-time. See the methodology used by the IATI annual report - see p134 - http://www.aidtransparency.net/annual-report-2013

bill-anderson commented Feb 20, 2014

A3) registry logs provide information on actual updates - this is more reliable than last-updated-time. See the methodology used by the IATI annual report - see p134 - http://www.aidtransparency.net/annual-report-2013

@HeatherDHanson

This comment has been minimized.

Show comment
Hide comment
@HeatherDHanson

HeatherDHanson Feb 21, 2014

We believe frequency of publication is important to usefulness of the data. Erin's proposed weighting seems to fairly reflect the key differences between annual, semi-annual, quarterly and monthly publication.
At the same time, frequency of publication is an area where changes in scoring are not likely to motivate much behavior change on the part of donors like MCC, who have reporting cycles that are largely defined by standard government practices. Because we are interested in knowing more about use, I would in fact like to know if monthly or quarterly makes any different to end users of IATI data.

HeatherDHanson commented Feb 21, 2014

We believe frequency of publication is important to usefulness of the data. Erin's proposed weighting seems to fairly reflect the key differences between annual, semi-annual, quarterly and monthly publication.
At the same time, frequency of publication is an area where changes in scoring are not likely to motivate much behavior change on the part of donors like MCC, who have reporting cycles that are largely defined by standard government practices. Because we are interested in knowing more about use, I would in fact like to know if monthly or quarterly makes any different to end users of IATI data.

@YohannaLoucheur

This comment has been minimized.

Show comment
Hide comment
@YohannaLoucheur

YohannaLoucheur Feb 22, 2014

Like Heather, I'd be interested to know whether, indeed, frequency of publication makes a real difference for end users (esp. between monthly and quarterly).
In a universe of limited resources, it'd be useful to know whether this should take precedence in our efforts over other potential improvements (eg transactions, ID numbers, budget identifier, forward-looking activity data).

YohannaLoucheur commented Feb 22, 2014

Like Heather, I'd be interested to know whether, indeed, frequency of publication makes a real difference for end users (esp. between monthly and quarterly).
In a universe of limited resources, it'd be useful to know whether this should take precedence in our efforts over other potential improvements (eg transactions, ID numbers, budget identifier, forward-looking activity data).

@markbrough

This comment has been minimized.

Show comment
Hide comment
@markbrough

markbrough Mar 11, 2014

Member

Thanks for all the comments.

For 2014
In 2014, a more nuanced scoring approach to frequency will be used to monitor if donors are publishing monthly, quarterly or less frequently, in line with the Common Standard commitment and partner countries request for data at a minimum on a quarterly basis.

In 2014, publishing monthly allows an organisation to achieve the maximum score of 100 points; publishing quarterly up to 95 points; and publishing less than quarterly up to 75 points.

The frequency of publication is calculated based on the number of months in which there are updates in the previous six month period as recorded in the IATI Registry logs. To score as a monthly publisher, an organisation needs to update its files in five of the previous six months (January–June 2014, at the end of data collection). For quarterly, the organisation needs to update its files in two of the previous six months. The frequency of publication used for organisations included in the Index can be found in the IATI Updates section of the Aid Transparency Tracker. The six month window is defined as 184 days, which is the maximum number of days in any six month period.

As in 2013, timeliness will not be captured separately this year, as we want to do some more work in investigating possible methodologies before including such a measure.

A note on frequency

More frequent publication has long been a key demand of partner countries. This demand was recognised by the Steering Committee decision of 7 July 2010: "the frequency of publishing data will be as soon as possible and at least quarterly". Monthly publication was also encouraged by partner countries; the Malawian representative noted that "this was the minimum for that data to be useful for internal budget reporting and IMF reporting". The IATI Country Survey of AIMS emphasises the importance of quarterly publication to 87% of respondents, and the importance of monthly publication to 61% of respondents.

We are conscious of the needs of users of this data. However, we are also conscious of the comments arguing that there is an opportunity cost to increased frequency of publication, and we do not want to divert resources away from more complete publication entirely.

Member

markbrough commented Mar 11, 2014

Thanks for all the comments.

For 2014
In 2014, a more nuanced scoring approach to frequency will be used to monitor if donors are publishing monthly, quarterly or less frequently, in line with the Common Standard commitment and partner countries request for data at a minimum on a quarterly basis.

In 2014, publishing monthly allows an organisation to achieve the maximum score of 100 points; publishing quarterly up to 95 points; and publishing less than quarterly up to 75 points.

The frequency of publication is calculated based on the number of months in which there are updates in the previous six month period as recorded in the IATI Registry logs. To score as a monthly publisher, an organisation needs to update its files in five of the previous six months (January–June 2014, at the end of data collection). For quarterly, the organisation needs to update its files in two of the previous six months. The frequency of publication used for organisations included in the Index can be found in the IATI Updates section of the Aid Transparency Tracker. The six month window is defined as 184 days, which is the maximum number of days in any six month period.

As in 2013, timeliness will not be captured separately this year, as we want to do some more work in investigating possible methodologies before including such a measure.

A note on frequency

More frequent publication has long been a key demand of partner countries. This demand was recognised by the Steering Committee decision of 7 July 2010: "the frequency of publishing data will be as soon as possible and at least quarterly". Monthly publication was also encouraged by partner countries; the Malawian representative noted that "this was the minimum for that data to be useful for internal budget reporting and IMF reporting". The IATI Country Survey of AIMS emphasises the importance of quarterly publication to 87% of respondents, and the importance of monthly publication to 61% of respondents.

We are conscious of the needs of users of this data. However, we are also conscious of the comments arguing that there is an opportunity cost to increased frequency of publication, and we do not want to divert resources away from more complete publication entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment