Tracking implementation of properties

[jpmckinney]

As explained in #60, if there are no implementations for a property, that property may be a candidate for removal (other considerations as in @practicalparticipation's comment may keep it in). Here's a script to check which properties releases use. For brevity, not all unused fields are listed in this issue description.

• To do: Check use of codes in code lists (enum) as in IATI Dashboard

Development Gateway

The DG data doesn't have any planning, contracts or performance fields. Within buyer, formation and awards, there are some unused fields as well.

buyandsell.gc.ca sample

(Note: Didn't process the full 1GB JSON.) No planning or performance fields. Within buyer, formation, awards, contracts, there are some unused fields as well. Conformance issues

[birdsarah]

This is a very supply-centric position. We have looked closely at supply-side, but are also looking at demand-side stakeholders. The standard is also part of advocay, and what open contracting data could and should be, not just what it is.

Looking at what fields a release uses is probable not ideal, you probably want to look at a compiledRelease from a record as releases are expected to be sparse.

You can also see the mappings that I've done for the canadian data here: https://github.com/open-contracting/sample-data/tree/master/buyandsell/mappings

[jpmckinney]

Not sure that a standard should be doing the advocacy itself unless it's first been validated that data can be supplied in the format being advocated... I believe advocacy is important, but how can you standardize and technically define a field if you haven't yet seen the supply-side version of that data?

Re: compiledRelease, if I check all the releases ever published, doesn't it come out to the same result? A compiled release doesn't add in new data that wasn't in any of its component releases... If you can point me to any compileRelease, I'll analyze it, but I think at this early stage the only independent OCDS data is that provided by Development Gateway in the linked GitHub repo. I'd be eager to analyze any others.

I appreciate the mappings, but they can't be counted as an implementation of the spec.

[birdsarah]

I completely understand what you're saying. I'm going to step aside this discussion, I think people from the open contracting partnership are best placed to strategize on this topic. @practicalparticipation - who do you think we should consult on this?

"if I check all the releases ever published, doesn't it come out to the same result?" - yes :)

I'm reticent on the emphasis you're putting on the DG data - it is, by definition, awards data - to say that it doesn't have planning, contracts, performance is a bit circular. We know that there's lots more data out there, that's why we did an extensive supply-side analysis to get to where we are (as well as including demand-side stakeholders). So I appreciate what you're saying in terms of wanting to review the data, but we have looked at nearly 500 fields from different publishers to get here.

I'm curious why the canadian sample data is less of an implementation than the DG data? I mean its only a test, but is the DG data really an implementation, or is it also a data transformation / dump? the DG data doesn't include records which are required.

[birdsarah]

So we're clear, I think tracking implementation of properties is an excellent idea.

This whole conversation is probably better on #60 as my issue is with cutting fields that aren't implemented, which seems counter to the whole strategy we've been working with. But as I said, others in the partnership probably have much clearer direction on this. @practicalparticipation and @michaeloroberts may have more light too.

[jpmckinney]

I'll update this issue as more OCDS data is published. Right now I only know of DG, so that's all I've tested for now.

An important step on the way from draft to "standard" is to check that the documentation works: the documentation should effectively communicate how to implement the spec. If two implementers understand the spec differently, then the documentation isn't working. I know that you can implement the spec - you're on the team that wrote it! - that's why the Canadian data that you produced doesn't count; it's not a good test of whether the documentation works. That's why #60 requires independent implementations.

Similarly, if no implementers implement a particular field, then that field is not a good candidate for standardization.

An easy way to avoid cutting fields is to extend the period in which you collect implementations. On a short timeline, you aren't going to get full coverage.

[birdsarah]

"An important step on the way from draft to "standard" is to check that the documentation works: the documentation should effectively communicate how to implement the spec." - this is a great point, thanks!

[practicalparticipation]

This is a really valuable discussion. I'll try and give some reflections below. What follows has turned out a bit longer and discursive than I was aiming for, but I will post the discursive version here, and will reply on #60 around the specifics of process.

Two senses of standard

In Open Contracting I believe we're dealing with two different senses of 'standard', and two purposes which we need to keep in balance. Namely:

• Standards as a basis for interoperability - as in *"their data complies with the standard, and can be used by standards-compliant tools."
• Standards as targets - as in, "they have achieved a high standard of disclosure".

To unpack these a bit:

(Note: the arguments below are predominantly theoretical, and so some of the edge cases considered may not come up at all in practice in the Open Contracting Data Standard, but considering them is a useful exercise to test the intuitions and principles directing our action.)

Standards as interoperability

We're interested in interoperability in two directions: vertical (can a single dataset be used by other actors and tools in a value-chain of re-use), and horizontal (can two datasets from different publishers be easily analysed alongside one another).

Where data is already published, then the goal should be to achieve the largest possible set of data publishers who can richly represent their data in the standard, and of data users who can draw on data in the standard to meet their needs. This supports the idea that for any element in the standard where (a) data already exists; and (b) use cases already exist; we should be looking for reference implementations to test that data can be rendered in the standard, and that users (or tools they create) can read, analyse and use that data effectively.

However, it is important that in this we look at both both horizontal and vertical interoperability in making this judgement. E.g. there could be a country as the sole publisher of a field that is used by 5 different users in their country. This should clearly not be a required field in a standard, but articulating how it is standardised is useful to this community of users (one way to accommodate such cases may be in extensions, although the judgement on whether or not to move something to an extension might come down to whether it is likely that other publishers could be providing this data in future).

In many cases, underlying data from different sources is not perfectly interoperable, or there is a mismatch between the requirements of users, and the requirements of data holders. In these cases, the way a standard is designed affects the distribution of labour between publishers and users with respect to rendering data interoperable. For example, a use case might involve 'Identifying which different government agencies, each publishing data independently, have contracts with a particular firm'. In this case, a standard could require all publishers, who may store different identifiers in their systems, to map these to a common identifier, or a standard could allow publishers to use whatever identifier they hold, leaving the costs of reconciling these on the user. Making things interoperable then involves can involve then a process of negotiation, and this process may play out differently in different places at different times, leaving certain elements of a standard less stable than others. The concept of 'designing for the tussle' (PDF) may be relevant here, thinking about how we can modularise stable (or 'neutral') and unstable elements of a standard (this is what the proposed Organisation ID standard does, but having a common way to represent identifiers, but separating this off from the choice of identifier itself, and then allowing for the emergence of a set of third-party tools and validation routines to help manage the tussle).

In seeking to maximise the set of publishers and users interoperable through the standard we need to be critically aware of both short-term and long-term interoperability, as organisations modify their practices in order to be able to publish to, or draw upon, a common standard. We need to balance out a 'Lowest Common Denominator' (LCD) of 'Minimum Viable Product' (MVP) approach that means that the majority of publishers can achieve substantial coverage of the standard, with a richer standard that supports the greatest chance of different producer and consumer groups being able to exchange data through the standard.

(Initial attempt to sketch distinction between maximising set of common fields across publisher and users, and maximising set of publishers and users)

Standards as targets

Open Contracting is a political process. The Open Contracting Partnership have articulated a set of Global Principles which set out the sorts of information about contracting that governments and other parties should disclose, and they are working to secure government sign-up to these principles. In policy circles, a standard is often seen as a form of measure, qualitative or quantitative, against which process towards some policy goal is measured. Some targets might be based on 'best practice', others are based on 'stretch goals': things which perhaps no-one is yet doing particularly well, but which a community of actors agree are worth aiming for. A standard, whether specified in terms of indicators and measures, or in terms of fields and formats, provides a means of agreeing what meeting the target will look like.

The Open Contracting Principles call for a lot of things which no governments appear to yet be publishing in machine-readable forms. In many cases we've not touched the standardisation of these right now (e.g. "Risk assessments, including environmental and social impact assessments") recognising that standards for these will either exist in different domains that can be linked or embedded into our standard, or, recognising that interoperability of such information is hard to achieve and ultimately what is needed for most use cases may be legal text or plain language documents, rather than structured data. However, there may be cases where something is a strong candidate for standardisation, having both the potential to be published (i.e. this is something which evidence suggests governments either do, or could, capture in their existing information systems), and for which clearly articulated use cases exist. In these cases a proposed field-level standard can act as an important target for those seeking to provide this data to move towards. It also acts to challenge unwarranted 'first mover advantage' where the first person to publish, even if publishing less than an idea target would require, gets to set the standard, and instead makes the 'target' subject to community discussion.

Clearly any 'aspirational' elements of a standard should not predominate or make up the majority of a standard if it seeks to effectively support interoperability, but in standards that play a part in policy and political processes (as, in practice, all standards do to some extent (c.f. Lessig).

Implications

There are a number of ways we might respond to a recognition of the dual role that standardisation plays in Open Contracting.

Purposes and validation sets

One approach, suggested in the early technical scoping is to identify different sets of users, or 'purposes' for the standard, and for each of these to identify the kinds of fields (subset of the data) these purposes require. As Jeni Tennison's work on the scoping describes "...each purpose can have a status (eg proposed vs implemented) and ... purposes are only marked as implemented when there are implementations that use the given subset of data for the specified purpose".

If their are neither purposes requiring a field, nor datasets providing a field, then it would not be suitable for inclusion in a standard. And if a purpose either went unimplemented for a long period, or required a field that no supplier could publish, then careful evaluation would be needed of whether to remove that purpose (or remove that field from the purpose) against which elements of the standard could be evaluated for relevance to remain in the model.

Purposes could also be used to validate datasets, identifying how many datasets are fit for which purpose.

Stable, ordinary and target elements

We could maintain a distinction in how the standard is described between fields and elements which are 'stable' (and thus very unlikely to change), 'ordinary' elements (which may have reference implementations, but could change if there was some majority interest amongst those governing a standard in seeing changes), and 'target' elements, which may lack any reference implementations, but which are considered useful to help publishers moving towards implementing a political commitment to publish.

Q: Could we build this information into the schema meta-data somehow?

We might need to have quite a long time horizon for keeping target elements provisionally in the standard, and to only remove them if there is agreement that no-one is likely to publish to them. However, being able to represent them visually as distinct in the schema, and clearly documenting the distinction may be valuable.

Extensions

Some 'target' elements may best belong in extensions, with some process for merging extensions into the core standard if they are widely enough adopted.

Regular implementation monitoring

The IATI Team run a dashboard which tracks use of particular fields in the data. Doing similar for Open Contracting would be valuable, and it may even be useful to feed such information into the display of the schema or documentation (or at least to make it easy for publishers and users to look up who is implementing a given property)

Implementation schedules

Another approach IATI uses for 'target elements' is to ask publishers to prepare 'Implementation Schedules' which outline which fields they expect to be able to publish by when. This allows an indication of whether there is political will to reach some of the 'stretch targets' that might be involved in a standard, and holds out the potential to convene together to define and refine target standardisations those who are most likely to publish that data in the near to medium term.

[jpmckinney]

Thanks for the detailed reflections! I've updated the issue description so that implementation is not framed as the only criteria for inclusion. My script can be used to build an eventual dashboard. I like the proposals contained in the implications above.

[birdsarah]

This has been a great discussion. By launch, I'd like us to close this issue and re-formulate any specific points as new issues for consideration by the governance body / working groups as required.

[LindseyAM]

there is a lot here @marcelarozo @sdavenpo422

[birdsarah]

This also got turned into a blog post - http://www.timdavies.org.uk/2014/09/16/two-senses-of-standard/

[birdsarah]

Assigning @jpmckinney to break out any specific issues that remain for RC1.

[jpmckinney]

Nothing for RC1

[birdsarah]

I'm moving this to after RC1 then.

[timgdavies]

@kindly Can you review this conversation - and consider if there is anything we can do with data from CoVE (OCDS Validator) or any other tools to analyse current usage of properties across the standard, and to identify properties currently heavily or lightly used.

This may also be something @ekoner could explore through a sprint of research on our existing data.

[timgdavies]

@ekoner has worked on a manual analysis of properties currently in use across a sample of files. This is something we will look to develop automated tooling for in future, particularly building on the new API specification to support harvesting of data.

The analysis is available here.

This highlights:

• No use of documents at the contract implementation, and implementation milestone level
• No use of transaction fields in implementation to record actual payments
• Limited use of amendment fields
• Limited use of additional identifier fields

Specific fields that do not occur in our sample include:

• planning/budget/description
• planning/budget/id
• planning/budget/projectID
• planning/budget/source
• planning/budget/uri
• planning/rationale
• tender/awardCriteriaDetails
• tender/minValue/currency
• tender/submissionMethodDetails

However, we have encountered a number of planning blocks in draft data, not included in the analysis, so consider that planning should not be considered for deprecation.

We should investigate use of awardCriteriaDetails and submissionMethodDetails in more depth to see if these should be proposed for removal.

Implications

• We should consider removing documents from milestones. This does not appear to be used, and adds an extra level of depth and complexity to the standard.
• We should consider the use cases for transactions. There is a case to say it remains early days to see integration of spend data into OCDS, but this remains desirable. However, if there is an indication that it is prohibitively complex, or could be better handled by revised engagement with the Fiscal Data Package we should consider this.

[LindseyAM]

@gabelula can you remind us - is there documents associated with milestones in the cdmx data?

[timgdavies]

A new phase of implementation research is currently planned and we'll open a new thread with results.