Merge Lots extension

[jpmckinney]

Discussion

In retrospect, it would have been better to have lots in OCDS (not in an extension), and to always have at least one lot in every process. For example, in the EU, for eForms and the e-Procurement Ontology, the proposal is that, even if the procedure is not divided into lots, there will still be a single, virtual lot.

The advantage is that it makes analysis identical whether the procedure is divided into multiple lots or not. Right now, analysts need to analyze fields on tender and then also fields on lots (if present). The analyst doesn't know what to do if there is conflicting data between lots and tender (e.g. do lots take priority? how to reconcile the two?).

The current work to improve OCDS to cover EU requirements is leading to the duplication of dozens of fields on tender and lots, to satisfy cases where procedures are divided into multiple lots or not. It would be much simpler for users if instead there were just one unit of analysis (the lot) instead of two (the lot and tender objects).

Proposal

For OCDS 1.2, the proposal is to deprecate the fields on tender that ought to be on lots, and to merge the lots extension into the release schema. Then, in OCDS 2.0, those fields would be removed on tender.

[ColinMaudry]

For OCDS 1.2, the proposal is to deprecate the fields on tender that ought to be on lots, and to merge the lots extension into the release schema

The fields that would remain in tender would be those that systematically apply to all lots.

[jpmckinney]

Yes, like the procuring entity, procurement method, etc.

[jpmckinney]

A couple relevant consecutive comments in this issue: #888 (comment) In particular:

[In OCDS, 'Lot' might be better understood as] 'ItemContainer', … described as "A group of one or more items, where each group is bid on, awarded, and supplied separately. If a contracting process has no groups, it is interpreted as having a single group for all items."

Expressed that way, it is easier to explain why OCDS would have a lot even if the original procedure doesn't explicitly refer to any lot.

[jpmckinney]

We need to also think through how to integrate extensions into the docs without overwhelming users.

[ColinMaudry]

We need to also think through how to integrate extensions into the docs without overwhelming users.

By the docs I assum you mean the core docs.

@jpmckinney What would overwhelm them? The quantity of the docs? Or the complexity (regarding this issue, I think it brings simplicity)?

[jpmckinney]

By the docs I assum you mean the core docs.

Yes.

@jpmckinney What would overwhelm them? The quantity of the docs? Or the complexity (regarding this issue, I think it brings simplicity)?

Adding only lots might not be too complex, but I'm thinking if we add many more extensions into core, it'll get quite complex unless we do some things to address that complexity.

Adding more extensions to core would be good – extensions have issues as described here. A simple example is that there is no probability of two EU publishers choosing the correct/same combination of extensions for sub-threshold procedures.

[jpmckinney]

There are deficiencies in the lots modelling, that interacts with deficiencies in the awards/contracts modelling.

A contract object in OCDS should match as much as possible the real-world contract, so that we have the same number of contracts in OCDS as in the real world, with the same values, etc.

Now, multiple lots with distinct values can be awarded on different dates to the same supplier, and later combined into one contract. However, a contract has a single awardID; as such, the lots' awards need to be combined into a single award. (The alternative is to have one award per lot, each with one contract, but then this breaks the above expectation.) When combining the awards, we lose the information about each lot's value and awarded date. We instead have the total value of all lots and some date.

I think changing the cardinality of awardID is too significant a change for a minor release. The only options then are to accept the information loss, or to change the lots extension (open-contracting/ocds-extensions#162). However, this would create very unusual and hard-to-describe semantics (viz. #895 #1175), e.g. an award is as currently defined in #895, except if multiple lots are awarded to the same supplier, in which case an award is an aggregate of other awards...

Originated in #1175 (comment)

[JachymHercher]

To evaluate the options, e.g. the "we live with the information loss" approach, we need to have an intuition on when we can move to OCDS 2.0 to solve it. Do you have one?

I feel like "2.0" necessarily sounds like "revolution", but in practice it could be way smaller than a "1.2", as long as we propose just a limited set of changes that are not backwards compatible, but still pretty straightforward (e.g. award ID arrays).

[jpmckinney]

Yes, I think a "minimal" 2.0 is preferred to a "revolutionary" 2.0 (which would likely wait for a 3.0). I imagine a minimal 2.0 would address #866, which would require the introduction of contracting planning processes, and a solution for framework agreements, which is not so straightforward.

However, I have no estimate on when that will occur. It is likely on the scale of years.

[JachymHercher]

Now, multiple lots with distinct values can be awarded on different dates to the same supplier, and later combined into one contract. However, a contract has a single awardID; as such, the lots' awards need to be combined into a single award. (The alternative is to have one award per lot, each with one contract, but then this breaks the above expectation.) When combining the awards, we lose the information about each lot's value and awarded date. We instead have the total value of all lots and some date.

I think "live with the information loss" is the right approach, mainly because no user has complained so far, just me. The likely reason is that the described case is not that common.

That being said, I would "vote" for an OCDS 2.0 sooner rather than later. For example, in the EU, eForms should be usable from November 2022 and mandatory from October 2023. Since they allow the described case, its occurrence is likely to rise, which could cause problems for eForms publishers wanting to use an OCDS-based system. (Having the eForms-OCDS mapping finished before completing OCDS 2.0 might also be useful - in case new differences are discovered and they would require backwards incompatible changes to solve - but then the calendar starts looking stretched.)

[jpmckinney]

@ColinMaudry Is already working on eForms-OCDS. At any rate, we can't do 2.0 until we do 1.2; we'd need at least twice as many people to do both at once. We've tried to find experts to work on this, and they are very few.

[duncandewhurst]

Does this issue require further discussion or can a PR be prepared based on the proposal in the issue description?

[jpmckinney]

Moving toward a more-or-less mandatory lot is a significant change that will affect a lot of the documentation – not just the schema. For this issue, I think we need to do some community consultation before preparing the (large) PR.

[jpmckinney]

Postponing to 1.3 due to low capacity, and in favor of completing 1.2 on a shorter timeline with fewer controversial or complicated changes.

[jpmckinney]

Some interesting context from eForms: https://op.europa.eu/en/web/ted-eforms/minutes-4th-eforms

Q: There is a duplicity in filling data into form. In case the order has no lots (i.e. has 1 lot), the data in GR-Lot-PreviousPlanning, GR-Lot-Description, GR-Lot-PlaceOfPerformance are the very same as the ones filled in GR Procedure.
Answer: There's currently no solution for this in the SDK. There is always one technical lot at least. The XML must contain all the required fields as this is how the information has been conceived by the eForms regulation.