Vendor performance ratings

[LindseyAM]

Oscar has documented some examples of governments that have put vender performance rating/tracking systems in place in this memo https://docs.google.com/document/d/17MHslJa4osUGss9q7M4ypS3gUS_0QE0GronR-zbmdDs/edit

@yolile @duncandewhurst is this a topic we are seeing demand for from publishers?

@oahernandez can you add some more info on the most important types of information to track/publish about vendor performance to this issue? Perhaps it could be relevant to a future OCDS extension or local use case of some of our publishers.

[jpmckinney]

@oahernandez More specifically, which pieces of information relevant to vendor performance are made public? I understand that governments often have private, internal feedback and reporting mechanisms, but OCDS is generally not concerned with data that is never made public.

[yolile]

@LindseyAM In Paraguay DNCP publishes the penalties applied to a provider: https://www.contrataciones.gov.py/buscador/sanciones.html
A penalty could derivate into a disqualification, making the supplier unable to participate in procurement processes.

[yolile]

Perhaps it could be relevant to a future OCDS extension or local use case of some of our publishers.

I'm not sure if this information should be an OCDS extension, I think that is a separate data set with which someone can cross the OCDS data

[yolile]

In Chile, they have a rating for the suppliers: http://webportal.mercadopublico.cl/proveedor/85395500-0?_ga=2.226724637.1029645931.1554823751-500745908.1554823751
and also a flag that indicates if the supplier is enable or not to participate in procurement processes.

In Colombia they publish the list of penalties and fees applied to the suppliers: https://www.datos.gov.co/Gastos-Gubernamentales/Multas-y-Sanciones-SECOP-I/4n4q-k399/data
https://www.contratos.gov.co/consultas/detalleProcesoAACS.do?numProcAACS=2618

[jpmckinney]

Thanks, @yolile. I agree that these are separate datasets with different lifecycles – and companies can be added to a buyer's blacklist, etc. for activity unrelated to a contracting process with that buyer. For example, there was a case of SNC Lavalin being barred from bidding on Canadian contracts for having bribed officials in Libya.

Since we have not found evidence of public data on vendor performance, I will close this issue for the time being.

[mw10104587]

An example from Taiwan's New Taipei City.
A vendor is marked in disallowed list for a year.
http://web.pcc.gov.tw/prkms/prms-viewTenderDetailClient.do?ds=20200522&fn=RML-1-50019888.xml

Sharing it here per Charlie's request.

[oahernandez]

I was in a panel at Oxford University a few weeks ago and the topic of vendor ratings & open contracting came up. Here are my notes relevant to this ticket:

Data:

• The US federal qualification database is at www.sam.gov. You need to know to drill down by contractor and by representation (FAR 52.209-5) to find out about debarment from principals. It's hard to use the data unless you have years of experience.
• Germany (under development): https://www.bundeskartellamt.de/EN/CompetitionRegister/CompReg_node.html

Policies:

• U.S. & Korea have reputation ratings.
• The next European directive (Art. 57) will allow centralization of qualification information (to assess performance & repetitional risk) in national or Europe-wide databases and debarments.

Evidence:

• Decarolis, Spagnolo, Pacini (2016). Past performance & procurement outcomes. https://www.nber.org/papers/w22814
• Spagnolo (2012). Reputation, competition, and entry in procurement. https://ideas-repec-org.ez.urosario.edu.co/p/hhs/hasite/0014.html

[jpmckinney]

@oahernandez Is any of this information published in the context of a specific contracting process?

From what I've seen, this info is usually managed and updated outside that context, and therefore would have to be an independent dataset to OCDS.

[LindseyAM]

It would be interesting if a contract record were updated if the vendor was sanctioned/debarred/or otherwise had a performance review related to the contract. in that situation it would make sense as part of OCDS (but as James said, it might be a totally separate dataset that references the performed contracts instead).

[oahernandez]

I think this is similar to other information about the contract implementation stage. I think this would be analogous to information on payments, which usually lives in a separate system but could be modeled under OCDS (?).

The best example is the State of Colorado's contract management system: https://contractsweb.state.co.us/default.aspx The unit of analysis is still the individual procurement process although the information is published separately from information on bids, awards, etc.

[jpmckinney]

Ah, yes, that is something different than what we've seen so far (which were debarments, blacklists, etc. – which can occur due to e.g. criminal charges outside any public contract).

I'll re-open the issue to focus on performance ratings, which are clearly in context of a public contract.

It seems Colorado just stores a single bit of information: whether the contractor performed above, below or at standard.

[jpmckinney]

The MAPS glossary defines:

Term	Definition
Performance	The ability of an entity to acquire resources economically and use those resources efficiently and effectively in achieving performance targets.
Performance Information	Performance information can be generated by both government and nongovernmental organisations, and can be both qualitative and quantitative. Performance information refers to metrics/indicators/general information on the inputs, processes, outputs and outcomes of government policies/programmes/organisations, and can be ultimately used to assess their effectiveness, cost effectiveness and efficiency. Performance information can be found in statistics; the financial and/or operational accounts of government organisations; performance reports generated by government organisations; evaluations of policies, programmes or organisations; or spending reviews, for instance.

[jpmckinney]

For debarments (though this issue is about performance ratings - we just don't have a dedicated issue for debarments that I can find - #901 is related re: the exclusion grounds), the WBG recently published a Global Suspension & Debarment Directory: https://www.worldbank.org/en/about/unit/sanctions-system/osd/brief/exclusion-survey

[jpmckinney]

Other than Colorado, we have not seen evidence of performance ratings in the context of a specific contracting process.

We see plenty of disbarments (or similar), but those are outside the context of a contracting process (and thus of OCDS).

It's not anticipated that we will see many more examples like Colorado, since governments are likely concerned about suppliers taking legal action viz. reputational damage – i.e. there is not much benefit to publication, but there is risk.

So, closing in the absence of data to standardize.