*_digiwhist spiders are no longer available

[yolile]

No description provided.

[yolile]

It has been more than 6 months but we know they are working on bringing them back again, so I will not delete these ones.

[yolile]

It seems they are publishing OCDS data again https://opentender.eu/all/download

[yolile]

URLs look like https://opentender.eu/data/downloads/data-ee-ocds.zip
And now the zip files have one JSONL file per year, and an additional file with "unavailable" OCDS data, which we should omit when processing the ZIP files, example:

{"uri":"https://opentender.eu//protected/master_tender/id/4d4c2023-6a7f-4bcc-b780-d220f97c8a94?format=ocds","version":"1.1","publishedDate":"2024-04-09T23:34:00.979Z","publisher":{"name":"opentender.eu"},"extensions":["https://raw.githubusercontent.com/open-contracting/ocds_lots_extension/v1.1.1/extension.json","https://raw.githubusercontent.com/open-contracting/ocds_bid_extension/v1.1.1/extension.json","https://raw.githubusercontent.com/open-contracting/ocds_requirements_extension/master/extension.json"],"metaData":{"lastModified":"2024-01-30T13:53:25.942576"}}
{"uri":"https://opentender.eu//protected/master_tender/id/1fd8ad40-aa8b-4161-8a60-a37e48623852?format=ocds","version":"1.1","publishedDate":"2024-04-09T23:34:02.435Z","publisher":{"name":"opentender.eu"},"extensions":["https://raw.githubusercontent.com/open-contracting/ocds_lots_extension/v1.1.1/extension.json","https://raw.githubusercontent.com/open-contracting/ocds_bid_extension/v1.1.1/extension.json","https://raw.githubusercontent.com/open-contracting/ocds_requirements_extension/master/extension.json"],"metaData":{"lastModified":"2024-01-30T16:33:08.291837"}}
{"uri":"https://opentender.eu//protected/master_tender/id/e623379f-18ed-43d5-b98f-bb0d77112bbe?format=ocds","version":"1.1","publishedDate":"2024-04-09T23:34:02.467Z","publisher":{"name":"opentender.eu"},"extensions":["https://raw.githubusercontent.com/open-contracting/ocds_lots_extension/v1.1.1/extension.json","https://raw.githubusercontent.com/open-contracting/ocds_bid_extension/v1.1.1/extension.json","https://raw.githubusercontent.com/open-contracting/ocds_requirements_extension/master/extension.json"],"metaData":{"lastModified":"2024-01-30T16:23:48.080752"}}

[jpmckinney]

Hmm, it’s “year unavailable” - why not process it? The example has no releases, but maybe other ZIPs do?

[jpmckinney]

We can also check for improvement as I think they said they were making some specific changes.

[yolile]

Hmm, it’s “year unavailable” - why not process it? The example has no releases, but maybe other ZIPs do?

Okay, I didn't exclude them but Kingfisher Process is not able to process this type of file, throwing a UnknownFormatError, example.

[yolile]

Ignoring them now ba34c47

[jpmckinney]

Hmm, it's fine for most of the rows. It only fails on those like (reindented):

{
  "uri": "https://staging.opentender.eu//protected/master_tender/id/02a99dc5-0d65-4379-b2b7-708d75bc1d1b?format=ocds",
  "version": "1.1",
  "publishedDate": "2024-04-10T08:17:05.643Z",
  "publisher": {
    "name": "staging.opentender.eu"
  },
  "extensions": [
    "https://raw.githubusercontent.com/open-contracting/ocds_lots_extension/v1.1.1/extension.json",
    "https://raw.githubusercontent.com/open-contracting/ocds_bid_extension/v1.1.1/extension.json",
    "https://raw.githubusercontent.com/open-contracting/ocds_requirements_extension/master/extension.json"
  ],
  "metaData": {
    "lastModified": "2022-09-10T03:59:29.973141487"
  }
}

I'll update OCDS Kit to detect empty packages like this.

[jpmckinney]

Done now. Can you revert ba34c47, and then deploy both Process and Collect? Not sure if you already started crawls after your commit.

[yolile]

Can you revert ba34c47

Done. I started Serbia after my commit but it is finished already. But Uruguay is running in the registry right now (but that one would take several days to be complete so I will deploy now anyway)