CSW harvesting fails for all records

[FuhuXia]

On Catalog, CSW harvest stopped working. The harvesting generated an error for each record:

Error getting the CSW record with GUID <###>

How to reproduce

https://catalog.data.gov/harvest/opentopography-csw/job/55b8e5dd-7700-4aa1-9956-2338168c2c2c

Expected behavior

Successful harvest report

Actual behavior

Error for all records

Context

Harvest source side there is no changes. It should be related to recent requirements update we did a few weeks ago.

Sketch

• Roll back to working version of code (test CSW harvest locally to verify)
• Explore where CSW harvest code is located (ckanext-spatial, geodatagov, etc)
• Debug offending code
• Raise to team for broken components and possible fixes
• Make fixes (on fork if necessary)
• Add integration test for CSW harvest (to not backslide)
• Deploy latest working changes

[jbrown-xentity]

First step is to roll back to working version of code, we believe this broke on this PR.

[jbrown-xentity]

Using the latest catalog main, csw harvests work for ckan2.9 on python3.
Next I will examine the fcs branch.

[jbrown-xentity]

On FCS, the spatial code does not seem to work. I believe it is due to needed later versions of OWSLib, but those are not working well on Python2. I can't find much documentation on the support of OWSLib.
The error comes from here in the ckanext-spatial code. After debugging, I get the following:

fetch_stage error: descriptor 'lower' of 'str' object needs an argument

I believe this is deep in OWSLib, and debugging is painful. My suggestion is to pin spatial in FCS to not get the latest, and only use the latest for Py3 and CKAN2.9.

[nickumia-reisys]

I agree with James. I think the main concern was that we weren't sure if harvesting was working for PY3 and it was verified that it does. Tracking two different versions for FCS and cloud.gov is the right approach from my perspective too. I think it's just important to make sure that all of the relevant components were upgraded and they are..

ckanext-spatial,

• catalog.data.gov main version, gsa/ckanext-spatial.git@36981c39d624685bc85b84f801f2a2d6eaea67f6
• gsa/ckanext-spatial datagov version, 36981c39d624685bc85b84f801f2a2d6eaea67f6

ckanext-harvest,

• catalog.data.gov main version, GSA/ckanext-harvest.git@fd5a6b739666a62cff486873877eef10523aec1a
• gsa/ckanext-harvest json-serialization-fix, fd5a6b739666a62cff486873877eef10523aec1a

ckanext-geodatagov,

• catalog.data.gov main version GSA/ckanext-geodatagov.git@0853bee402148f7ccabc8f37ca8c10088e6dcb42
• gsa/ckanext-geodatagov, 0853bee402148f7ccabc8f37ca8c10088e6dcb42

ckanext-harvest does have upstream edits and we have two PY3 upgrades that we needed to do, this will get messy if it continues,

• json-serialization-fix is 8 commits behind upstream
  • https://github.com/ckan/ckanext-harvest/commits/master
  • https://github.com/GSA/ckanext-harvest/commits/json-serialization-fix
• Fix harvest extras for packages ckan/ckanext-harvest#458
• Harvest Source Extra Fields are not encoded into JSON appropriately ckan/ckanext-harvest#450