WAF link fixes #51
WAF link fixes #51
Conversation
Determines the ERDDAP version running by scraping the HTML, then adds conditional handling to fetch ERDDAP WAF links depending on the version running: version < 1.82: Fetches links from <pre> element version >= 1.82: Fetches links from <table> inside <div> element
ghost
assigned 
|
Related to ioos/catalog#64 . |
|
I would hope that someone from @axiom-data-science could take a look here since they apparently have customized the ERDDAP services deployed for CENCOOS and SECOORA, and would have some insight as to why CENCOOS is working and SECOORA is not working (ioos/catalog#64 (comment)) |
|
Parent/referring issue should describe the error condition we're running into. I ran locally with these changes and was able to get things harvested properly. |
|
@benjwadams If you think it works via your local testing, lets go ahead and merge and run on the live Registry. There are only a few ERDDAP WAF sources in there, so if there happens to be an issue it will not be a back breaker. |
Removes re module in favor of using str `in` to detect ERDDAP version. Adds function to prevent failure if text is None when scanning for text nodes ending in '.xml' Fixes a couple indentation issues.
|
@rsignell-usgs There is no difference between the CeNCOOS and SECOORA ERDDAP servers. I harvest all of the WAFS defined in the registry fine (including ERDDAP): for x in $(curl 'https://registry.ioos.us/api/v1/Harvests' | jq -r '.data[].url'); do
wget -r -np -A "*.xml" $x
done |
|
@kwilcox , okay, thanks! So @benjwadams , can't we just look at what's happening with CENCOOS and do the same for SECOORA? |
|
@rsignell-usgs , the code merged in here has been moved to production and now is working again. Either the WAF or ERDDAP-WAF harvest types should work for ERDDAP 1.82 now. The previous errors were related to fetching |
Fixes some bugs mostly related to harvesting of ERDDAP 1.82 WAF harvesting