Registration of new CariCOOS webservices

[robragsdale]

@jorgecapella I have submitted the new CariCOOS services
I had to register one catalog level down from the the top level catalog http://dm2.caricoos.org/thredds/catalog .html because the NGDC metadata crawler cannot search through catalogRefs –

I will track these through the registration process. We will focus on getting the buoy data service http://dm2.caricoos.org/thredds/catalog/buoys/catalog.html in the catalog first.

Anna, does Jorge need to create aggregate files from these services?

Rich, do you have time to help Jorge aggregate their ROMS?

Thanks,
Rob

On Tue, Mar 10, 2015 at 5:19 PM, Jorge E. Capella jorge.capella@upr.edu wrote:
Rob:

As of today the only CariCOOS entry in the IOOS Catalog http://catalog.ioos.us/services/filter/CARICOOS is for the CariCOOS SWAN Best-Aggregated Output, service types WMS and DAP;

NGDC's Collection Source List
https://www.ngdc.noaa.gov/docucomp/collectionSource/list?recordSetId=3573894&componentId=&serviceType=&serviceStatus=SUBMITTED&serviceUrl=&search=List+Collection+Sources
includes just three items for CariCOOS that to me look like a random pick;

NGDC's ISO Metadata Records
http://www.ngdc.noaa.gov/metadata/published/test/NOAA/IOOS/CariCOOS/iso_u/
is a long list of satellite imagery and non-aggregated SWAN items.

Focusing on the IOOS Catalog:

1. Aggregated SWAN seems to be the only bright spot. We keep the non-aggregated model output in the server for historical, legacy, issues. We have been asked to remove the non-aggregated output, due to the large # of files, but we would rather keep it even if it is hidden. Which would be the preferred option?
   http://dm2.caricoos.org/thredds/dodsC/swan/SWAN_Aggregation_best.ncd
   http://dm2.caricoos.org/thredds/catalog/content/swan/catalog.html

2. Yes, we need to best-aggregate ROMS; even though we have an entry for it, we haven't been able to accomplish the aggregation due to the structure of the output. Is Rich Signell still available for IOOS technical consultation in this regard? Same as for SWAN, some users may still want access to the "raw"output.
   http://dm2.caricoos.org/thredds/catalog/content/roms/catalog.html

3. The buoy data are our primary concern since these have been aggregated and tested but are not being ingested into the Catalog or by NGDC.
   http://dm2.caricoos.org/thredds/catalog/buoys/catalog.html

http://dm2.caricoos.org/thredds/UMO_SOS_historical_realtime_agg_caricoos.html

4. The Mesonet and WindNet meteo data have been tested for compliance but have not been aggregated.
   http://dm2.caricoos.org/thredds/catalog/content/Mesonet/catalog.html

http://dm2.caricoos.org/thredds/catalog/content/WindNet/catalog.html

5. The Waverider data files are simply a copy of CDIP's.
   http://dm2.caricoos.org/thredds/catalog/content/Rincon_Waverider/catalog.html

6. The ESA imagery stopped when the satellite went out of service, I believe; some are aggregated.
   http://dm2.caricoos.org/thredds/catalog/content/esa_globcolour/catalog.html

http://dm2.caricoos.org/thredds/esa_globcolour_agg.html

7. We cook the NASA satellite imagery locally, guess they should also be aggregated.
   http://dm2.caricoos.org/thredds/catalog/content/nasa_oceancolor/catalog.html

8. WRF is run locally; need to aggregate.
   http://dm2.caricoos.org/thredds/catalog/content/wrf_archive/catalog.html

Looking back at what I just wrote, getting our buoy data into the Catalog and data set aggregation seem to be the most pressing issues.

But first, could you please check on the duplicate service and ncsos compliance issues to see if these still occur on our new software implementations?

Thanks,

Jorge

[robragsdale]

@jcapella The buoy data services was not successfully harvested last night by the IOOS Registry. The only services that were harvested were related to the waverider, GlobColour daily merged MERIS/MODIS product, and SWAN model. There were no validation errors reported in the harvesting.

@amilan17 do you see any reason why only these three services were harvested but
http://dm2.caricoos.org/thredds/catalog/buoys/catalog.xml and
http://dm2.caricoos.org/thredds/UMO_SOS_historical_realtime_agg_caricoos.xml

were not picked up? Does this have to do with aggregating the files?

Thanks.

[robragsdale]

@jcappella @amilan17 None of the CariCOOS services are reporting being invalid. Only the SWAN model simulation service is being picked up by the catalog and is currently reporting as down. Can we look into why these services are not being harvested? I updated the status of two additional services that are not being harvested into the test WAF to see if they would get picked . @amilan17 is on leave the next 2 weeks, and will not be able to respond to this until she gets back.

[jcapella]

Our thredds was hacked and has been down since 3/26.  José is
working on this and he informs me the server should be back up by
tomorrow.

On 4/1/2015 3:27 PM, robragsdale wrote:


  @jcappella @amilan17
    None of the CariCOOS services are reporting
    being invalid. Only the SWAN
      model simulation service is being picked up by the catalog
    and is currently reporting as down. Can we look into why these
    services are not being harvested? I updated the status of two
      additional services that are not being harvested into the
    test
      WAF to se e if they would get picked . @amilan17 is on leave the next 2
    weeks, and will not be able to respond to this until she gets
    back. 
  —
    Reply to this email directly or view
      it on GitHub.

[amilan17]

@jcapella @robragsdale

There are 15 services submitted in the IOOS registry for CariCOOS
https://www.ngdc.noaa.gov/docucomp/collectionSource/list?recordSetId=3573894&componentId=&serviceType=&serviceStatus=SUBMITTED&serviceUrl=&search=List+Collection+Sources&max=20
and
~3000 records harvested into the test WAF.
http://www.ngdc.noaa.gov/metadata/published/test/NOAA/IOOS/CariCOOS/iso_u/

It looks like to me that only a couple of the services registered were successful and those are too granular (e.g. GlobColour daily merged MERIS/MODIS product...).

Maybe we should schedule a meeting to discuss in more detail?

[robragsdale]

@amilan17 @jcapella Could we schedule time to talk next week?

I believe we should focus on why the buoy data is not being harvested? @jcapella has questions about whether duplication of the services is still an issue as well that I would like to resolve.

Is there a time next week that is good for everyone?

[amilan17]

Which services in the collectionSource list represent buoy data?

[jcapella]

Of course. Just give me a date and time, except for Thursday am.

On 04/14/2015 01:14 PM, robragsdale wrote:

@amilan17 https://github.com/amilan17 @jcapella
https://github.com/jcapella Could we schedule time to talk next week?

I believe we should focus on why the buoy data is not being harvested?
@jcapella https://github.com/jcapella has questions about whether
duplication of the services is still an issue as well that I would
like to resolve.

Is there a time next week that is good for everyone?

—
Reply to this email directly or view it on GitHub
#71 (comment).

[jcapella]

I don't see the non-aggregated data.

The aggregation is

http://dm2.caricoos.org/thredds/UMO_SOS_historical_realtime_agg_caricoos.xml

On 04/14/2015 01:14 PM, robragsdale wrote:

@amilan17 https://github.com/amilan17 @jcapella
https://github.com/jcapella Could we schedule time to talk next week?

I believe we should focus on why the buoy data is not being harvested?
@jcapella https://github.com/jcapella has questions about whether
duplication of the services is still an issue as well that I would
like to resolve.

Is there a time next week that is good for everyone?

—
Reply to this email directly or view it on GitHub
#71 (comment).

[abirger]

@robragsdale , the catalog harvester was down for quite some time, maybe that was the reason? Luke Campbell has the details.