Expert eyes

[ocefpaf]

@emiliom if you have some spare time can you take a look the notebook [1] and make a few comments? Specially regarding to what we should've found but didn't.

[1] https://github.com/ocefpaf/sscsw/blob/master/IOOS_data_demo.ipynb

[emiliom]

@ocefpaf, I'm happy to review it. But this notebook doesn't include any results; oddly, none of the cells are numbered, either. Reviewing a notebook that doesn't show results is kind of tricky, and probably not very useful.

If you could share with me the environment.yml file you used to run this notebook, even better. BTW, I see you're now using Bokeh? Cool. I've never used it myself.

[ocefpaf]

Sorry @emiliom I was going to push a new run but found an error in my workflow while doing that.

Here is the latest (and partily fixed) run: http://nbviewer.ipython.org/urls/gist.githubusercontent.com/ocefpaf/727e00bb5ad27e022034/raw/9dd04d012cc66b7d1469968ae8326c65fa6d417e/IOOS_data_demo.ipynb

[emiliom]

Awesome, thanks! What's the best way for me to comment on this? I wish I could annotate a notebook inline ... Maybe I can import and run the notebook on my end, add comments on each cell, and send it back to you (post it on my gists)? I have several comments and questions already.

One broad comment off the bat: I'm puzzled why no stations from the NANOOS SOS are returned by the CSW query. That doesn't make sense to me, but it's not something I've explored recently (ie, searching via the NGDC CSW). I've noticed that i52North SOS endpoints don't appear much in the system-test examples you've posted on the SIT blog, and that's also been puzzling.

I'm very glad the CMOP SELFE model end point does show up!

So, should I create an env using the IOOS environment.yml, and just add Bokeh? Or is it easier if you share your own env?

[emiliom]

Another general comment: there seems to be quite a bit of functionality in the utilities package that just makes pyoos easier to work with, or even duplicates pyoos functionality. We should push as much of that into pyoos itself as possible, not keep it hidden in utilities (looks like most of those functions are in utilities.secoora. Those duplicate or complementary functions are part of what makes some of these notebooks confusing sometimes, b/c they use pyoos sometimes, and sometimes get around pyoos.

[emiliom]

@ocefpaf, I've been playing around with your notebook (I installed the IOOS env). Lots of interesting findings. But for now I have what's hopefully a simple CSW question: How do I encode a variable name to be added to name_list, if it has a blank space? Specifically, I want to add 'Water Temperature' to name_list, but I suspect that I first need to encode it to replace the blank character somehow. Thanks.

[emiliom]

Ignore that. Sorry to bug.

[emiliom]

Damn you, @ocefpaf 😉. You've made me "waste" a lot of time looking into many of the details going on in that notebook! The good news is that I think I've made much progress:

• I tweaked the NANOOS SOS server, and now it shows up in the CSW query (it took me some effort to understand why it wasn't showing up).
• I've gone through the NDBC pyoos collector to better understand how it works vs. what you're doing in the notebook with all those utilities functions. I'm now able to work directly off of the pyoos functionality to get all the metadata needed (eg, NDBC long name), in a way that I think makes more sense. Plus I can get extra metadata found in the SOS SensorML. I have more work to do to clean it up and enable one more thing.
• I'm hoping something similar can be done for COOPS, but I can't be sure until I try it. Not soon.
• We (Rich, NANOOS' Craig Risien and I) are working on enhancing the OSU ROMS model a bit so that its endpoint is returned by the CSW query.

Stay tuned ... I'll try to share a notebook with you by Wednesday.

[ocefpaf]

Great! And now that I am back to this I see that I got way more that I expected! Thanks @emiliom!!

• I tweaked the NANOOS SOS server, and now it shows up in the CSW query (it took me some effort to understand why it wasn't showing up).

Hopefully is was not the case of the missing s 😉

• I've gone through the NDBC pyoos collector to better understand how it works vs. what you're doing in the notebook with all those utilities functions.

I am keen to kill as much utilities functionalities as possible. They are all remnants from the system-test era (buggy, old, and sometimes broken).

I'm now able to work directly off of the pyoos functionality to get all the metadata needed (eg, NDBC long name), in a way that I think makes more sense. Plus I can get extra metadata found in the SOS SensorML. I have more work to do to clean it up and enable one more thing.

Cool.

• We (Rich, NANOOS' Craig Risien and I) are working on enhancing the OSU ROMS model a bit so that its endpoint is returned by the CSW query.
  Stay tuned ... I'll try to share a notebook with you by Wednesday.

Thanks @emiliom! And sorry for all the trouble!!

PS: Every time we run this kind of exercise we find something to fix, right @rsignell-usgs.

[ocefpaf]

Pinging @jenniferbosch here. Jen, do you know Kathy's GitHub username?

[rsignell-usgs]

Yes, @ocefpaf, it's by running actual tests like this that we discover the problems. I'd like to suggest to the ioos_tech mailing list that DMAC leads try running this notebook, switching the bounding box to their region, and make sure that they can get everything they think they should be getting.

Are we ready for that or would you like to wait a bit more?

[ocefpaf]

@rsignell-usgs Give me 2-3 days. I want to put togehter a simpler notebook with that purpose in mind.

[ocefpaf]

@emiliom I see http://data.nanoos.org/52nsos/sos/kvp?service=SOS&request=GetCapabilities&acceptVersions=1.0.0 now 🎉

[jbosch-noaa]

@ocefpaf - Kathy's Gihub username is @kbailey-noaa.

This notebook looks good. @rsignell-usgs - Thank you for helping Filipe debug this. The more we can use pyoos instead of the utilities functions the better. As I understand it, Filipe is in the process of cleaning utilities functions up.

[ocefpaf]

I wish I could annotate a notebook inline ..

@emiliom For annotations we can try https://via.hypothes.is. Click on the URL below, take a look at the upper right tools, and try to add a comment. I confess I never used it myself, but I do get URL feedback from it a lot.

https://via.hypothes.is/http://nbviewer.ipython.org/urls/gist.githubusercontent.com/ocefpaf/727e00bb5ad27e022034/raw/9dd04d012cc66b7d1469968ae8326c65fa6d417e/IOOS_data_demo.ipynb

If you could share with me the environment.yml file you used to run this notebook, even better. BTW, I see you're now using Bokeh? Cool. I've never used it myself.

Regarding the env, I am not done with folium and gridgeo releases. As soon as I get them uploaded and packaged I will add the environment.yml file here. (Bokeh has a few advantages over mpld3 and vega, but it is not ideal yet. Interactive HTML plots in python are still harder than they should be.)

Another general comment: there seems to be quite a bit of functionality in the utilities package that just makes pyoos easier to work with, or even duplicates pyoos functionality. We should push as much of that into pyoos itself as possible, not keep it hidden in utilities (looks like most of those functions are in utilities.secoora. Those duplicate or complementary functions are part of what makes some of these notebooks confusing sometimes, b/c they use pyoos sometimes, and sometimes get around pyoos.

💯

[ocefpaf]

@emiliom I am thinking about using pyoos.collectors.ioos.swe_sos.IoosSweSos for all the SOS returned in the csw search. Let me know if that is the approach you are seeking there.

[emiliom]

@ocefpaf, the pyoos NDBC collector already uses pyoos.collectors.ioos.swe_sos.IoosSweSos, but only partially. The problem is that the NDBC SOS (and COOPS SOS, unless it's been upgraded) doesn't meet the IOOS SOS standard implemented in IoosSweSos and used with ncSOS and the IOOS 52North SOS. So, IoosSweSos can't fully work on NDBC. Also, no custom "collect" method has been created for pyoos.collectors.ndbc.ndbc_sos.NdbcSos, so to get data (observations) the "raw" method with a 'text/csv' response format is the main option. A "metadata" method is enabled and does return the station SensorML (which I'm able to use/parse). The collect method with a 'text/csv' response format results in an error, so I'm issuing it with responseFormat='text/xml;subtype="om/1.0.0"'.

Unless you're a pyoos pro, much of that won't make much sense to you! I'll share a cleaned-up NDBC pyoos notebook by tomorrow.

[emiliom]

And thanks for pointing me to https://via.hypothes.is ! I didn't know about that system.

[emiliom]

@ocefpaf, I've used via.hypothesis.is to make comments on the notebook.

I'm hoping to share with you today a notebook on pyoos NDBC parsing, using the same OOI Endurance Array region as in your notebook.

[emiliom]

BTW, I will also share with you later (next week) the code or notebook to parse and plot the time series returned by the NANOOS SOS (ie, an IOOS-SOS compliant server) via the pyoos IoosSweSos collector.

[emiliom]

@ocefpaf: It took me a while, but finally here's the pyoos NdbcSos demo I was working on:
http://nbviewer.ipython.org/github/emiliom/stuff/blob/master/pyoos_ndbc_demo.ipynb
I hope you find something useful there!

[ocefpaf]

@ocefpaf, the pyoos NDBC collector already uses pyoos.collectors.ioos.swe_sos.IoosSweSos, but only partially. The problem is that the NDBC SOS (and COOPS SOS, unless it's been upgraded) doesn't meet the IOOS SOS standard implemented in IoosSweSos and used with ncSOS and the IOOS 52North SOS. So, IoosSweSos can't fully work on NDBC.

Got it. I guess pyoos could use some love...

A "metadata" method is enabled and does return the station SensorML (which I'm able to use/parse). The collect method with a 'text/csv' response format results in an error, so I'm issuing it with responseFormat='text/xml;subtype="om/1.0.0"'.

Nice. I was hacking my way into SensorML without pyoos. As you know, pyoos has great docs 😉

Unless you're a pyoos pro, much of that won't make much sense to you! I'll share a cleaned-up NDBC pyoos notebook by tomorrow.

I am not a pyoos pro! That notebook will be very useful.

And thanks for pointing me to https://via.hypothes.is ! I didn't know about that system.

I had a second intention when I showed that to you. To get feedback on the system-test pages 😜

A quick look at your notebook already gave me many ideas to improve this one. Thanks!

@kbailey-noaa I will have something ready for you soon.

[emiliom]

Nice. I was hacking my way into SensorML without pyoos. As you know, pyoos has great docs

Understatement ... BTW, much of the SensorML machinery comes from OWSLib, though it looks like the IoosSweSos collector includes some additional convenience parsing for SensorML for the IOOS SOS conventions.

Regarding my use of via.hypothesis.is, you'll notice that I didn't fully know how to use it when I commented on your notebook. I didn't know how to link notes to specific text. I learned that later.

I started working on a notebook demoeing pyoos with the NANOOS 52N SOS, but I've run into problems. First, I had never used pyoos with a bounding box request on a 52N SOS server. I got an error from pyoos (before it issued the request to the server). When testing directly on my server, I'm running into other issues, including 504 Time Out. Arrgh. I'll need more time to investigate this, so it'll have to wait till next week.

Feliz Natal!

[emiliom]

@ocefpaf, for the NANOOS 52N SOS / pyoos IoosSweSos notebook demo I'm creating, I'm adapting another notebook I created a couple of months ago. That one is too messy to share, but it did work (on a single station). If you're impatient or too curious, here are some old (early 2014) notebooks of mine using pyoos IoosSweSos, on Wakari: pyoos_ioos_sos_demo1, and a much messier but complementary pyoos_swesos_axiom52north_test_2

[ocefpaf]

Arrgh. I'll need more time to investigate this, so it'll have to wait till next week.

I am interested into hearing more about these problems. Maybe we can create a post on system-test about this to help others.

If you're impatient or too curious, here are some old (early 2014) notebooks of mine using pyoos IoosSweSos

Not as messy as you think. Those are very helpful!

Feliz Natal e Bom Ano Novo!

[emiliom]

@ocefpaf: a couple of additional comments about the latest, near-final notebook you mentioned in #2:

• COOPS time series are not handled yet. That makes sense, b/c your code is relying on the SOS end points returned by the CSW query, and COOPS isn't one of them (b/c the temporal extent doesn't match the query). You'd have to add it in a hard-wired way, as before. Assuming the pyoos coops collector is working, that's probably worthwhile.
• The NANOOS SOS is returned by the CSW query, but fails with the pyoos ioosswesos collector. That's a problem on my end, with the SOS server. I've been looking into it, but I probably won't be able to fix it in time, so no worries. Just FYI, your collector2table() function won't work on an IoosSweSos collector, b/c there's no csv response; we can work on that next month, maybe even next week.

[ocefpaf]

collector2table() function won't work on an IoosSweSos collector, b/c there's no csv response; we can work on that next month, maybe even next week.

Yes. I am trying to improve that. (The function is already ugly as is, but...)

[emiliom]

If I manage to get fix the NANOOS SOS server today, I'll contribute code tomorrow to fix collector2table so it can ingest from the IoosSweSos collector.

[rsignell-usgs]

@emiliom , I asked @kwilcox to add this to the Axiom WAF in the meantime, so it should show up at NGDC either tomorrow or next day, I guess.

[emiliom]

@rsignell-usgs, maybe I'm misunderstanding this. I already have a registered NANOOS WAF with other records. Are you saying that if I just extract the OSU ROMS THREDDS iso (from its own, updated ncISO) and dump it on my WAF, that would be enough? Wouldn't that risk creating a duplicate record, though (2 iso records with the same doc ID)?

If doing this as a temporary measure is harmless, I'll do it right now. No need to bug @kwilcox! Thanks.

[rsignell-usgs]

@emiliom, do you crawl a list of thredds catalogs periodically (e.g. at least once per day), extracting the ISO metadata using the ncISO services you find on THREDDS and dropping them into your WAF?

If the answer is yes, then yes, just add that OSU thredds catalog to the list of catalogs you already scan, and we can tell NGDC to stop accessing that data via the OPeNDAP link (and processing with their old standalong ncISO)

[emiliom]

@emiliom, do you crawl a list of thredds catalogs periodically (e.g. at least once per day), extracting the ISO metadata using the ncISO services you find on THREDDS and dropping them into your WAF?

No, I have a WAF with manually placed iso xml's.

Doing what you describe is what I intend to do next month.

[rsignell-usgs]

So for now, I think adding the OSU model to the Axiom WAF is fine. Time range searches will only find one dataset: the one from the Axiom WAF.

[emiliom]

@ocefpaf: The NANOOS SOS is working now! But since I have never issued a bounding-box filtered request with pyoos IoosSweSos, I can't get that to work yet (and don't know for sure that it's implemented correctly). Independent of pyoos, I was able to successfully issue a POST request with a bounding box filter for the OOI endurance array box (and the temporal and variable name filters in your notebook), so that's a good sign.

[jbosch-noaa]

hey folks - This is the main thread where you all have started using my old username. I changed it from @jenniferbosch to @jbosch-noaa at the request of Derrick. This is so we have some consistency in IOOS github presence. :)

Happy New Year!

[ocefpaf]

I changed it from @jenniferbosch to @jbosch-noaa

Thanks for letting us know @jbosch-noaa! (I just invited you to the ioos org BTW.)

[emiliom]

@rsignell-usgs (et al?)

@emiliom , I asked @kwilcox to add this to the Axiom WAF in the meantime, so it should show up at NGDC either tomorrow or next day, I guess.

FYI, the OSU ROMS model record is not showing up yet. I reran @ocefpaf's CSW query, and also looked manually on the NGDC GeoPortal browse page under AOOS and CeNCOOS (the "Axiom WAF"?), but didn't find it.

It won't make it to the OOI notebook, but no worries. We'll address this on the NANOOS end over the next couple of weeks. Thanks.

[rsignell-usgs]

Strange, as it's showing up here:
http://thredds.axiomdatascience.com/iso/nanoos_osu/OCOS.iso.xml

[kwilcox]

The nanoos_osu WAF isn't being harvested unless someone notified the IOOS Catalog person(s) of the new WAF.

[rsignell-usgs]

@kwilcox , crap, I didn't realize that. @robragsdale, I think it's too late now to show up in time for the OOI presentation, but could you make this happen anyway?

[robragsdale]

@rsignell-usgs I am updating the NANOOS WAF.

[emiliom]

@robragsdale, there's some confusion here. The WAF @rsignell-usgs is referring to is apparently not registered. @kwilcox and @rsignell-usgs set it up only as a temporary measure to store a NANOOS model ncISO ISO record, so it'd make it into the NGDC GeoPortal in time for the OOI workshop that starts tomorrow; to overcome a limitation with the older version of ncISO @amilan17 is using.

I've added a manually extracted version of the ISO record in question to the existing, registered NANOOS WAF just now (it's here). If you updated that WAF right after I uploaded the iso file to it, hopefully it got pulled in. FYI, this is a temporary measure, too.

Thanks all! No worries if we can't manage to make it show up by tomorrow. I suspect @kbailey-noaa needs it sooner than tomorrow anyway, b/c she was going to take snapshots from the notebook.

[robragsdale]

Thanks for the clarification @emiliom. The registry harvests overnight, so it will not be discoverable in the NCEI (formerly NGDC) GeoPortal until tomorrow morning.

[rsignell-usgs]

I just tried a search on geoportal, and it's not there yet. 😢

Also, @emiliom or @crisien, I noticed that the ISO metadata for this dataset
http://ona.coas.oregonstate.edu:8080/thredds/iso/NANOOS/OCOS
from the TDS still has an ending date of Dec 25, 2015.

Is this aggregation updating correctly?

[crisien]

Hi Rich,

The end date is now Dec 31, 2015. The reason it ends on Dec 31 is that
Alex and Lana are no longer running this particular model. I started
plotting the new model fields, which extend up into Canada, for NVS late
last month, see
http://nvs.nanoos.org/Explorer?action=overlay:osu_roms_sst. Some time in
the coming weeks I will try to make these fields available via ona's
THREDDS server.

Regards,

-Craig

On 1/5/2016 3:02 AM, Rich Signell wrote:

I just tried a search on geoportal, and it's not there yet. 😢

Also, @emiliom https://github.com/emiliom or @crisien
https://github.com/crisien, I noticed that the ISO metadata for this
dataset
http://ona.coas.oregonstate.edu:8080/thredds/iso/NANOOS/OCOS
from the TDS still has an ending date of Dec 25, 2015.

Is this aggregation updating correctly?

—
Reply to this email directly or view it on GitHub
#3 (comment).

[emiliom]

I just tried a search on geoportal, and it's not there yet.

Ditto, just now. Thanks for testing, @rsignell-usgs

@crisien, thanks for your response on the TDS ending date issue. I hadn't realized that the new model had not made it over to TDS. Darn. Lots to catch up on next week.

[amilan17]

I re-synced geoportal - it should be there now..

Anna
~~~~~~Metadata Adds Meaning~~~~~~
Anna.Milan@noaa.gov, 303-497-5099
NOAA National Centers for Environmental Information
(formerly NOAA’s National Geophysical Data Center)
http://www.ngdc.noaa.gov/metadata/emma
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Tue, Jan 5, 2016 at 10:39 AM, Emilio Mayorga notifications@github.com
wrote:

I just tried a search on geoportal, and it's not there yet.

Ditto, just now. Thanks for testing, @rsignell-usgs
https://github.com/rsignell-usgs

@crisien https://github.com/crisien, thanks for your response on the
TDS ending date issue. I hadn't realized that the new model had not made it
over to TDS. Darn. Lots to catch up on next week.

—
Reply to this email directly or view it on GitHub
#3 (comment).

[emiliom]

FYI, the NANOOS WAF OSU ROMS record has not showed up on geoportal.

Let's give up on providing a temporary access point for this record. However, I do want to make sure there isn't a broader problem with harvesting from that NANOOS WAF; the 4 iso xml's that have been there for over a year are still on geoportal. The only unusual thing about the new record is that it's potentially a duplicate of the primary OSU ROMS record (from THREDDS), at least in terms of the iso document id.

If there is a real, broader problem with the WAF, I can create a new and more specific issue under ioos/registry to follow up. Otherwise I think we're done with this topic on this sprawling issue!