include web Mercator projection (EPSG:3857) in ERDDAP WMS for using easy leaflet maps with nice basemaps #24
Comments
|
Yes. "Add support for Web Mercator projection" is already on the To Do list. Because of the nature of the graphics library that ERDDAP uses to generate maps (which was selected long ago, before most of the current libraries existed), it is not possible to simply add a new projection. So it is a more difficult project. Similarly, it would/will not be easy to switch to another graphing library. The To Do list is long. I just haven't done this item yet. I'm sorry. |
|
Ok, glad to hear Web Mercator projection for slippy maps is on your TODO. Too bad it's not an easy fix. I'm not a Java programmer, but please let me know if I can help. Seems like a common approach for serving WMS image and other OGC services (WCS raster, WFS vector) is to setup a GeoServer alongside ERDDAP for the same data sources. For instance, I see this with these projects: This might not be the best place to ask, but do you have any pointers on how to best to link ERDDAP and GeoServer? Ideally, you could discover one through the other (eg metadata mention of alternate providers through catalog search), but not sure where such information would live other than highest level of a website. CC'ing Axiom folks who work setup docker-erddap and docker-geoserver in case they might have any tips: @kwilcox, @shane-axiom, @rsignell-usgs |
|
@bbest I'm going to guess the dataset you are trying to access through ERDDAP's WMS is netCDF based, and the short answer is that netCDF and GeoServer don't play nicely together. There was some movement towards a GeoServer netCDF backend at some point.. maybe 8 years ago... but I believe progress stalled and was never completed. This could use a little bit of research (let me know if you find anything recent). If you are looking for WMS from a netCDF data source in a projection other than 4326, you'll have to look towards: Can you link me to the ERDDAP dataset you are trying to plot? I'll poke at it to see if it moves. |
|
IIRC when I looked at the GeoServer netCDF reader a few years ago we decided against using it because it didn't handle a lot of real world complexities (e.g. anti-meridian wrapping) and didn't include support for community conventions like CF because it didn't make use of Unidata's netcdf-java library. That being said, if your use case isn't complex you might take a look and see if it works for you. |
|
Thank you very much @kwilcox and @srstclair for weighing in on this given all your experience. I'm generally interested in quickly visualizing remotely sensed variables like SST and chlorophyll globally (ie WMS), as well as summarizing these data over time by arbitrary polygons (ie ERDDAP slicing/fetching). For example from the addWMSTiles(
baseUrl = 'https://coastwatch.pfeg.noaa.gov/erddap/wms/jplMURSST41mday/request?',
layers = "jplMURSST41mday:sst",
options = WMSTileOptions(
version = "1.3.0", format = "image/png", transparent = T, opacity = 0.7,
time = "2018-07-16T00:00:00Z"))The ncWMS and sci-wms look great, particularly for slicing by time and variable. The NetCDF — GeoServer 2.15.x User Manual suggests good support for COARDS compliance (custom, Time, Elevation, Lat, Lon). I'm also working with MBON and interested in being able to programmatically search the catalog of datasets in the portal at https://mbon.ioos.us developed by Axiom, but not sure how to do that, even with the excellent documentation. Do you know the base ERDDAP URL? I'm able to search through the GeoServer OGC GetCapabilities endpoints, but that's just a short name and not the other metadata, hence interest in how datasets could be cross-tagged across ERDDAP and GeoServer. |
|
@kwilcox @srstclair @bbest I dug up some old links on GeoServer and netCDF support I had been forwarded previously. NOAA/NWS had funded recently (< 8 yrs ago) some work to allow GeoServer to support a particular model dataset (Rapid Refresh: RAP) in GRIB2 . Info here: https://github.com/bencaradocdavies/geoserver/wiki/RAP-Native-Grid and Including use of netCDF-Java to read the source data to feed to GeoTools and in turn GeoServer (see the first link). However, I'm not sure whether they were merged into the primary GeoServer product or if it's only a custom build as outlined in the wiki link. It looks like part of the project involved adding support for the specific RAP grid to netCDF-Java, so clearly netCDF-Java is part of the equation for at least this fork. There's a lot of configuration involved in order to make GeoServer read netCDF (or GRIB2 in this case) via GeoTools ImageMosaic, so that's still a limitation/inconvenience, but maybe even so there's a path forward to make ERDDAP and GeoServer play well together. I would be interested to see that. It could go a ways towards adding some of the functionalities you're looking for to ERDDAP (or maybe alongside ERDDAP). Maybe the magic of GitHub mentions will help, if @bencaradocdavies can share if this has gone anywhere since he worked on it. |
|
@mwengren @kwilcox @srstclair @bbest the current GeoServer NetCDF ("netcdf") plugin is built on the Unidata NetCDF-Java library and is well supported. I added a NetCDF example in the GeoServer Quickstart in the OSGeoLive image. The NetCDF Output ("netcdf-out") plugin is used for WCS delivery of NetCDF 3 and 4. I do not think this what you want. The RAP Native work I did in 2016-2017 added projection support and netcdf-out improvements. It was all merged into GeoServer 2.9. The use-case in the links above is the most complicated, and involves a mosaic that concatenates multiple GRIB2 files for delivery as NetCDF via WCS. GeoServer configuration to deliver WMS from NetCDF is much, much simpler. |
|
Connecting ERDDAP and GeoServer has been on my list of things to To Do (or at least explore) for years. If any of you want to pursue it, please do. If, as it sounds, there is a netcdf-java to GeoServer connection, then it seems likely that there would be a way to take all the information that an ERDDAP has about its datasets and automatically make corresponding datasets in GeoServer (at least for the gridded datasets). I suspect that the current netcdf-java to GeoServer connection assumes that the data is gridded geographic data. It would also be interesting to see if the tabular datasets in ERDDAP which have a CF DSG/cdm_data_type/featureType could also be made into appropriate datasets in GeoServer and hence available via GeoServer's WFS. If anyone wants to work on this and wants to bounce ideas off me, please email me. |
|
@BobSimons I suspect that you are right: as far as I know, GeoServer supports gridded NetCDF data sources ("coverages") but not NetCDF vector data sources (equivalent to ERDDAP tabular?). The GeoServer devel list would be a great place to start. I am no longer active in the GeoServer project but there are several active developers supporting NetCDF data sources. The NetCDF libraries are there; support would require a GeoTools DataStore implementation and a GeoServer plugin to package it and possibly add a web UI for configuration. A prototype DataStore could even be a GSoC project. |
|
@bencaradocdavies thanks for the update! That would be great if an ERDDAP GeoTools DataStore implementation could be built, at least for gridded datasets. I like the idea of a GSoC project as well. @BobSimons if you wanted to suggest that, you could go through the ESIP GSoC repository to suggest it, see: https://github.com/ESIPFed/gsoc/issues. I don't know if they are still accepting new project ideas or not, but they may still be! Someone would have to provide guidance from the GeoTools standpoint too, since it would obviously be developed in GeoTools. OSGeo also participates in GSoC. Maybe someone in the GeoTools project could be looped in later. What do you think? On the projection question, I am not sure if gridded geographic data is a requirement or not, it may be that as long as the projection can be specified within GeoTools/GeoServer, any projection may work (maybe someone knows better about this than I do). GeoServer can reproject dynamically, of course at a performance cost. |
|
@mwengren the geotools-devel mailing list should be the first stop for anyone contemplating a new DataStore implementation because you can find other interested developers and mentors. The GeoTools Developer Guide has more. It is straightforward to get permission to create a new unsupported module and the commit access to maintain it. Contributors need to submit signed code contribution agreements; this can be the slowest step for employees of bureaucracies! |
|
Thank you all for weighing in on this issue around connecting GeoServer and ERDDAP. Perhaps I don't understand the suggestions for more direct integration between GeoServer and ERDDAP, but having a separate data catalog that describes the various representations and web services for a given dataset seems the way to go to me. The logical choice for data catalog software is CKAN, which is in keeping and replicable with data.ioos.us (see github.com/ioos/catalog) and data.gov (see github.com/GSA/data.gov). This way the underlying dataset storage could be much more flexible than any one suite of software and be transformed for optimization of services. For instance, netcdf could be served through ERDDAP, but also transformed into a stack of TIFFs, which may be more efficient for serving through GeoServer (as suggested by this tutorial Using the ImageMosaic plugin for raster time-series data — GeoServer 2.16.x User Manual). Decoupling the data catalog from the underlying data representations and web services enables many more possibilities, while ensuring queryable access through the CKAN API with Python and R clients. I'm curious what you all think about this approach which seems to already be in place? Overviewdatasets | data.ioos.usNote how ERDDAP and other web services using GeoServer (ie WMS, WCS) are both visible and queryable for dataset search and browse:
dataset detail | data.ioos.usNote how all the links are available through the catalog to a variety of representations served by different endpoints, ie ERDDAP, GeoServer or other...
|
|
The system you propose may make sense for various reasons, but not for the
immediate need here and not for the other uses I had hoped for, which is to
allow ERDDAP to use the WMS/mapping/reprojection features of GeoServer to
make Web Mercator maps (and do lots of other things). For these needs and
for ease of setup, I think it would be great if an administrator could
install ERDDAP and GeoServer, and then have all of the datasets in ERDDAP
automatically also show up as datasets in GeoServer. ERDDAP has aggregated
datasets (not just a bunch of .nc files), with metadata about each dataset
(enhanced over what is in the .nc files) and it would be nice if that
information could be used to automate the connection to GeoServer. Then,
ERDDAP could know that there are corresponding datasets in GeoServer and
thus use/rely on those services. With your architecture, those things are
not true and it would be more work for the administrator (setting up
datasets separately in ERDDAP, GeoServer, and CKAN), and working to make
the metadata the same in ERDDAP, GeoServer and CKAN..
But that's just my view for my/ERDDAP's purposes. Anyone can set up any
architecture they want.
…On Mon, Mar 18, 2019 at 11:55 AM Ben Best ***@***.***> wrote:
Thank you all for weighing in on this issue around connecting GeoServer
and ERDDAP. Perhaps I don't understand the suggestions for more direct
integration between GeoServer and ERDDAP, but having a separate data
catalog that describes the various representations and web services for a
given dataset seems the way to go to me.
The logical choice for data catalog software is CKAN <https://ckan.org/>,
which is in keeping and replicable with data.ioos.us (see
github.com/ioos/catalog) and data.gov (see github.com/GSA/data.gov).
This way the underlying dataset storage could be much more flexible than
any one suite of software and be transformed for optimization of services.
For instance, netcdf could be served through ERDDAP, but also transformed
into a stack of TIFFs, which may be more efficient for serving through
GeoServer (as suggested by this tutorial Using the ImageMosaic plugin for
raster time-series data — GeoServer 2.16.x User Manual
<https://docs.geoserver.org/latest/en/user/tutorials/imagemosaic_timeseries/imagemosaic_timeseries.html>
).
Decoupling the data catalog from the underlying data representations and
web services enables many more possibilities, while ensuring queryable
access through the CKAN API <https://docs.ckan.org/en/2.8/api/> with
Python <https://github.com/ckan/ckanapi> and R
<https://github.com/ropensci/ckanr> clients.
I'm curious what you all think about this approach which seems to already
be in place?
Overview
netcdf → ERDDAP → various formats (csv, png, ...) ↘
↓ → CKAN
tiff → GeoServer → OGC services (WMS, WFS, WCS,...) ↗
datasets | data.ioos.us
Note how ERDDAP and other web services using GeoServer (ie WMS, WCS) are
both visible and queryable for dataset search and browse:
[image: image]
<https://user-images.githubusercontent.com/2837257/54555122-4fc6b500-4973-11e9-93c7-039c82d6de4d.png>
dataset detail | data.ioos.us
Note how all the links are available through the catalog to a variety of
representations served by different endpoints, ie ERDDAP, GeoServer or
other...
[image: image]
<https://user-images.githubusercontent.com/2837257/54555165-6705a280-4973-11e9-9abd-ac3721964e3c.png>
...
[image: image]
<https://user-images.githubusercontent.com/2837257/54555195-72f16480-4973-11e9-913b-cb1fcbb4fbf9.png>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABarOIMmfDz-jDWN3ZsgPohMFDlz5-ksks5vX-EvgaJpZM4apFpZ>
.
--
Sincerely,
Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
99 Pacific St., Suite 255A
Monterey, CA 93940
Phone: (831)333-9878
Fax: (831)648-8440
Email: bob.simons@noaa.gov
The opinions in this message are mine personally and do
not necessarily reflect any position of the U.S. Government
or the National Oceanic and Atmospheric Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><
|
|
Ah thank you @BobSimons for this extra insight. I will be developing a data management portal for IEA ecosystem status indicators, so will become more familiarized with ERDDAP configuration. In the process I can chip away at sync'ing datasets between ERDDAP and GeoServer initially with some simple scripts and keep these considerations in mind for trying to generalize more broadly for a tighter integration. Everybody's input here has been much appreciated 🙏 |
|
Apologies for hijacking this thread with a tangent, but: are ERDDAP, CKAN, THREDDS, (others?) all providing a similar "middleman" functionality? I have minimal exposure to CKAN and THREDDS and want to better understand the niche of each of these relative to ERDDAP. |
|
@BobSimons @bbest I think both pursuits would be worthwhile. I think @BobSimons is considering general purpose ERDDAP use, and I totally agree about the value of a GeoServer 'extension' point for ERDDAP (if that's an appropriate phrase). I would love to see that happen. @bbest it sounds like your goal is to develop a purpose-built application on both of those tools, as well as CKAN for the data catalog - also a good idea, in my book! Maybe they should be separated into distinct issues though to avoid confusion. @bbest on the architecture you showed, I would add that there is some - possibly manual - metadata configuration that happens in the middle in order for the CKAN representation to look the way it does. PacIOOS does a phenomenal job maintaining their metadata WAFs, but I don't know how much of it is a fully automated process. Either way, there would be some additional software to stitch together the TIFF-based services emanating from GeoServer and those from ERDDAP. Nonetheless, no reason it wouldn't work! |
|
I view them as significantly different. Each has its own niche that makes
each better for certain purposes.
THREDDS is a DAP+WMS+WCS data server (just for gridded data) that just
provides 1 catalog view: a hierarchy of terms, eventually terminating in
either data files or aggregated datasets at the leaves. THREDDS has its own
metadata system which is separate from the dataset's metadata. So if you
specify, e.g., a new summary, it appears in the THREDDS catalog for the
dataset, not in the dataset's .das metadata (although there may be a way to
do that separately). The THREDDS catalog has no system for full-text
searches, which is why e.g., NOAA groups and others uses/used CKAN,
Lucene/Solr, or ESRI GeoPortal for catalog services. For brokering, THREDDS
can only point to the URLS of datasets in other THREDDS (and simplistically
to other URLs), but since there are no catalog services that is no help for
searching.
CKAN is a more general purpose catalog system, not a data server (in any
significant way). As such, it is useful in that it can point to all kinds
of data services in all kinds of remote data servers. But since the data
sources are diverse and because CKAN makes no effort to broker those data
services, there is no consistent way to access data from the different
datasets.
ERDDAP is a dataserver (for gridded datasets (slightly more limited than
THREDDS) and tabular datasets) with catalog services (full text search,
faceted search, view all, and advanced search) for its datasets. ERDDAP
emphasizes brokering: ERDDAP seeks to be able to connect to data from lots
of remote data services (e.g., THREDDS, GRADS, DAPPER, relational
databases, Cassandras, AWS S3 buckets, WFS servers, other ERDDAPs, ...) and
present those datasets in exactly the same way that it presents local
datasets (e.g., .nc files, ,csv files, and lots of specialized file types).
I hope that is reasonably fair. Some of that is debatable. I hope that
helps.
---
Let me separately say: one of the goals of ERDDAP is to make aggregated
datasets (not treat a bunch of files as just a bunch of files). Then,
another goal of ERDDAP is to encourage better metadata for each of those
datasets. (I don't think THREDDS or CKAN view it the same way.) Part of the
problem is that the community's idea of "good metadata" changes over time
(e.g,. new versions of CF, ACDD, and ISO 19115) so ERDDAP helps by making
it easy to modify each dataset's metadata via the ERDDAP datasets.xml setup
for each dataset, without editing/changing each data file (which e.g., NCEI
doesn't want to do because they want to preserve the original data
submission) or trying to get a remote data source to change the metadata.
Then, if each dataset is aggregated and has extensive, good metadata, then:
1) Search in ERDDAP works better because there is more and better metadata
available to search.
2) You can automatically create ISO 19115 metadata from the dataset's
metadata (as ERDDAP does).
That's an important point, because then you don't have to separately
maintain the dataset's local metadata and the dataset's ISO 19115 metadata.
3) You can automatically populate other catalog systems directly or via the
ISO 19115 documents.
4) You should be able to make a system to automatically connect to other
data servers, e.g., GeoServer.
…On Mon, Mar 18, 2019 at 1:03 PM Tylar ***@***.***> wrote:
Apologies for hijacking this thread with a tangent, but: are ERDDAP, CKAN,
THREDDS, (others?) all providing a similar "middleman" functionality?
I have minimal exposure to CKAN and THREDDS and want to better understand
the niche of each of these relative to ERDDAP.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABarOE5YlV8CKMnrbchPb_-bkixLCFloks5vX_EGgaJpZM4apFpZ>
.
--
Sincerely,
Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
99 Pacific St., Suite 255A
Monterey, CA 93940
Phone: (831)333-9878
Fax: (831)648-8440
Email: bob.simons@noaa.gov
The opinions in this message are mine personally and do
not necessarily reflect any position of the U.S. Government
or the National Oceanic and Atmospheric Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><
|
|
Many thanks for the detailed response. Following this discussion has been a great help for me to better understand the relationships between these softwares. |
|
Hi! and for the North Polar stereographic - EPSG:5041 (from ncWMS GODIVA client): and for Antarctic Polas Stereographic - EPSG:3031 (from a web page using LeafletJS): Hope this ca nbe useful. Ciao Marco |
|
Awesome, thanks so much for sharing @MarcoAlbaETT! Look forward to giving ncWMS a spin |
CC: @ehazen, @7yl4r,
Hi @BobSimons,
Thank you, thank you for a great and useful product of ERDDAP focused on the many ways to slice, dice and represent data for a plethora of analysis and visualization.
One of the more useful and easy representations of data is as a Web Map Service (WMS) to easily drop into a "slippy" online map using JavaScript libraries like leaflet and all the pretty basemap's like Esri.OceanBasemap and others from Leaflet Provider Demo:
However, the ERDDAP WMS currently only serves geographic projection EPSG:4326 and not the web Mercator EPSG:3857 needed to use these great basemaps. Heres' some futzing I did trying various combinations to make this work:
WMS Testing for Fixing Explorer Environmental Map
Could this be a matter of simply adding or updating instances in these Java files where 4326 shows up?
EDDTableFromMultidimNcFiles.java
EDDTableFromNcCFFiles.java
Thanks very much, Ben
The text was updated successfully, but these errors were encountered: