Endpoint for fetching shapefiles of TOS plots

[Nageshbansal]

Is there any endpoint in API from where I can fetch shapefiles that describe monitoring locations for plants, insects, etc. for all of the sites, similar to the files given by the link: https://www.neonscience.org/sites/default/files/All_NEON_TOS_Plots_V9_0.zip ?

[cklunch]

@Nageshbansal Not precisely, but there are a number of ways to get spatial data, both from the API and elsewhere.

The Spatial Data and Maps page on the NEON website has shapefiles and KMZ files with location data for a variety of landscape features and sampling areas.

The locations endpoint of the API enables access to spatial data in the NEON database, based on the names of the locations, which are generally found in data files for observational files and in the sensor_positions metadata files for instrumentation data. This endpoint returns data in json format.

The geoNEON R package provides a couple of wrapper functions for the locations endpoint, and returns data in tabular form. See the package readme file for documentation.

[Nageshbansal]

Thanks, @cklunch for letting me know about this.

I'm trying to get the ALL_NEON_TOS shapefiles. so which methods will be most suitable for this, I tried using the locations endpoint, but I have to query the JSON response for each site for getting the desired results and in the case of the spatial data and maps page I have to download the files manually and I don't want to do that because doing this will be a problem if the data gets updated in the future.

[cklunch]

As far as I know, the website is the only way to get the TOS shapefiles. You can watch the Data Notifications for alerts when new versions are released, but they'd have to be downloaded manually.

Tagging in @rkrauss and @gitbarnitt in case they have other suggestions.

[rkrauss]

Another possible option is through ArcGIS online (AGOL) and the ArcGIS API. There is a link to the NEON landing page on the Spatial Data and Maps webpage Claire mentioned above.

Under content you can search "NEON TOS Plot Centroids" or "NEON TOS Plot Polygons" and there is access to feature layers and shapefiles. I personally don’t have any experience programmatically interacting with AGOL but have seen tutorials for python and java while searching other AGOL tasks.

[MarconiS]

Hello everybody! Joining the discussion to give a little bit of background as to why we want to query the API automatically to download the most updated ALL_NEON_TOS folder.

We are trying to create a dataretriever package to automatically download, stack, geolocate and clean both AOP and TOS data across NEON terrestrial sites. We successfully used the NEONutilities and geoNEON package for the R version of the package (https://github.com/weecology/neonVegWrangleR), but that won't be possible for the python version of the package without making sure things won't break in the future (e.g. using reticulate with certainty that it will be working and maintained in the future).

The three options we have on the table are:

(1) download the latest file and hold it in a misc folder, the drawback being that if plots-suplots-points get moved or added in an updated version, that won't reflect on our geolocation retrieving functions; also, is it OK for you if we host a version of the file and redistribute in a differnet repo?

(2) mimic what is in getLocByName (I believe in L60, https://github.com/NEONScience/NEON-geolocation/blob/a886dded5fb688dfa674416231764aa068f9402d/geoNEON/R/getLocByName.R#L60 is where the spatial data is queried);

(3) create a function that automatically checks whether the latest version of the ALL_NEON_TOS file.

Above all, we would prefer to have (3) implemented, therefore I don't think we can use AGOL or data notifications without any manual workaround. We understand however that it may not be possible. An alternative solution may be proceeding with solution 1 and pushing upstream the latest file any time we get alerted by the Data notification?

@Nageshbansal would that work for you? Also, @rkrauss and @cklunch would you be OK with us hosting a version of the File in a misc folder of our package?

[Nageshbansal]

@MarconiS, Thanks for joining the discussion.
I think (1) will be good for us if it's okay to redistribute the files.
As in (3) we need to do web scraping from NEON Spatial Data and Maps webpages for getting the latest version of ALL_NEON_TOS files and there is a chance that in future this function can break our pipeline.

[cklunch]

@MarconiS @Nageshbansal Thanks for the context! We have no objections to you storing a version of the file and redistributing from there, so option (1) is great from our point of view.

I'd be very interested in hearing more about the packages you're developing. If and when neonVegWrangleR is ready for sharing, please consider submitting it to the Code Hub: https://www.neonscience.org/resources/code-hub

And I'm curious how you're handling the data stacking step in your Python package. We've talked internally about developing a Python wrapper for neonUtilities, but so far haven't had the resources to do it, so if that's something you're working on, I'd love to chat about it.

[MarconiS]

Awesome, thanks!!

As for stacking, right now we are mimicking the functions in the NeonUtilities, but we have been focusing only on the VST and eventually CFC data. We have been talking about making it in a way it can be generalized to the other data-types, though. Happy to chat any time!

[Nageshbansal]

Thanks, @cklunch @rkrauss @MarconiS.
We are using neon-data-API in our package, and it's really helpful, thanks for building and maintaining this API, and we will be happy to add our package to code-hub.
Let me close this issue.