echometrics improvements #12
Comments
|
That was one thing I hadn't quite figured out was how to run things within the test harness. I am familiar with pytest. To produce those results, I was manually running gutils_binary_to_ascii_watch and gutils_ascii_to_netcdf_watch. From email:
The dbdreader has its own quirks. |
|
You can run the existing EcoMetrics tests with I pushed a branch |
|
It seems to be working as is. With tearDown() disabled... The test produces three netCDF files. The last one has the desired information. The first two will need empty variables. Continuing with other tasks... adding information seems straightforward. Do non-standard attributes cause problems? Fiddling with deployment.json and instrument.json a bit: The acoustics has two components with separate serial numbers. If this is ok, I can look at removing the hard coded options. |
|
Moved config options to the instrument since it impacts all the eco* variables. |
|
Think about grouping these so other features can be added later and not get mixed up with other provided keywords. replace: with? |
|
Grouping the |
|
Current tasks:
Interim testing:
|
|
@jr3cermak I played around with hosting the datasets as-in (with the
|
|
I also experimented with storing the pseudogram with the time dimension. Since the pseudogram time coordinates are different from the CTD profile, the resultant netCDF files became very large. So, I would say writing the pseudogram data out to a separate file sounds like the best option at the moment. |
|
😒 Here is an ERDDAP Dataset that just serves the pseudogram data. I'm playing with some ideas to get this into AOOS, stay tuned. |
|
I am just rounding the corner where I can almost get the latest glider deployment loaded under ERDDAP. I can see it is complaining about something. This is the combined case. It does not seem happy at all with the That test looks suspicious... Onto the separated case... |
|
Resync branch after PR #18 and carry on. |
|
Resync with master to take a look at the new pathway. |
|
Running the latest deployment through the current code shows a single netcdf file now. Are the profiles combined? This is quite different than what was shown in an earlier email with the tabledap link: https://gliders.ioos.us/erddap/tabledap/extras_test-20220329T0000.htmlTable?trajectory%2Cwmo_id%2Cprofile_id%2Ctime%2Clatitude%2Clongitude%2Cdepth%2Cpseudogram_depth%2Cpseudogram_sv%2Cpseudogram_time%2Csci_echodroid_aggindex%2Csci_echodroid_ctrmass%2Csci_echodroid_eqarea%2Csci_echodroid_inertia%2Csci_echodroid_propocc%2Csci_echodroid_sa%2Csci_echodroid_sv&time%3E=2021-12-02T00%3A00%3A00Z&time%3C=2021-12-09T17%3A33%3A35Z that references: https://gliders.ioos.us/erddap/files/extras_test-20220329T0000/ On the DAC for unit_507, there are two separate sets of files It looks like the pseudogram is folded back into the profiles as a single file now. |
|
The latest master of GUTILS is great for backend storage of echodroid/pseudogram data. Tossing the Python code to pull from the aggregation just for reference. #$ cat plotGretelDepl2.py
import io, os, sys, struct, datetime
import subprocess
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from matplotlib.figure import Figure
from matplotlib.colors import LinearSegmentedColormap, Colormap
import matplotlib.dates as dates
import matplotlib.ticker as mticker
from matplotlib.patches import Rectangle
import json
import xarray as xr
#get_ipython().run_line_magic('matplotlib', 'inline')
def newFigure(figsize = (10,8), dpi = 100):
fig = Figure(figsize=figsize, dpi=dpi)
return fig
# Fetch Sv data
def fetchSv(start_time, end_time, ds):
# Copy data into a numpy array and resort Sv(dB) values for plotting
# Convert TS to string
# datetime.datetime.strftime(datetime.datetime.utcfromtimestamp(dt), "%Y-%m-%d %H:%M:%S.%f")
# Convert string to TS
# datetime.datetime.strptime(dtSTR, "%Y-%m-%d %H:%M:%S.%f").timestamp()
time_dim = 'time'
sv_ts = np.unique(ds[time_dim])
startDTTM = start_time
#startVal = datetime.datetime.strptime(startDTTM, "%Y-%m-%d %H:%M:%S.%f").timestamp()
startVal = np.datetime64(datetime.datetime.strptime(startDTTM, "%Y-%m-%d %H:%M:%S.%f"))
endDTTM = end_time
#endVal = datetime.datetime.strptime(endDTTM, "%Y-%m-%d %H:%M:%S.%f").timestamp()
endVal = np.datetime64(datetime.datetime.strptime(endDTTM, "%Y-%m-%d %H:%M:%S.%f"))
# This obtains time indicies for the unique time values
a = np.abs(sv_ts-startVal).argmin()
b = np.abs(sv_ts-endVal).argmin()
#print(a,b)
#print(time_array.shape)
#print(list(sv.variables))
# https://xarray.pydata.org/en/v0.11.0/time-series.html
sv_data = ds['pseudogram_sv'].sel(time=slice(pd.Timestamp(sv_ts[a]),pd.Timestamp(sv_ts[b])))
sv_time = [pd.Timestamp(t.values).timestamp() for t in ds[time_dim].sel(time=slice(pd.Timestamp(sv_ts[a]),pd.Timestamp(sv_ts[b])))]
sv_depth = ds['depth'].sel(time=slice(pd.Timestamp(sv_ts[a]),pd.Timestamp(sv_ts[b])))
return (sv_time, sv_depth, sv_data)
# Make plots from intermediate deployment data
def makePlot(sv_time, sv_depth, sv_data):
# Set the default SIMRAD EK500 color table plus grey for NoData.
simrad_color_table = [(1, 1, 1),
(0.6235, 0.6235, 0.6235),
(0.3725, 0.3725, 0.3725),
(0, 0, 1),
(0, 0, 0.5),
(0, 0.7490, 0),
(0, 0.5, 0),
(1, 1, 0),
(1, 0.5, 0),
(1, 0, 0.7490),
(1, 0, 0),
(0.6509, 0.3255, 0.2353),
(0.4705, 0.2353, 0.1568)]
simrad_cmap = (LinearSegmentedColormap.from_list
('Simrad', simrad_color_table))
simrad_cmap.set_bad(color='lightgrey')
# Convert sv_time to something useful
svData = np.column_stack((sv_time, sv_depth, sv_data))
# Filter out the noisy -5.0 and -10.0 data
svData = np.where(svData == -5.0, -60.0, svData)
svData = np.where(svData == -15.0, -60.0, svData)
# Sort Sv(dB) from lowest to highest so higher values are plotted last
svData = svData[np.argsort(svData[:,2])]
# Plot simply x, y, z data (time, depth, dB)
#fig, ax = plt.subplots(figsize=(10,8))
fig = newFigure()
ax = fig.subplots()
#ax.xaxis.set_minor_locator(dates.MinuteLocator(interval=10)) # every 10 minutes
#ax.xaxis.set_minor_locator(dates.HourLocator(interval=3)) # every 3 hours
#ax.xaxis.set_minor_formatter(dates.DateFormatter('%H')) # hours
#ax.xaxis.set_minor_formatter(dates.DateFormatter('%H:%M')) # hours and minutes
ax.xaxis.set_major_locator(dates.DayLocator(interval=2)) # every day
#ax.xaxis.set_major_formatter(dates.DateFormatter('\n%m-%d-%Y'))
ax.xaxis.set_major_formatter(dates.DateFormatter('%m/%d'))
ax.tick_params(which='major', labelrotation=45)
#ax.set_facecolor('lightgray')
ax.set_facecolor('white')
dateData = [datetime.datetime.fromtimestamp(ts) for ts in svData[:,0]]
#im = plt.scatter(dateData, svData[:,1], c=svData[:,2], cmap=simrad_cmap, s=30.0)
im = ax.scatter(dateData, svData[:,1], c=svData[:,2], cmap=simrad_cmap, s=30.0)
#cbar = plt.colorbar(orientation='vertical', label='Sv (dB)', shrink=0.40)
fig.colorbar(im, orientation='vertical', label='Sv (dB)', shrink=0.40)
#plt.ylim(0, sv_depth.max())
#plt.gca().invert_yaxis()
#plt.ylabel('Depth (m)')
#plt.xlabel('Date (UTC)')
ax.set(ylim=[0, sv_depth.max()], xlabel='Date (UTC)', ylabel='Depth (m)')
#plt.clim(0, -55)
im.set_clim(0, -55)
# Invert axis after limits are set
im.axes.invert_yaxis()
#plt.title("Acoustic Scattering Volume (dB) Pseudogram")
ax.set_title("Acoustic Scattering Volume (dB) Pseudogram")
return fig, ax
ds = xr.open_dataset('http://mom6node0:8080/thredds/dodsC/GretelExtra.nc')
# Find the timespan of the dataset
ts_min = ds['time'].min()
ts_max = ds['time'].max()
# use the entire deployment
start_dt_string = str(ts_min.dt.strftime("%Y-%m-%d %H:%M:%S.%f").values)
end_dt_string = str(ts_max.dt.strftime("%Y-%m-%d %H:%M:%S.%f").values)
(sv_time, sv_depth, sv_data) = fetchSv(start_dt_string, end_dt_string, ds)
if len(sv_data) > 100:
(fig, ax) = makePlot(sv_time, sv_depth, sv_data)
imageOut = "Sv_%s_all.png" % (str(ts_min.dt.strftime("%Y%m%d").values))
fig.savefig(imageOut, bbox_inches='tight', dpi=100)
ds.close() |
|
Nice, an added benefit I didn't even think about! |
|
Cycling back around to provide an update to support future deployments. Will resync with master and move forward. Please let me know what things you need to support of echometrics, low resolution / echograms (formerly pseudograms). This update will provide:
Because of the two stage processing of GUTILS, to provide a data frame, the 2nd pass script would have to provide the DBD files and the cache file directory to decode and provide a direct data frame object. Otherwise, continue to use the 1st pass and produce the csv file and then read the csv file in the 2nd pass to recover the data frame (kinda of what happens now). The intermediate output file can be anything -- a pickled object with the data frame needed in the 2nd stage, etc. We need to know what target(s) to hit for you so we can get them built into the CI testing. Once it all passes again, move ahead with other fun things. It looks like python 3.7 is EOL. Is there a particular version of python we should use? We are at the stage of reworking the tests and updating code. I am anticipating at least two to four weeks of additional effort on our side before a reasonable PR is ready. This could change based on the requirements/targets provided. |
|
Main branch readme => python 3.9 :) |
|
Unfortunately, our work has snowballed a bit. So, we will need to submit at least three PRs in total as of this writing. The first is ready to go when CI tests pass.
|
|
Latest checks have passed. I have refreshed documentation in the README.pdf and have it out on a website (that may be down at some point for an OS update). https://nasfish.fish.washington.edu/echotools/docs/html/echotools/html/echotools/README.html The important bit is walking from the produced netCDF files (*_extra.nc) to a time series plot of the echogram profiles given any time range. So, I think that is the target product that is desired on the data portal. https://nasfish.fish.washington.edu/echotools/docs/html/echotools/html/echotools/README.html#product That should give us the pivot point to start heading down the |
|
Just a little more work on some additional "profile" products for echometrics. We stood up a prototype that will be used internally once implemented in some fashion on the data portal. https://nasfish.fish.washington.edu/echotools/dppp/egramBrowser/portal.html |
|
Ready for me to take a look? |
|
There is at least one more pending update with additional "profile" products to be sent. I will post another note when things settled. |
|
You can move ahead with the current code in the PR. This other new part needs some more R&D before it can be implemented. I originally thought it was going to be an easy drop in addition. That is not the case. |
|
A proposal for additional CF standard names has been submitted to improve standards compliance for proposed acoustic datasets. For future use in deployment.json and other configuration files. |
Continued from PR #10
test_slocum.pyto exercise echometrics/pseudogram code to produce desired netCDF resultsDeferred:
The text was updated successfully, but these errors were encountered: