Update Timeseries Files

[2320sharon]

Currently the application creates a CSV file for each transect that was present in the session when shorelines were extracted. When there are hundreds of transects presents this creates a lot of unnecessary files. In our meeting today we discussed that would be better to create a single file with all the transects present in it. If the user wants to create a CSV file for every single transect then they can do that using a utility script instead. This script will be found SDS tools or the scripts folder in coastseg.

We also discussed changing the file format for the tidally corrected version of the time series. We think would be better if it mirrored the format that the original raw Time series was in AKA the dates as rows, The Columns being the transect ID, and the content of each cell being the cross distance of the shoreline point across the transect. We would create a separate file that would capture the tide predictions for each transect ID. This would involve rewriting a bit of the code and updating the merge sessions functions, but would ultimately make the process of tidal correction much easier.

[dbuscombe-usgs]

To reorder the data, use a pivot table

transect_time_series_tidally_corrected_file = 'transect_time_series_tidally_corrected.csv'

merged_transect_time_series_tidally_corrected = pd.read_csv(transect_time_series_tidally_corrected_file)

df_distances_by_time_and_transect = merged_transect_time_series_tidally_corrected.pivot(index='dates',columns='transect_id', values='cross_distance')

to reorder by tide, use

df_tides_by_time_and_transect = merged_transect_time_series_tidally_corrected.pivot(index='dates',columns='transect_id', values='tide')

Creates matrices of M date rows by N transect columns

Can then be visualized in one line
plt.imshow(df_distances_by_time_and_transect); plt.show

[2320sharon]

This is awesome! Thank you for contributing the codes here! I'll try to incorporate the changes coming soon.

[2320sharon]

Here is a relevant update from our discussion on Slack:
"""
Ideally, I would like there to be the following files made by both workflows

1. merged raw timeseries (columns: time, x, y, transect, shoreline)
2. merged tidally corrected timeseries (columns: time, x, y, transect, shoreline, tide)
3. raw shorelines as matrix of time versus transects (pivot table idea on GH)
4. tidally corrected shorelines as matrix of time versus transects
5. tidal heights as matrix of time versus transects

I think the individual csv files for transects can be scrapped.

Obviously, 2, 4 and 5 would only get made if tidal correction was applied

"""

Tasks

• individual csv files for transects can be scrapped
• make merged raw timeseries (columns: time, x, y, transect, shoreline)
• tidally corrected shorelines as matrix of time versus transects (same format as point 3)
• modify the tide_predicitions.csv to match the format tidal heights as matrix of time versus transects

[2320sharon]

In today's development meeting we decided it would be good to add the satellite the shoreline was derived from to the merged raw timeseries and merged tidally corrected shorelines if possible.

Tasks

• Add satellite to merged raw timeseries (columns: time, x, y, transect, shoreline, satellite)
• Add satellite to merged tidally corrected timeseries (columns: time, x, y, transect, shoreline, satellite)

[2320sharon]

We also need a script to create CSV files for each transect in case the user wants this data because this feature will be removed from coastseg.

• Create script to generate CSV per transect

[2320sharon]

I'm working on this issue and I realized something. If we add the satellite to the raw timeseries outputs and the tide corrected outputs then this will make them incompatible with any coastsat related post processing. Basically a user couldn't use their timeseries.csv with any of the existing CoastSat codes that use the timeseries. However if we are okay with this change I can modify the rest of the CoastSeg workflows and coastsat_package to handle this change.

@dbuscombe-usgs we might want to re-visit this idea later to save some time.

[2320sharon]

This feature has been implemented since 1.1.34 so I'm going to close it.
The satellites are not listed in the csv files but if we decide to add them at a later time we can re-open the issue then.