Add historical data to l3 #252

BaptisteVandecrux · 2024-06-04T11:16:51Z

This PR builds on top of #251. It:

reads merged (raw+tx for v2+v3 stations) aws-l2 data
runs toL3 on that L2 file
loads and postprocesses the historical GC-Net data
appends the historical GC-Net data before the freshly generated L3 data
makes hourly, daily, monthly resampling and fills metdata

…laciology-and-Climate/pypromice into add-historical-data-to-l3

src/pypromice/process/L0toL1.py

ladsmund

I haven't looked at join_3 yet.

src/pypromice/process/write.py

src/pypromice/process/get_l2tol3.py

src/pypromice/process/get_l2.py

src/pypromice/process/resample.py

src/pypromice/process/utilities.py

src/pypromice/ressources/variables.csv

src/pypromice/process/L1toL2.py

src/pypromice/process/utilities.py

src/pypromice/process/get_l2tol3.py

src/pypromice/process/join_l3.py

…gging, attributes and tests,

…258) * numpy dependency <2.0 * resample rules updated (deprecation warning) * fillna replaced with ffill (deprecation warning) * get_l3 called directly rather than from file * process action restructured

renamed new variable.csv moved old variable.csv renamed new variables.csv recreate variables.csv

src/pypromice/process/join_l3.py

src/pypromice/ressources/variable_aliases_GC-Net.csv

setup.py

src/pypromice/process/join_l3.py

…te.py

…etcdf

instead of just reading the stid to join

PennyHow

I've looked through it and generally I think it looks fine - there will be edits we need to make after all of these changes, but they can wait. @ladsmund has done a very thorough job reviewing this, and I just want to keep this moving forward. I just had one thing about the "ressources" module, which I opened in a PR to this branch #261.

src/pypromice/ressources/__init__.py

src/pypromice/process/get_l2.py

…her it's a Nead file

…laciology-and-Climate/pypromice into add-historical-data-to-l3

…tly from the dataset when sorting

src/pypromice/process/get_l2tol3.py

BaptisteVandecrux linked an issue Jun 4, 2024 that may be closed by this pull request

Merging of the station records at each site including historical stations #246

Open

BaptisteVandecrux added 2 commits June 12, 2024 10:46

update get_l3 to add historical data

0607d31

update get_l3 to add historical data

fe7149c

BaptisteVandecrux force-pushed the add-historical-data-to-l3 branch from f364c08 to fe7149c Compare June 12, 2024 08:48

Create get_l3_new.py

8f9e69d

BaptisteVandecrux mentioned this pull request Jun 13, 2024

Write L2 to file and leave all additional variable derivation for the L2toL3 step #255

Merged

BaptisteVandecrux and others added 14 commits June 13, 2024 11:45

update get_l3 to add historical data

e914403

update get_l3 to add historical data

a8b37ec

Create get_l3_new.py

e276064

Merge branch 'add-historical-data-to-l3' of https://github.com/GEUS-G…

50ef93e

…laciology-and-Climate/pypromice into add-historical-data-to-l3

further work on join_l3, varible_aliases in ressource folder

ef4ae44

cleaning up debug code in join_l3

b0ceb91

small fix in join_l3

a2fa829

working verion

4533b7f

delete encoding info after reading netcdf, debug of getColNames

320ab52

delete get_l3.py

1499e5d

removing variables and output files metadata

efbe35f

new variable description files

28160e0

added back ressource files, use level attributes for output definition

a847958

make list of sites from station_config, switched print to logger.info

569b8bd

BaptisteVandecrux added a commit that referenced this pull request Jun 16, 2024

brought changes from #252

5d140cb

ladsmund changed the base branch from main to 250-make-a-fast-track-l2-product-before-any-derived-variables-are-being-calculated June 17, 2024 10:46

ladsmund reviewed Jun 17, 2024

View reviewed changes

src/pypromice/process/L0toL1.py Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

ladsmund reviewed Jun 17, 2024

View reviewed changes

ladsmund reviewed Jun 18, 2024

View reviewed changes

BaptisteVandecrux force-pushed the add-historical-data-to-l3 branch from 352774f to c1a8735 Compare June 18, 2024 12:19

BaptisteVandecrux and others added 3 commits June 18, 2024 14:25

removing get_l3, remove inst. values from averaged files, fixes on lo…

22b88dd

…gging, attributes and tests,

Updates to numpy dependency version and pandas deprecation warnings (#…

a42c814

…258) * numpy dependency <2.0 * resample rules updated (deprecation warning) * fillna replaced with ffill (deprecation warning) * get_l3 called directly rather than from file * process action restructured

small changes following review, restored variable.csv history

96e6674

renamed new variable.csv moved old variable.csv renamed new variables.csv recreate variables.csv

BaptisteVandecrux force-pushed the add-historical-data-to-l3 branch from c1a8735 to 96e6674 Compare June 18, 2024 12:26

ladsmund reviewed Jun 18, 2024

View reviewed changes

BaptisteVandecrux added 6 commits June 18, 2024 15:41

buiding a station list instead of a station_dict

34643df

renamed l3m to l3_merged, reintroduced getVars and getMeta

35ea174

moving gcnet_postprocessing as part of readNead

ce802a7

sorting out the station_list in reverse chronological order

5e7673b

using tilde notation in setup.py

1272a7b

better initialisation of station_attributes attribute

d445a45

ladsmund reviewed Jun 19, 2024

View reviewed changes

src/pypromice/process/join_l3.py Outdated Show resolved Hide resolved

BaptisteVandecrux added 3 commits June 19, 2024 13:13

moved addMeta, addVars, roundValues, reformatTime, reformatLon to wri…

7756f30

…te.py

Inline comment describing enocding attribute removal when reading a n…

48817e6

…etcdf

loading toml file as dictionary within join_l3

a49de43

instead of just reading the stid to join

PennyHow approved these changes Jun 19, 2024

View reviewed changes

src/pypromice/ressources/__init__.py Outdated Show resolved Hide resolved

src/pypromice/process/get_l2.py Show resolved Hide resolved

PennyHow and others added 9 commits June 19, 2024 10:41

ressources renamed to resources (#261)

48f6823

using project attribute of a station locate AWS file and specify whet…

79a5972

…her it's a Nead file

Merge branch 'add-historical-data-to-l3' of https://github.com/GEUS-G…

ba8cd10

…laciology-and-Climate/pypromice into add-historical-data-to-l3

update test after moving addVars and addMeta

a184277

fixed logger message in resample

a2fe858

better definition of monthly sample rates in addMeta

bd1983c

dummy datasaet built in unit test now has 'level' attribute

036ba78

not storing timestamp_max for each station but pulling the info direc…

9d56d84

…tly from the dataset when sorting

removing unecessary import of addMeta, roundValues

1263d7f

ladsmund reviewed Jun 20, 2024

View reviewed changes

src/pypromice/process/get_l2tol3.py Show resolved Hide resolved

BaptisteVandecrux added a commit that referenced this pull request Jun 24, 2024

bringing latest changes from #262 and #252

dafd74e

ladsmund mentioned this pull request Jun 25, 2024

Align NetCDF data product attributes with ACDD Metadata Standards #259

Open

BaptisteVandecrux mentioned this pull request Jun 25, 2024

Add BUFR file creation script and fix unit test bug in test_get_bufr.py #263

Open

BaptisteVandecrux changed the base branch from 250-make-a-fast-track-l2-product-before-any-derived-variables-are-being-calculated to develop June 26, 2024 19:17

BaptisteVandecrux merged commit b82b586 into develop Jun 26, 2024
8 checks passed

BaptisteVandecrux deleted the add-historical-data-to-l3 branch June 26, 2024 19:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add historical data to l3 #252

Add historical data to l3 #252

BaptisteVandecrux commented Jun 4, 2024

This comment was marked as resolved.

ladsmund left a comment

PennyHow left a comment

Add historical data to l3 #252

Add historical data to l3 #252

Conversation

BaptisteVandecrux commented Jun 4, 2024

This comment was marked as resolved.

ladsmund left a comment

Choose a reason for hiding this comment

PennyHow left a comment

Choose a reason for hiding this comment