Ipc forests #746

dulte · 2022-09-26T11:22:02Z

Reader for IPC Forests data for deposition

…t correction

# Conflicts: # pyaerocom/aeroval/glob_defaults.py

jgriesfeller · 2023-01-26T12:29:01Z

The reading of the variable fakedryo3 fails in the EBAS reader:

Running MOCAGE.cams2.40 (dryo3) vs. EBASMC (fakedryo3)
Cache file does not exist: /lustre/storeB/project/aerocom/aerocom2/pyaerocom_out/_cache/jang/EBASMC_fakedryo3.pkl
Fetching data files. This might take a while...
Retrieving EBAS files for variables
['fakedryo3']
Number of files to read reduced to 5736
Reading EBAS data from /lustre/storeB/project/aerocom/aerocom1/AEROCOM_OBSDATA/EBASMultiColumn/data/data
^M  0%|          | 0/5736 [00:00<?, ?it/s]Reading NASA Ames file:
/lustre/storeB/project/aerocom/aerocom1/AEROCOM_OBSDATA/EBASMultiColumn/data/data/AM0001R.20081231200000.20120419000000.uv_abs.ozon
e.air.4h.1h.AM01L_uv_abs_02.AM01L_uv_abs..nas
^M  0%|          | 0/5736 [00:00<?, ?it/s]
Failed to perform analysis: Traceback (most recent call last):
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/colocation_auto.py", line 802, in run
    coldata = self._run_helper(mod_var, obs_var)
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/colocation_auto.py", line 1429, in _run_helper
    args = self._prepare_colocation_args(model_var, obs_var)
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/colocation_auto.py", line 1372, in _prepare_colocation_args
    obs_data = self.get_obs_data(obs_var)
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/colocation_auto.py", line 705, in get_obs_data
    return self._read_ungridded(obs_var)
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/colocation_auto.py", line 908, in _read_ungridded
    obs_data = obs_reader.read(
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/io/readungridded.py", line 649, in read
    self.read_dataset(
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/io/readungridded.py", line 393, in read_dataset
    data_read = reader.read(vars_to_read, **kwargs)
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/io/read_ebas.py", line 1745, in read
    data = self._read_files(files, vars_to_retrieve, files_contain, constraints)
  File "/modules/rhel8/user-apps/aerocom/conda2022/envs/pya_para_ipc/lib/python3.10/site-packages/pyaerocom/io/read_ebas.py", line 1861, in _read_files
    data_obj._data[start:stop, data_obj._DATAINDEX] = values
ValueError: could not convert string to float: 'dtime'

avaldebe · 2023-01-26T12:34:10Z

File "/modules/centos7/user-apps/aerocom/anaconda3/envs/pya_para_ipc/lib/python3.9/site-packages/pyaerocom/io/ipcforests/reader.py", line 354, in ReadIPCForest
start: str | datetime,
TypeError: unsupported operand type(s) for |: 'type' and 'type'

type | type was introduced on Python 3.10. On Python 3.8 and 3.9 you need to add the following import as the first import to get this behaviour:

from __future__ import annotations

jgriesfeller · 2023-01-26T12:41:03Z

I updated the comment meanwhile, but yes, some of Daniel's code is using Python3.10 features and was never tested on the earlier version we also still support.

# Conflicts: # pyaerocom/aeroval/glob_defaults.py

# Conflicts: # pyaerocom/aeroval/glob_defaults.py # pyaerocom/molmasses.py

avaldebe

This PR extends the EBAS variables with many helper functions for derived quantities. However, I do not see the tests for those helper fuctions.

pyaerocom/aeroval/glob_defaults.py

pyaerocom/aeroval/coldatatojson_helpers.py

avaldebe · 2023-10-20T09:09:49Z

pyaerocom/aeroval/coldatatojson_helpers.py

+        "seasonal": {"obs": deepcopy(yeardict), "mod": deepcopy(yeardict)},
+        "yearly": {"obs": deepcopy(yeardict), "mod": deepcopy(yeardict)},


I understand the need for deepcopy to create different copies of the same empty dictionary, but would not be better to use collections.defaultdict (see docs)?

Here is how it looks like with collections.defaultdict

from collections import defaultdict ... ts_data = { "time": time, "seasonal": {"obs": defaultdict(dict), "mod": defaultdict(dict)}, "yearly": {"obs": defaultdict(dict), "mod": defaultdict(dict)}, }

I just did not know about defaultdict, but did you see the init of yeardict?

pyaerocom/pyaerocom/aeroval/coldatatojson_helpers.py

Lines 422 to 425 in d0b18cd

years = list(repw_res["seasonal"].year.values)

yeardict = {}

for year in years:

yeardict[f"{year}"] = {}

yeardict has some predifined keys which wouldn't be in your defaultdict(dict), right?

pyaerocom/aeroval/coldatatojson_helpers.py

# Conflicts: # pyaerocom/aeroval/glob_defaults.py # pyaerocom/data/variables.ini # tests/test_varcollection.py

jgriesfeller · 2023-10-31T13:03:58Z

This PR extends the EBAS variables with many helper functions for derived quantities. However, I do not see the tests for those helper fuctions.

Adding data to the EBAS test dataset is a pain and takes hours to do. I have added some of the new variables to test_ebas_varinfo.py though. Is that satisfying for you?

avaldebe · 2023-10-31T13:25:43Z

This PR extends the EBAS variables with many helper functions for derived quantities. However, I do not see the tests for those helper fuctions.

Adding data to the EBAS test dataset is a pain and takes hours to do. I have added some of the new variables to test_ebas_varinfo.py though. Is that satisfying for you?

I'm fine with the extra tests

dulte and others added 17 commits August 10, 2022 11:58

starts ipc forests reader

0e07a83

Halfway through making reader

b165d8b

First working version of reader

4ee082c

Sampler type filter added

c4bf576

Adds sea salt correction to so4 dep

0381478

Changes how and where ts_type is calculated

d5c78c1

Adds some docstrings to metadata.py

c047ba2

Adds total dep, check quality flags, and uses only the lowest sea sal…

638316a

…t correction

Merge branch 'main-dev' into ipc-forests

c0ef4d4

Can now read new model data

b4b14ee

Adds multiple variables

8dfe6bb

More variables for task4041

9b250a0

Adds vmr to conc for aerocom model data, in a compicated way

6527c4f

Adds more variables to task4041

f7ec1a7

Merge branch 'main-dev' into ipc-forests

fb69568

Adds fakedrydep and totdep

a73f473

change path to standard aerocm location

6c90924

jgriesfeller self-assigned this Jan 2, 2023

Jan Jurgen Griesfeller and others added 6 commits January 9, 2023 14:32

Merge branch 'griesie_fix_802' into ipc-forests

42a29db

# Conflicts: # pyaerocom/aeroval/glob_defaults.py

switch to storeB

bedee59

remove another storeA location

7ba24e0

Merge remote-tracking branch 'origin/ipc-forests' into ipc-forests

e3eb3ce

Merge branch 'griesie-fix-759' into ipc-forests

f23f897

Merge remote-tracking branch 'origin/main-dev' into ipc-forests

d844879

# Conflicts: # pyaerocom/aeroval/glob_defaults.py

jgriesfeller and others added 3 commits January 26, 2023 13:46

make compatible to Python versions < 3.10

539a7f0

added udunit for mg/m2/h

13ff522

Merge remote-tracking branch 'origin/main-dev' into ipc-forests

79d5aba

jgriesfeller and others added 16 commits October 6, 2023 15:56

Merge remote-tracking branch 'origin/ipc-forests' into ipc-forests

143cc9a

Merge remote-tracking branch 'origin/fix922' into ipc-forests

827ca7a

various changes to variable wetna for aeroval

f5d9504

added deposition to forrest variables

41b4669

Merge remote-tracking branch 'origin/ipc-forests' into ipc-forests

4021179

# Conflicts: # pyaerocom/aeroval/glob_defaults.py

Merge remote-tracking branch 'origin/main-dev' into ipc-forests

3bf5929

# Conflicts: # pyaerocom/aeroval/glob_defaults.py # pyaerocom/molmasses.py

add statistics_model_only again that was accidently removed during merge

7ee8042

added proxy wetdeps

ec52c40

Merge remote-tracking branch 'origin/fix_diurnal' into ipc-forests

d3a65af

Merge remote-tracking branch 'origin/main-dev' into ipc-forests

3d1d71b

linting

671f417

adjust to new plugin structure for obs reading

f3b8b3d

Merge remote-tracking branch 'origin/main-dev' into ipc-forests

b943b20

adjust tests

8e0ab9d

adjust tests

322d9f1

linting

7862fcd

jgriesfeller requested a review from avaldebe October 19, 2023 08:39

avaldebe requested changes Oct 20, 2023

View reviewed changes

Jan Jurgen Griesfeller added 7 commits October 31, 2023 11:20

Merge remote-tracking branch 'origin/main-dev' into ipc-forests

1f84fce

# Conflicts: # pyaerocom/aeroval/glob_defaults.py # pyaerocom/data/variables.ini # tests/test_varcollection.py

added more variables

47529cb

removed duplicate wetpm10 entry introduced by merging mein-dev

d0b18cd

removed unused code

288de77

adjusted to review

e978e28

adjusted to review

da622bf

adjusted test

1016ee7

avaldebe approved these changes Oct 31, 2023

View reviewed changes

jgriesfeller merged commit bfc8073 into main-dev Oct 31, 2023
19 of 21 checks passed

jgriesfeller deleted the ipc-forests branch October 31, 2023 14:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ipc forests #746

Ipc forests #746

dulte commented Sep 26, 2022

jgriesfeller commented Jan 26, 2023 •

edited

Loading

avaldebe commented Jan 26, 2023

jgriesfeller commented Jan 26, 2023

avaldebe left a comment

avaldebe Oct 20, 2023

jgriesfeller Oct 31, 2023 •

edited

Loading

jgriesfeller commented Oct 31, 2023

avaldebe commented Oct 31, 2023

		"seasonal": {"obs": deepcopy(yeardict), "mod": deepcopy(yeardict)},
		"yearly": {"obs": deepcopy(yeardict), "mod": deepcopy(yeardict)},

	years = list(repw_res["seasonal"].year.values)
	yeardict = {}
	for year in years:
	yeardict[f"{year}"] = {}

Ipc forests #746

Ipc forests #746

Conversation

dulte commented Sep 26, 2022

jgriesfeller commented Jan 26, 2023 • edited Loading

avaldebe commented Jan 26, 2023

jgriesfeller commented Jan 26, 2023

avaldebe left a comment

Choose a reason for hiding this comment

avaldebe Oct 20, 2023

Choose a reason for hiding this comment

jgriesfeller Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

jgriesfeller commented Oct 31, 2023

avaldebe commented Oct 31, 2023

jgriesfeller commented Jan 26, 2023 •

edited

Loading

jgriesfeller Oct 31, 2023 •

edited

Loading