Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable variable ratpm10pm25 via aeroval #641

Merged
merged 10 commits into from
May 20, 2022
Merged

Conversation

jgriesfeller
Copy link
Member

This PR makes it possible to calculate the variable ratpm10pm25 via aeroval for observation.
The minimum snippet to use is this (example for AirNow obs network):

ODCSFUN = "AirNow;concpm10/AirNow;concpm25"
OBS_GROUNDBASED = {
    "AirNow": dict(
        obs_id="blablubb",
        obs_vars=["ratpm10pm25"],
        obs_type="ungridded",
        obs_vert_type="Surface",
        obs_merge_how={
            "ratpm10pm25": "eval",
        },
        obs_aux_requires={
            "ratpm10pm25": {
                "AirNow": ["concpm10", "concpm25"],
            }
        },
        obs_aux_funs={"ratpm10pm25": ODCSFUN},
        obs_aux_units={"ratpm10pm25": "1"},
    ),
}

Tests are still missing

@jgriesfeller jgriesfeller self-assigned this May 4, 2022
@codecov
Copy link

codecov bot commented May 4, 2022

Codecov Report

Merging #641 (541d99d) into main-dev (654ea63) will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           main-dev     #641      +/-   ##
============================================
+ Coverage     76.89%   76.90%   +0.01%     
============================================
  Files            98       98              
  Lines         17644    17643       -1     
============================================
+ Hits          13568    13569       +1     
+ Misses         4076     4074       -2     
Flag Coverage Δ
unittests 76.90% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pyaerocom/ungriddeddata.py 61.79% <ø> (-0.03%) ⬇️
pyaerocom/units_helpers.py 100.00% <ø> (ø)
pyaerocom/_lowlevel_helpers.py 82.99% <0.00%> (+0.50%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 654ea63...541d99d. Read the comment docs.

@jgriesfeller jgriesfeller requested a review from avaldebe May 4, 2022 13:04
@@ -48,6 +48,10 @@
["concoa", "ug C/m3", "ug m-3", 1.0],
["concoc", "ug C/m3", "ug m-3", 1.0],
["conctc", "ug C/m3", "ug m-3", 1.0],
# a little hacky for ratpm10pm25...
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed because pyaerocom always tries to convert evrerything into the target unit first

Copy link
Collaborator

@avaldebe avaldebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would appreciate some tests to cover the new unit helpers and the NotImplementedError obs filter

pyaerocom/units_helpers.py Show resolved Hide resolved
Comment on lines +1233 to +1238
except (
VarNotAvailableError,
TimeMatchError,
DataCoverageError,
NotImplementedError,
) as e:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if this lines are covered somewhere on the test suite, but it would be great to test that NotImplementedError can be used to filter obs

Copy link
Member Author

@jgriesfeller jgriesfeller May 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how to test these exceptions effectively. Any ideas?

Copy link
Member Author

@jgriesfeller jgriesfeller May 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only ideas I have right now is a fake reader that throws these exceptions or a fabricated data set. Both takes some time to create.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will extend these exceptions for the next release (e.g. with the exception raised when a station moved more than a kilometer).
How about postponing that test until then?

pyaerocom/data/variables.ini Outdated Show resolved Hide resolved
@jgriesfeller
Copy link
Member Author

after speaking with @avaldebe : merging

@jgriesfeller jgriesfeller merged commit 391375d into main-dev May 20, 2022
@jgriesfeller jgriesfeller deleted the griesie_fix519_v3 branch May 20, 2022 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EEA NRT reading: concpm25 reading fails due to overlapping station data (two seperate stations)
2 participants