Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Earlinet reader and implement vertical profiles #855

Merged
merged 166 commits into from
Oct 3, 2023
Merged

Conversation

lewisblake
Copy link
Member

Need to upgrade the EARLINET reader to the new format, and move VerticalProfiles through the rest of the pyaerocom pipeline to Aeroval.

@lewisblake lewisblake self-assigned this Apr 25, 2023
@lewisblake lewisblake added obs-update Update related to existing observation dataset CAMS2_82 Issues related to the CAMS2_82 contract labels Apr 25, 2023
@lewisblake lewisblake mentioned this pull request May 4, 2023
4 tasks
@lewisblake
Copy link
Member Author

Some stations have netcdf with multiple observations (timesteps). We currently kick them out. Could think of a more elegant solution, but that would need some input from others

Copy link
Member

@jgriesfeller jgriesfeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the small change I commented on and merge then. No need for a new review.

Comment on lines 400 to 401
if regrid_res_deg is not None:
data = _regrid_gridded(data, regrid_scheme, regrid_res_deg)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if these lines are nor covered by test, maybe the if clause should be marked with # pragma: no cover

Comment on lines 412 to 415
if use_climatology_ref:
col_freq = "monthly"
obs_start = const.CLIM_START
obs_stop = const.CLIM_STOP
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seem my prebious comment about not covered lines

Comment on lines +15 to +17
TEST_FILE = [
ROOT / "earlinet_example_for_ci.pkl",
]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that pickeø files will work across different versions of Python and numpy/netCDF4

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this is a concern. Either we find a more robust way of storing the UngriddedData object (not obvious to me now, would require some research), or I write a fixture to create this UngriddedData object on the fly. I'm happy to open a PR to patch this after the fact if you are okay merging for now

Copy link
Collaborator

@avaldebe avaldebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is the use of pickle files on testing.
This works now, but there is not guaranty that it will continue to work with new version of Pyhon/numpy/NetcDF4/....

@lewisblake lewisblake merged commit 53d9387 into main-dev Oct 3, 2023
15 of 19 checks passed
charlienegri added a commit that referenced this pull request Oct 12, 2023
This reverts commit 53d9387, reversing
changes made to 74f6fba.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CAMS2_82 Issues related to the CAMS2_82 contract obs-update Update related to existing observation dataset
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Vertical Profiles Earlinet and CAMS extinction profile comparison
3 participants