Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New overlapping criteria, fluffiness, and prm cleanup #86

Merged
merged 62 commits into from
Oct 2, 2023
Merged

Conversation

fpavogt
Copy link
Member

@fpavogt fpavogt commented Oct 18, 2022

Description:

This PR introduces a series of updates to the code, following the suggestions by @loforest in #83, including:

  • the default value for OKTA_LIM8 has been changed to 99 following the large scale tests of @loforest.
  • the GROUPING affinity has been changed from manhattan to euclidean, since the former is detrimental to rising/dropping layers.
  • the "vertical extrema" of slices (to assess if they overlap) are now computed as min/max +- alt_pad_perc * (thickness). alt_pad_perc is a new ampycloud GROUPING parameter that replaces overlap !
  • the minimum-required separation between layers can now be adjusted as a function of altitude, and is no longer hardcoded to +- 10'000 ft.
  • the cloud base is now computed as a percentile of the total amount of points (starting from the smallest ones), with the option to define a lookback percentile to only use a fraction of the most recent measurements.
  • the metarize routine now also computes the "fluffiness" of layers (see Compute the fluffiness of cloud layers #89). This requires to add scipy and statsmodels to the list of required dependencies.
  • the list of user-tunable parameters has been cleaned from all the superfluous ones.
  • the parameter OKTA_LIM0 has been replaced by MAX_HITS_OKTA0, which now specifies the maximum (absolute) number of hits in a given slice/group/layer for it to be considered to be 0 oktas.
  • the parameter OKTA_LIM8 has been replaced by MAX_HOLES_OKTA8, which now specifies the maximum (absolute) number of holes (= non-detection of cloud bases) in a given slice/group/layer for it to be considered to be 8 oktas.
  • layers are now required to be 250ft apart (below 10'000 ft) in order for a group to be split.

It also contains other less critical changes, including:

  • improved docstrings
  • more robust tests
  • bug fix in step_scale()

Error(s) fixed:

Fixes #85 (scientific validity tests are now always run fully, even if one fails). Fixes #84. Fixes #89. Fixes #90. Fixes #87. Fixes #92. Fixes #98.

With the merging of #101 this also: fixes #102, fixes #95, fixes #100, fixes #93, and fixes #99.

Checklists:

  • New code includes dedicated tests.
  • New code has been linted.
  • New code follows the project's style.
  • New code is compatible with the 3-Clause BSD license.
  • CHANGELOG has been updated.
  • AUTHORS has been updated.
  • Copyright years in module docstrings have been updated.

@fpavogt fpavogt self-assigned this Oct 18, 2022
@fpavogt fpavogt changed the title New overlapping criteria New overlapping criteria & fluffiness Oct 19, 2022
@fpavogt fpavogt mentioned this pull request Oct 20, 2022
7 tasks
@fpavogt fpavogt changed the title New overlapping criteria & fluffiness New overlapping criteria, fluffiness, and prm cleanup Oct 20, 2022
@fpavogt fpavogt mentioned this pull request Jul 13, 2023
7 tasks
@fpavogt fpavogt requested a review from regDaniel July 19, 2023 08:56
@fpavogt
Copy link
Member Author

fpavogt commented Jul 19, 2023

Hey @loforest, @regDaniel, @pjuda, @sibalm - fyi, this PR is now ready once again to have its scientific stability/validity evaluated. I have just merged #101 into it (which included a handful of small "logistical" Actions/Tests tweaks) so that all the tests are now green again.

Happy to talk live about the stability tests when you want, and in particular the necessity (maybe?) to fine-tune some of the parameters (old and new).

Side-note: the pytest action will be triggered twice per week as soon as this gets merged into develop. Only then will its result be accessible via RestAPI.

@fpavogt fpavogt merged commit 59408bc into develop Oct 2, 2023
15 checks passed
@fpavogt fpavogt mentioned this pull request Oct 2, 2023
7 tasks

# Run the code
out = run(data)
assert out.metar_msg() == 'SCT020'
assert out.metar_msg() == 'NCD'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be a minimal test case with an output which is not NCD but actually some cloud layers identified?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment