Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SCALAR: Improve handling skipping of empty data #889

Merged
merged 2 commits into from
Feb 26, 2023
Merged

SCALAR: Improve handling skipping of empty data #889

merged 2 commits into from
Feb 26, 2023

Conversation

gutzbenj
Copy link
Member

@gutzbenj gutzbenj commented Feb 19, 2023

This PR changes how we work with empty data on skip_empty=True.

When using filter_by_rank now empty stations are skipped based on actual percentage of values which is calculated for all requested parameters per station.

Thresholds for it can be defined with skip_criteria which is one of "min", "mean", "max", where

  • "min" means the lowest found percentage of all parameters
  • "mean" means the mean value of actual percentages of all parameters
  • "max" means the highest actual percentage of all parameters

Because we don't know from the start which station has enough values, after collecting all data there is a property on the values class named df_stations that holds all stations that were finally collected, so that you can get a sense for the data.

@gutzbenj gutzbenj force-pushed the skip-empty branch 2 times, most recently from a4b992e to a1dddef Compare February 25, 2023 18:06
@gutzbenj gutzbenj marked this pull request as ready for review February 25, 2023 23:57
@gutzbenj gutzbenj force-pushed the skip-empty branch 5 times, most recently from 9cb8cef to 1122f8d Compare February 26, 2023 11:24
@codecov
Copy link

codecov bot commented Feb 26, 2023

Codecov Report

Base: 89.65% // Head: 89.65% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (c2dac57) compared to base (568cc07).
Patch coverage: 92.30% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #889   +/-   ##
=======================================
  Coverage   89.65%   89.65%           
=======================================
  Files         104      104           
  Lines        7615     7666   +51     
  Branches      952      965   +13     
=======================================
+ Hits         6827     6873   +46     
- Misses        599      601    +2     
- Partials      189      192    +3     
Flag Coverage Δ
unittests 89.65% <92.30%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
wetterdienst/core/scalar/tools.py 91.66% <ø> (ø)
wetterdienst/core/scalar/request.py 86.63% <79.16%> (+0.23%) ⬆️
wetterdienst/core/scalar/values.py 82.47% <93.33%> (+0.53%) ⬆️
wetterdienst/core/scalar/result.py 90.19% <100.00%> (+1.30%) ⬆️
wetterdienst/provider/dwd/mosmix/api.py 86.54% <100.00%> (-0.09%) ⬇️
wetterdienst/provider/dwd/observation/api.py 87.66% <100.00%> (-2.12%) ⬇️
wetterdienst/provider/dwd/observation/fileindex.py 100.00% <100.00%> (ø)
wetterdienst/settings.py 89.28% <100.00%> (+0.39%) ⬆️
wetterdienst/ui/cli.py 73.84% <100.00%> (+0.10%) ⬆️
wetterdienst/ui/core.py 62.60% <100.00%> (ø)
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@gutzbenj gutzbenj force-pushed the skip-empty branch 8 times, most recently from 3568095 to 38db95b Compare February 26, 2023 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant