Refactor query method, change workflow with tidying data and remove u…

…nnecessary enums

CHANGELOG.rst

@@ -10,6 +10,10 @@ Development
 - DOCS: Fix precipitation height unit
 - DOCS: Fix examples with "recent" period
 - Make all parameter levels equal for all weather services to reduce complexity in code
+- Change ``tidy`` option to ``shape``, where ``shape="long"`` equals ``tidy=True`` and ``shape="wide"`` equals ``tidy=False``
+- Naming things: All things "Scalar" are now called "Timeseries", with settings prefix ``ts_``
+- Drop some unnecessary enums
+- Rename Environment Agency to ea in subspace
 
 0.53.0 (07.02.2023)
 *******************

README.rst

@@ -295,9 +295,9 @@ Library
     >>> from wetterdienst import Settings
     >>> from wetterdienst.provider.dwd.observation import DwdObservationRequest
     >>> settings = Settings( # default
-    ...     tidy=True,  # tidy data
-    ...     humanize=True,  # humanized parameters
-    ...     si_units=True  # convert values to SI units
+    ...     ts_shape="long",  # tidy data
+    ...     ts_humanize=True,  # humanized parameters
+    ...     ts_si_units=True  # convert values to SI units
     ... )
     >>> request = DwdObservationRequest(
     ...    parameter=["climate_summary"],

docs/contribution/services.rst

@@ -153,7 +153,7 @@ Step 2: Request class
 *********************
 
 The request class represents a request and carries all the required attributes as well as the values class that is
-responsible for acquiring the data later on. The implementation is based on `ScalarRequestCore` from `wetterdienst.core`.
+responsible for acquiring the data later on. The implementation is based on `TimeseriesRequest` from `wetterdienst.core`.
 
 Attributes:
 
@@ -226,14 +226,7 @@ Attributes:
 
     @property
     @abstractmethod
-    def _has_tidy_data(self) -> bool:
-        """If data is generally provided tidy -> then data should not be tidied but
-        rather tabulated if data is requested to not being tidy"""
-        pass
-
-    @property
-    @abstractmethod
-    def _unit_tree(self):
+    def _unit_base(self):
         pass
 
     @property
@@ -242,7 +235,7 @@ Attributes:
         """Class to get the values for a request"""
         pass
 
-`ScalarRequestCore` has one abstract method that has to be implemented: the `_all` which manages to get a listing of
+`TimeseriesRequest` has one abstract method that has to be implemented: the `_all` which manages to get a listing of
 stations for the requested datasets/parameters. The listing includes:
 - station_id
 - from_date
@@ -258,6 +251,6 @@ The names can be mapped using the `Columns` enumeration.
 Step 3: Values class
 *********************
 
-The values class is based on `ScalarValuesCore` and manages the acquisition of actual data. The
-class is also part of the `ScalarRequestCore` being accessed via the `_values` property. It has to implement the
+The values class is based on `TimeseriesValues` and manages the acquisition of actual data. The
+class is also part of the `TimeseriesRequest` being accessed via the `_values` property. It has to implement the
 `_collect_station_parameter` method that takes care of getting values of a parameter/dataset for a station id.

docs/data/coverage/dwd/observation.rst

@@ -137,10 +137,10 @@ Cloud types
 | Automated     | -1   |
 +---------------+------+
 
-Tidy parameters
+Long parameters
 ===============
 
-The tidy version of DWD observation data excludes several parameters
+DWD observation data excludes several parameters
 which contain strings. Those parameters are:
 
 - cloud type abbreviations (1 - 4) in **hourly cloud type** dataset

docs/library/api.rst

@@ -16,8 +16,8 @@ Observation
 .. autoclass:: wetterdienst.provider.dwd.observation.api.DwdObservationRequest
     :members:
 
-Forecast (Mosmix)
------------------
+Mosmix
+------
 
 .. autoclass:: wetterdienst.provider.dwd.mosmix.api.DwdMosmixRequest
     :members:
@@ -28,6 +28,15 @@ Radar
 .. autoclass:: wetterdienst.provider.dwd.radar.api.DwdRadarValues
     :members:
 
+Eaufrance
+=========
+
+Hubeau
+------
+
+.. autoclass:: wetterdienst.provider.eaufrance.hubeau.api.HubeauRequest
+    :members:
+
 ECCC (Environment and Climate Change Canada)
 ============================================
 
@@ -37,6 +46,24 @@ Observation
 .. autoclass:: wetterdienst.provider.eccc.observation.api.EcccObservationRequest
     :members:
 
+Environment Agency UK
+=====================
+
+Observation
+-----------
+
+.. autoclass:: wetterdienst.provider.ea.hydrology.api.EaHydrologyRequest
+    :members:
+
+Geosphere
+=========
+
+Observation
+-----------
+
+.. autoclass:: wetterdienst.provider.geosphere.observation.api.GeosphereObservationRequest
+    :members:
+
 NOAA (National Oceanic and Atmospheric Administration)
 ======================================================
 
@@ -45,3 +72,21 @@ Global Historical Climatology Network
 
 .. autoclass:: wetterdienst.provider.noaa.ghcn.api.NoaaGhcnRequest
     :members:
+
+National Weather Service
+========================
+
+Observation
+-----------
+
+.. autoclass:: wetterdienst.provider.nws.observation.api.NwsObservationRequest
+    :members:
+
+WSV
+====
+
+Observation
+-----------
+
+.. autoclass:: wetterdienst.provider.wsv.pegel.api.WsvPegelRequest
+    :members:

docs/library/core.rst

@@ -15,7 +15,7 @@ Core
 .. automodule:: wetterdienst.core.core
     :members:
 
-.. automodule:: wetterdienst.core.scalar
+.. automodule:: wetterdienst.core.timeseries
     :members:
 
 Settings

docs/usage/cli.rst

@@ -51,7 +51,7 @@ Command Line Interface
 
             # Output options
             [--format=<format>] [--pretty]
-            [--tidy] [--humanize] [--si-units]
+            [--shape=<shape>] [--humanize] [--si-units]
             [--dropna] [--skip_empty] [--skip_threshold=0.95]
 
             # Export options
@@ -112,7 +112,7 @@ Command Line Interface
         --sql-values                SQL filter to apply to values
 
     Transformation options:
-        --tidy                      Tidy DataFrame
+        --shape                     Shape of DataFrame, "long" or "wide"
         --humanize                  Humanize parameters
         --si-units                  Convert to SI units
         --skip_empty                Skip empty stations according to skip_threshold
@@ -173,8 +173,8 @@ Command Line Interface
         # Get daily climate summary data for specific stations in CSV format
         wetterdienst values --provider=dwd --network=observation --parameter=kl --resolution=daily --period=recent --station=1048,4411
 
-        # Get daily climate summary data for specific stations in tidy format
-        wetterdienst values --provider=dwd --network=observation --parameter=kl --resolution=daily --period=recent --station=1048,4411 --tidy
+        # Get daily climate summary data for specific stations in long format
+        wetterdienst values --provider=dwd --network=observation --parameter=kl --resolution=daily --period=recent --station=1048,4411 --shape="long"
 
         # Limit output to specific date
         wetterdienst values --provider=dwd --network=observation --parameter=kl --resolution=daily --period=recent --date=2020-05-01 --station=1048,4411
@@ -244,10 +244,10 @@ Command Line Interface
         wetterdienst values --provider=dwd --network=observation --parameter=kl --resolution=daily --period=recent \
             --station=1048,4411 --sql-values="SELECT * FROM data WHERE wind_gust_max > 20.0;"
 
-        # Filter measurements: Same as above, but use tidy format.
+        # Filter measurements: Same as above, but use long format.
         wetterdienst values --provider=dwd --network=observation --parameter=kl --resolution=daily --period=recent \
             --station=1048,4411 \
-            --tidy --sql-values="SELECT * FROM data WHERE parameter='wind_gust_max' AND value > 20.0;"
+            --shape="long" --sql-values="SELECT * FROM data WHERE parameter='wind_gust_max' AND value > 20.0;"
 
     Inquire metadata:
 

docs/usage/python-api.rst

@@ -156,12 +156,12 @@ or modify them for your very own request like
 
     from wetterdienst import Settings
 
-    settings = Settings(tidy=False)
+    settings = Settings(ts_shape="wide")
     print(settings)
 
 Settings has four layers of which those arguments are sourced:
-- Settings arguments e.g. Settings(tidy=True)
-- environment variables e.g. WD_SCALAR_TIDY = "0"
+- Settings arguments e.g. Settings(ts_shape="long")
+- environment variables e.g. WD_TS_SHAPE = "wide"
 - local .env file in the same folder (same as above)
 - default arguments set by us
 
@@ -185,41 +185,41 @@ and to set it back to standard
 
     from wetterdienst import Settings
 
-    settings = Settings(tidy=False)
+    settings = Settings(ts_shape="wide")
     settings = settings.reset()
 
 The environmental settings recognized by our settings are
 
 - WD_CACHE_DISABLE
 - WD_FSSPEC_CLIENT_KWARGS
-- WD_SCALAR_HUMANIZE
-- WD_SCALAR_TIDY
-- WD_SCALAR_SI_UNITS
-- WD_SCALAR_SKIP_EMPTY
-- WD_SCALAR_SKIP_THRESHOLD
-- WD_SCALAR_SKIP_CRITERIA
-- WD_SCALAR_DROPNA
-- WD_SCALAR_INTERPOLATION_USE_NEARBY_STATION_UNTIL_KM
-
-Scalar arguments are:
-- `tidy` can be used to reshape the returned data to a `tidy format`_.
+- WD_TS_HUMANIZE
+- WD_TS_SHAPE
+- WD_TS_SI_UNITS
+- WD_TS_SKIP_EMPTY
+- WD_TS_SKIP_THRESHOLD
+- WD_TS_SKIP_CRITERIA
+- WD_TS_DROPNA
+- WD_TS_INTERPOLATION_USE_NEARBY_STATION_UNTIL_KM
+
+TS arguments are:
+- `shape` can be used to reshape the returned data to a `long/tidy format`_, one of "long", "wide".
 - `humanize` can be used to rename parameters to more meaningful
 names.
 - `si_units` can be used to convert values to SI units.
-- `skip_empty` (requires option `tidy`) can be used to skip empty stations
+- `skip_empty` (requires option `shape="long"`) can be used to skip empty stations
     - empty stations are defined via `skip_threshold` which defaults to 0.95
      and requires all parameters that are requested (for an entire dataset all of the dataset parameters)
      to have at least 95 per cent of actual values (relative to start and end date if provided)
-- `skip_criteria` (requires option `tidy`) is the statistical criteria on which the percentage of actual values is
+- `skip_criteria` (requires option `shape="long"`) is the statistical criteria on which the percentage of actual values is
     calculated with options "min", "mean", "max", where "min" means the percentage of the lowest available parameter is
     taken, while "mean" takes the average percentage of all parameters and "max" does so for the parameter with the most
     percentage
 - `skip_threshold` is used in combination with `skip_empty` to define when a station is empty, with 1.0 meaning no
  values per parameter should be missing and e.g. 0.9 meaning 10 per cent of values can be missing
-- `dropna` (requires option `tidy`) is used to drop all empty entries thus reducing the workload
+- `dropna` (requires option `shape="long"`) is used to drop all empty entries thus reducing the workload
 - `fsspec_client_kwargs` can be used to pass arguments to fsspec, especially for querying data behind a proxy
 
-All of `tidy`, `humanize` and `si_units` are defaulted to True.
+Both `humanize` and `si_units` are defaulted to True and `shape` defaults with "long".
 
 If your system is running behind a proxy e.g. like `here <https://github.com/earthobservations/wetterdienst/issues/524>`_
 you may want to use the `trust_env` like
@@ -231,7 +231,7 @@ you may want to use the `trust_env` like
 
 to allow requesting through a proxy.
 
-.. _tidy format: https://vita.had.co.nz/papers/tidy-data.pdf
+.. _long/tidy format: https://vita.had.co.nz/papers/tidy-data.pdf
 
 Historical Weather Observations
 ===============================
@@ -308,7 +308,7 @@ Use the ``DwdObservationRequest`` class in order to get hold of stations.
     from wetterdienst.provider.dwd.observation import DwdObservationRequest, DwdObservationDataset, DwdObservationPeriod, DwdObservationResolution
     from wetterdienst import Settings
 
-    settings = Settings(tidy=True, humanize=True, si_units=True)
+    settings = Settings(ts_shape="long", ts_humanize=True, ts_si_units=True)
 
     request = DwdObservationRequest(
         parameter=[DwdObservationDataset.CLIMATE_SUMMARY, DwdObservationDataset.SOLAR],
@@ -431,7 +431,7 @@ This will skip empty stations according to some settings. See the finally collec
     from wetterdienst import Settings
     from wetterdienst.provider.dwd.observation import DwdObservationRequest, DwdObservationDataset, DwdObservationPeriod, DwdObservationResolution
 
-    settings = Settings(skip_empty=True, ignore_env=True, skip_criteria="min")
+    settings = Settings(ts_skip_empty=True, ts_skip_criteria="min", ignore_env=True)
 
     stations = DwdObservationRequest(
         parameter=[DwdObservationDataset.CLIMATE_SUMMARY, DwdObservationDataset.SOLAR],
@@ -654,7 +654,7 @@ The result data is provided through a virtual table called ``data``.
     from wetterdienst.provider.dwd.observation import DwdObservationRequest, DwdObservationDataset, DwdObservationPeriod, DwdObservationResolution
     from wetterdienst import Settings
 
-    settings = Settings(tidy=True, humanize=True, si_units=True)  # defaults
+    settings = Settings(ts_shape="long", ts_humanize=True, ts_si_units=True)  # defaults
 
     stations = DwdObservationRequest(
         parameter=[DwdObservationDataset.TEMPERATURE_AIR],
@@ -687,7 +687,7 @@ Examples:
         DwdObservationPeriod, DwdObservationResolution
     from wetterdienst import Settings
 
-    settings = Settings(tidy=True, humanize=True, si_units=True)  # defaults
+    settings = Settings(ts_shape="long", ts_humanize=True, ts_si_units=True)  # defaults
 
     stations = DwdObservationRequest(
         parameter=[DwdObservationDataset.TEMPERATURE_AIR],

docs/usage/restapi.rst

@@ -35,4 +35,4 @@ Observations
     http localhost:7890/api/dwd/observation/values stations==1048,4411 parameter==kl resolution==daily period==recent date==2020-08-01/2020-08-05
 
     # Observations with SQL.
-    http localhost:7890/api/dwd/observation/values stations==1048,4411 parameter==kl resolution==daily period==recent tidy==False sql=="SELECT * FROM data WHERE temperature_air_max_200 < 2.0;"
+    http localhost:7890/api/dwd/observation/values stations==1048,4411 parameter==kl resolution==daily period==recent shape=="wide" sql=="SELECT * FROM data WHERE temperature_air_max_200 < 2.0;"

example/mosmix_forecasts.py

@@ -25,7 +25,7 @@
 def mosmix_example():
     """Retrieve Mosmix mosmix data by DWD."""
     # A. MOSMIX-L -- Specific stations_result - each station with own file
-    settings = Settings(tidy=True, humanize=True)
+    settings = Settings(ts_shape=True, ts_humanize=True)
 
     request = DwdMosmixRequest(
         parameter=["DD", "ww"],

example/observations_sql.py

@@ -30,7 +30,7 @@
 
 def sql_example():
     """Retrieve temperature data by DWD and filter by sql statement."""
-    settings = Settings(tidy=True, humanize=True, si_units=False)
+    settings = Settings(ts_shape="long", ts_humanize=True, ts_si_units=False)
 
     request = DwdObservationRequest(
         parameter=[DwdObservationDataset.TEMPERATURE_AIR],

example/observations_station_gaussian_model.py

@@ -42,7 +42,7 @@ def station_example(start_date="2018-12-25", end_date="2022-12-25", name="Frankf
         resolution=DwdObservationResolution.DAILY,
         start_date=start_date,
         end_date=end_date,
-        settings=Settings(si_units=False),
+        settings=Settings(ts_si_units=False),
     )
     return stations.filter_by_name(name=name)