Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dependencies: Bump a few versions, start using version range constraints #1017

Merged
merged 6 commits into from
Sep 29, 2023
Merged

Dependencies: Bump a few versions, start using version range constraints #1017

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b26e3d2
Tests: Skip IMGW meteorology tests because of a flaw in data
amotl Sep 29, 2023
750e3b5
Dependencies: Bump a few versions, start using version range constraints
amotl Sep 29, 2023
3118817
Dependencies: Use duckdb>=0.9,<0.10
amotl Sep 29, 2023
4cd6ffd
Dependencies: Use dash>=2.10,<2.14
amotl Sep 29, 2023
c7c0adf
CI: Configure `cache-dependency-path` for `actions/setup-python@v4`
amotl Sep 29, 2023
8df10dc
Dependencies: AttributeError: module 'percy' has no attribute 'Runner'
amotl Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ jobs:
python-version: "3.11"
architecture: x64
cache: poetry
cache-dependency-path: |
pyproject.toml
poetry.lock
Comment on lines +37 to +39
Copy link
Member Author

@amotl amotl Sep 29, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better safe than sorry. ;]


- name: Install project
run: .github/workflows/install.sh testing
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ if [ "${flavor}" = "testing" ]; then
--extras=restapi \
--extras=sql

# FIXME: Remove this again.
# Fix dependency woes about percy: AttributeError: module 'percy' has no attribute 'Runner'.
# https://github.com/earthobservations/wetterdienst/pull/1017#issuecomment-1741240635
# pytest tests/ui/explorer/test_explorer.py -k test_app_layout
poetry run poe fix-percy

elif [ "${flavor}" = "docs" ]; then
poetry install --verbose --no-interaction --with=docs --extras=interpolation

Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ jobs:
python-version: "3.11"
architecture: x64
cache: poetry
cache-dependency-path: |
pyproject.toml
poetry.lock

- name: Install project tooling
run: poetry install --no-interaction --no-root --with=dev
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ jobs:
python-version: "3.11"
architecture: x64
cache: poetry
cache-dependency-path: |
pyproject.toml
poetry.lock

- name: Install project
run: .github/workflows/install.sh testing
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ jobs:
python-version: ${{ matrix.python-version }}
architecture: x64
cache: poetry
cache-dependency-path: |
pyproject.toml
poetry.lock

- name: Install eccodes (Mac only)
if: startsWith(matrix.os, 'macOS')
Expand Down
1,218 changes: 607 additions & 611 deletions poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

133 changes: 67 additions & 66 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,73 +101,73 @@ classifiers = [
[tool.poetry.dependencies]
python = "^3.9,!=3.9.7,<3.12"

aenum = "^3.0"
aiohttp = "^3.8.1"
backports-datetime-fromisoformat = "^2.0.0"
beautifulsoup4 = "^4.9"
cachetools = "^5.2"
click = "^8.0"
click-params = "^0.4.1"
cloup = "^3.0.1"
deprecation = "^2.1"
diskcache = "^5.4.0"
environs = "^9.4.0"
aenum = ">=3,<3.2"
aiohttp = ">=3.8,<3.9"
backports-datetime-fromisoformat = ">=2,<3"
beautifulsoup4 = ">=4.9,<5"
cachetools = ">=5.2,<6"
click = ">=8,<9"
click-params = ">=0.4.1,<5"
cloup = ">=3.0.1,<4"
deprecation = ">=2.1,<3"
diskcache = ">=5.4.0,<6"
environs = ">=9.4.0,<10"
fsspec = "^2023.01"
lxml = "^4.9.1"
measurement = "^3.2"
numpy = "^1.22"
pandas = "^2.0"
Pint = "^0.17"
platformdirs = "^3.8"
lxml = ">=4.9.1,<5"
measurement = ">=3.2,<4"
numpy = ">=1.22,<1.27"
pandas = ">=2,<2.2"
Pint = ">=0.17,<0.23"
Copy link
Member Author

@amotl amotl Sep 29, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it technically works with Pint, because the package is compatible enough, I believe it is the right way to write down dependencies in ranges of version numbers, instead of anything else - modulo exceptions ;].

Only writing >=0.17 would be dangerous, because it is very likely to break until a 1.x release.

platformdirs = "<4"
polars = "^0.16"
pyarrow = "^13.0"
pypdf = "^3.12.1"
python-dateutil = "^2.8.2"
rapidfuzz = "^3.1"
requests = "^2.20"
python-dateutil = "<3"
rapidfuzz = ">=3.1,<4"
requests = ">=2.20,<3"
scikit-learn = "^1.0.2"
stamina = "^23.1.0"
stamina = ">=23,<24"
tabulate = "^0.8"
timezonefinder = "^6.1"
tqdm = "^4.47"
timezonefinder = "<7"
tqdm = ">=4,<5"
tzdata = "^2023.3"
xlsx2csv = "^0.8.1"
xlsxwriter = "^3.0.9"
xlsx2csv = ">=0.8.1,<0.9"
xlsxwriter = ">=3.0.9,<3.2"

crate = { version = "^0.30.1", optional = true } # Export feature.
dash = { version = "^2.8", optional = true } # Explorer UI feature.
dash-bootstrap-components = { version = "^1.4", optional = true } # Explorer UI feature.
dash-leaflet = { version = "^1.0.8", optional = true } # Explorer UI feature.
duckdb = { version = "^0.7.1", optional = true } # Export feature.
crate = { version = ">=0.30.1,<1", optional = true } # Export feature.
dash = { version = ">=2.10,<2.14", optional = true } # Explorer UI feature.
dash-bootstrap-components = { version = ">=1.4,<2", optional = true } # Explorer UI feature.
dash-leaflet = { version = ">=1.0.8,<2", optional = true } # Explorer UI feature.
duckdb = { version = ">=0.9,<0.10", optional = true } # Export feature.
eccodes = { version = "^1.5.2", optional = true }
fastapi = { version = "^0.95.1", optional = true } # HTTP REST API feature.
geojson = { version = "^3.0.0", optional = true } # Explorer UI feature.
httpx = {version = "^0.24.0", optional = true}
h5py = { version = "^3.1", optional = true } # Radar feature.
influxdb = { version = "^5.3", optional = true } # Export feature.
influxdb-client = { version = "^1.18", optional = true } # Export feature.
geojson = { version = ">=3,<4", optional = true } # Explorer UI feature.
httpx = {version = ">=0.24,<0.26", optional = true}
h5py = { version = ">=3.1,<4", optional = true } # Radar feature.
influxdb = { version = ">=5.3,<6", optional = true } # Export feature.
influxdb-client = { version = ">=1.18,<2", optional = true } # Export feature.
matplotlib = { version = "^3.3", optional = true }
mysqlclient = { version = "^2.0", optional = true } # Export feature.
openpyxl = { version = "^3.0", optional = true }
pdbufr = { version = "^0.10.2", optional = true, extras = ["eccodes"] }
mysqlclient = { version = ">=2,<3", optional = true } # Export feature.
openpyxl = { version = ">=3,<4", optional = true }
pdbufr = { version = ">=0.10.2,<0.12", optional = true, extras = ["eccodes"] }
plotly = { version = "^5.11", optional = true } # Explorer UI feature.
psycopg2-binary = { version = "^2.8", optional = true } # Export feature.
psycopg2-binary = { version = ">=2.8,<3", optional = true } # Export feature.
scipy = { version = "^1.9", optional = true } # Interpolation feature.
shapely = { version = "^2.0", optional = true } # Interpolation feature.
sqlalchemy = { version = "^2.0", optional = true } # Export feature.
streamlit = { version = "^1.27.0", optional = true } # Streamlit app
utm = { version = "^0.7", optional = true } # Interpolation feature.
uvicorn = { version = "^0.14", optional = true } # HTTP REST API feature.
sqlalchemy = { version = ">=2,<2.1", optional = true } # Export feature.
streamlit = { version = ">=1.27,<2", optional = true } # Streamlit app
utm = { version = ">=0.7,<1", optional = true } # Interpolation feature.
uvicorn = { version = ">=0.14,<1", optional = true } # HTTP REST API feature.
wradlib = { version = "^1.19", optional = true } # Radar feature.
xarray = { version = "^2023.1", optional = true }
xradar = { version = "^0.2.0", optional = true } # Radar feature.
zarr = { version = "^2.13", optional = true } # Export feature.
zarr = { version = ">=2.13,<3", optional = true } # Export feature.

[tool.poetry.group.dev]
optional = true

[tool.poetry.group.dev.dependencies]
black = { version = "^23.1", extras = ["jupyter"] }
black = { version = ">=23.1,<24", extras = ["jupyter"] }
# TODO: currently there is a version constraint with jsonschema here
#cff-from-621 = { version = "^0.0.1", python = ">= 3.10"}
poethepoet = "^0.18.1"
Expand All @@ -177,24 +177,24 @@ ruff = "^0.0.250"
optional = true

[tool.poetry.group.test.dependencies]
coverage = { version = "^7.3", extras = ["toml"] }
dash = { version = "^2.6", extras = ["testing"] }
dirty-equals = "^0.5.0"
freezegun = "^1.2"
ipykernel = "^6.19.4"
jsonschema = { version = "^4.17.3", extras = ["format-nongpl"] }
jupyter = "^1.0.0"
coverage = { version = ">=7.3,<8", extras = ["toml"] }
dash = { version = ">=2.10,<2.14", extras = ["testing"] }
dirty-equals = ">=0.5.0,<1"
freezegun = ">=1.2,<2"
ipykernel = ">=6.19.4,<7"
jsonschema = { version = ">=4.17.3,<5", extras = ["format-nongpl"] }
jupyter = ">=1,<2"
lmfit = "^1.1.0" # required for example observations_station_gaussian_model.py
percy-selenium = "^2.0.0"
pybufrkit = "^0.2"
pytest = "^7.2"
pytest-cov = "^4.0"
pytest-dictsdiff = "^0.5"
pytest-notebook = "^0.8"
pytest-xdist = "^3"
selenium = "^4.0"
percy-selenium = "<2"
pybufrkit = ">=0.2,<0.3"
pytest = ">=7.2,<8"
pytest-cov = ">=4,<5"
pytest-dictsdiff = ">=0.5,<0.6"
pytest-notebook = ">=0.8,<0.9"
pytest-xdist = ">=3,<4"
selenium = ">=4,<5"
surrogate = "^0.1"
webdriver-manager = "^4.0.0"
webdriver-manager = ">=4,<5"
Copy link
Member Author

@amotl amotl Sep 29, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me use this as an example to demonstrate why I prefer the version range notation: When version 5 of this package will be released, Dependabot will submit a change to >=4,<6, and the outcome from CI will immediately tell whether this is good or not. If version 5 works, and just because it is version 5 already, I am trusting their maintainers enough that it will not break during the whole version 5 release cycle.

On the other hand, a "point" bump would raise the lower version bound of this package to ">=5", which is an unneccessary restriction if it still works with version 4, and on the other hand, too dangerous without an upper bound, because nobody knows what version 6 will be all about.

Those thoughts include a series of assumptions and decisions I would like to make individually per dependency. They hopefully outline my thoughts why I am so much in favor of version range notations. Of course, it only explains my thoughts on behalf of a single example, but I am sure you can extrapolate from that.


h5py = { version = "^3.1", optional = true}

Expand All @@ -206,12 +206,12 @@ docformatter = "^1.4"
furo = "^2023.8.19"
ipython = "^8.5"
matplotlib = "^3.3"
sphinx = "^7.2.5"
sphinx-autodoc-typehints = "^1.11"
sphinx = ">=7,<8"
sphinx-autodoc-typehints = ">=1.11,<2"
sphinx-autobuild = "^2021.3"
sphinx_design = "^0.5.0"
sphinxcontrib-svg2pdfconverter = "^1.1"
tomlkit = "^0.7"
sphinx_design = ">=0.5,<1"
sphinxcontrib-svg2pdfconverter = ">=1.1,<2"
tomlkit = ">=0.7,<1"

[tool.poetry.extras]
bufr = ["pybufrkit", "pdbufr"]
Expand Down Expand Up @@ -313,6 +313,7 @@ update = "poetry update"
citation = "python -m tools.citation"

streamlit = "streamlit run ./wetterdienst/ui/streamlit/app.py"
fix-percy = "python -m pip install percy>=2,<3 --force"

[tool.pytest.ini_options]
addopts = "-rsfEX -p pytester --strict-markers --verbosity=3 --webdriver=Firefox --headless"
Expand Down
5 changes: 5 additions & 0 deletions tests/provider/imgw/meteorology/test_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,14 @@
from zoneinfo import ZoneInfo

import polars as pl
import pytest
from polars.testing import assert_frame_equal

from wetterdienst.provider.imgw.meteorology.api import ImgwMeteorologyRequest, ImgwMeteorologyResolution


# FIXME: Deactivated after flaw in data discovered on 2023-09-26
@pytest.mark.skip(reason="Deactivated after flaw in data discovered on 2023-09-26")
Comment on lines +14 to +15
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will you separately take care about this regression, @gutzbenj?

def test_imgw_meteorology_api_daily():
request = ImgwMeteorologyRequest(
parameter="klimat",
Expand Down Expand Up @@ -127,6 +130,8 @@ def test_imgw_meteorology_api_daily():
assert_frame_equal(values.df.sort("parameter"), df_expected_values)


# FIXME: Deactivated after flaw in data discovered on 2023-09-26
@pytest.mark.skip(reason="Deactivated after flaw in data discovered on 2023-09-26")
Comment on lines +133 to +134
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dito.

def test_imgw_meteorology_api_monthly():
request = ImgwMeteorologyRequest(
parameter="synop",
Expand Down
9 changes: 7 additions & 2 deletions tests/test_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,14 @@
# Environment and Climate Change Canada
("eccc", "observation", {"parameter": "daily", "resolution": "daily"}, None), # noqa: E800, ERA001
# IMGW Hydrology
("imgw", "hydrology", {"parameter": "hydrology", "resolution": "daily"}, None),
# FIXME: Deactivated after flaw in data discovered on 2023-09-26
pytest.param(
"imgw", "hydrology", {"parameter": "hydrology", "resolution": "daily"}, None, marks=pytest.mark.xfail
),
# IMGW Meteorology
("imgw", "meteorology", {"parameter": "climate", "resolution": "daily"}, "249200180"),
pytest.param(
"imgw", "meteorology", {"parameter": "climate", "resolution": "daily"}, "249200180", marks=pytest.mark.xfail
),
Comment on lines -38 to +45
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also here.

# NOAA Ghcn
("noaa", "ghcn", {"parameter": "precipitation_height"}, None),
# WSV Pegelonline
Expand Down