Skip to content

Commit

Permalink
Merge branch 'main' into pre-commit-ci-update-config
Browse files Browse the repository at this point in the history
  • Loading branch information
RaczeQ committed May 29, 2024
2 parents 793f614 + 8018c5f commit 1d74978
Show file tree
Hide file tree
Showing 38 changed files with 2,797 additions and 1,810 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
with: |
env_vars: OS,PYTHON
fail_ci_if_error: true
files: ./coverage.python${{ matrix.python-version }}.xml,
files: ./coverage.python${{ matrix.python-version }}.xml,!.
flags: ${{ matrix.os }}-python${{ matrix.python-version }}
verbose: true
token: ${{ secrets.CODECOV_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
python-version: "3.12"
- uses: actions/cache@v3
with:
path: ~/.cache/pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
python-version: "3.12"
- uses: actions/cache@v3
with:
path: ~/.cache/pip
Expand Down
94 changes: 57 additions & 37 deletions .github/workflows/manual_tests.yml
Original file line number Diff line number Diff line change
@@ -1,40 +1,60 @@
name: "Run tests with newest dependencies"
name: "Run tests with newest and oldest dependencies"
on: [workflow_dispatch, workflow_call]

jobs:
run-tests:
name: Run tests 🛠️ on multiple systems 🖥️ and Python 🐍 versions
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ["3.9", "3.10", "3.11", "3.12"]
include:
- os: macos-latest
python-version: "3.12"
- os: macos-13
python-version: "3.12"
- os: windows-latest
python-version: "3.12"
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v3
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install pdm
run: pip install pdm
- name: Generate lock with newest dependencies
run: pdm lock --lockfile pdm.newest.lock --strategy no_cross_platform -dG:all
- name: Install quackosm and tests dependencies
run: pdm install --lockfile pdm.newest.lock -dG:all
- name: Run tests with pytest
run: |
pdm run pytest -v -s --durations=20 --doctest-modules --doctest-continue-on-failure quackosm
pdm run pytest -v -s --durations=20 tests/base
pdm run pytest -v -s --durations=20 tests/optional_imports
pdm run pytest -v -s --durations=20 tests/benchmark
run-tests-newest:
name: Run tests 🛠️ on multiple systems 🖥️ and Python 🐍 versions (newest dependencies)
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-13, macos-latest, windows-latest]
python-version: ["3.12"]
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v3
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install pdm
run: pip install pdm
- name: Generate lock with newest dependencies
run: pdm lock --lockfile pdm.newest.lock --strategy no_cross_platform -dG:all
- name: Install quackosm and tests dependencies
run: pdm install --lockfile pdm.newest.lock -dG:all --skip=post_install
- name: Run tests with pytest
run: |
pdm run pytest -v -s --durations=20 --doctest-modules --doctest-continue-on-failure quackosm
pdm run pytest -v -s --durations=20 tests/base
pdm run pytest -v -s --durations=20 tests/optional_imports
run-tests-oldest:
name: Run tests 🛠️ on multiple systems 🖥️ and Python 🐍 versions (oldest dependencies)
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-13, macos-latest, windows-latest]
python-version: ["3.9"]
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v3
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install pdm
run: pip install pdm
- name: Generate lock with oldest dependencies
run: pdm lock --lockfile pdm.oldest.lock --strategy no_cross_platform,direct_minimal_versions -dG:all
- name: Install srai and tests dependencies
run: pdm install --lockfile pdm.oldest.lock -dG:all --skip=post_install
- name: Run tests with pytest
run: |
pdm run pytest -v -s --durations=20 tests/base
pdm run pytest -v -s --durations=20 tests/optional_imports
53 changes: 53 additions & 0 deletions .github/workflows/update_usage_plots.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: "Update resources usage plots 📈 and create PR"
on: workflow_dispatch

env:
PYTHON_VERSION: 3.12

jobs:
bump-n-pr:
name: Update resources usage plots 📈 and create PR
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.CICD_PAT_TOKEN }}
- name: Configure Git user
run: |
git config --local user.name "Kraina CI/CD"
git config --local user.email "cicd@kraina.ai"
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
- uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-dev-${{ hashFiles('**/pdm.lock') }}
restore-keys: |
${{ runner.os }}-pip-dev-
- name: Install pdm
run: pip install pdm
- name: Generate requirements.txt
run: pdm export --no-default -G docs -G visualization -G cli-dev -f requirements -o requirements.txt
- name: Install dependencies
run: pip install --no-deps -r requirements.txt
- name: Install quackosm
run: |
pdm build -v -d dist
pip install 'quackosm[cli] @ file://'"$(pwd)/$(find dist -name '*.whl')" --user
- name: Execute jupyter notebooks
run: jupyter nbconvert --to notebook --inplace --execute $(find dev/ -type f -name "*.ipynb") --ExecutePreprocessor.kernel_name='python3'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
with:
title: "chore: update resources usage plots"
branch: update-resources-usage-plots
base: main
labels: Skip-Changelog
add-paths: docs/assets/images/*.png
token: ${{ secrets.CICD_PAT_TOKEN }}
author: Kraina CI/CD <cicd@kraina.ai>
committer: Kraina CI/CD <cicd@kraina.ai>
commit-message: "chore: update resources usage plots"
78 changes: 77 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- Added new internal parquet dataset processing logic using multiprocessing
- Refactored nodes intersection step from `ST_Intersects` in DuckDB to Shapely's `STRtree`
- `PbfFileReader`'s internal `geometry_filter` is additionally clipped by PBF extract geometry to speed up intersections
- `OsmTagsFilter` and `GroupedOsmTagsFilter` type from `dict` to `Mapping` to make it covariant

### Added

- `geoarrow-rust-core` library to the main dependencies
- Test for hashing geometry filter with mixed order
- Test for parquet multiprocessing logic
- Test for new intersection step

## [0.8.1] - 2024-05-11

### Added

- Option to convert multiple `*.osm.pbf` files to a single `parquet` file

### Changed

- Names of the functions have been unified to all start with `convert_` prefix
- Simplified internal conversion API

### Deprecated

- Functions `convert_pbf_to_gpq`, `convert_geometry_to_gpq`/`convert_geometry_filter_to_gpq`, `get_features_gdf` and `get_features_gdf_from_geometry` have been deprecated in favor of `convert_pbf_to_parquet`, `convert_geometry_to_parquet`, `convert_pbf_to_geodataframe` and `convert_geometry_to_geodataframe`
- Parameter `file_paths` has been replaced with `pbf_path`

### Fixed

- Removed the `parquet` extension installation step after opening the DuckDB connection

## [0.8.0] - 2024-05-08

### Added

- Polars library to the main dependencies

### Changed

- Refactored ways grouping logic from duckdb to polars `LazyFrame` API for faster operations
- Default result file extension from `geoparquet` to `parquet` [#99](https://github.com/kraina-ai/quackosm/issues/99)
- Moved `rich` to the main dependencies [#95](https://github.com/kraina-ai/quackosm/issues/95)
- Set minimal versions of multiple dependencies
- Added tests for minimal dependencies versions

### Fixed

- Steps numbering after encountering `MemoryError`

### Removed

- `h3ronpy` from dependencies and replaced logic with pure `h3` calls

### Deprecated

- Reusing existing `geoparquet` files from cache will be supported, but will result in deprecation warning [#99](https://github.com/kraina-ai/quackosm/issues/99)

## [0.7.3] - 2024-05-07

### Added

- Debug mode that keeps all temporary files for further inspection, activated with `debug` flag

### Changed

- Refactored parsing native `LINESTRING_2D` types when reading them from saved parquet file

## [0.7.2] - 2024-04-28

### Changed
Expand Down Expand Up @@ -226,7 +296,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Created QuackOSM repository
- Implemented PbfFileReader

[Unreleased]: https://github.com/kraina-ai/quackosm/compare/0.7.2...HEAD
[Unreleased]: https://github.com/kraina-ai/quackosm/compare/0.8.1...HEAD

[0.8.1]: https://github.com/kraina-ai/quackosm/compare/0.8.0...0.8.1

[0.8.0]: https://github.com/kraina-ai/quackosm/compare/0.7.3...0.8.0

[0.7.3]: https://github.com/kraina-ai/quackosm/compare/0.7.2...0.7.3

[0.7.2]: https://github.com/kraina-ai/quackosm/compare/0.7.1...0.7.2

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ docs:
mkdocs serve --livereload -w quackosm

test:
pytest --durations=20 --doctest-modules --doctest-continue-on-failure srai
pytest --durations=20 --doctest-modules --doctest-continue-on-failure quackosm
pytest --durations=20 tests

.PHONY: install docs test
Loading

0 comments on commit 1d74978

Please sign in to comment.