Merge dev into main for 2023-11-16 #3054
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
zaneselvans
commented
Nov 16, 2023
zaneselvans
commented
- Check if package metadata matches Alembic migrations
- Update docs
- Fix XBRL extraction clobber (Fix XBRL extraction clobber #3026)
- Update black requirement from <23.11,>=23 to >=23,<23.12
- Update dask requirement from <2023.10.2,>=2022.5 to >=2022.5,<2023.11.1
- Update dagster-postgres requirement
- [pre-commit.ci] pre-commit autoupdate
- Always use tmp path for clobber tests.
- Also use tmp path in test_xbrl2sqlite
- Set up reproducible Python environments with conda-lock (Set up reproducible Python environments with conda-lock #2968)
- Update conda-lock.yml and rendered conda environment files.
- Add EIA860 2022 final release data
- Add unmapped utility and plant ids and update mcoe row count
- Update conda-lock.yml and rendered conda environment files.
- Update conda-lock.yml and rendered conda environment files.
- Eia861 2022 final release (Eia861 2022 final release #3048)
- Update expected row counts for service territory geometry tables.
Test that DB schema matches the Alembic migrations
Clobber once per form, instead of once per form-year.
Updates the requirements on [black](https://github.com/psf/black) to permit the latest version. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](psf/black@23.1a1...23.11.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Updates the requirements on [dask](https://github.com/dask/dask) to permit the latest version. - [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md) - [Commits](dask/dask@2022.05.0...2023.11.0) --- updated-dependencies: - dependency-name: dask dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…/dask-gte-2022.5-and-lt-2023.11.1 Update dask requirement from <2023.10.2,>=2022.5 to >=2022.5,<2023.11.1
Updates the requirements on [dagster-postgres](https://github.com/dagster-io/dagster) to permit the latest version. - [Release notes](https://github.com/dagster-io/dagster/releases) - [Changelog](https://github.com/dagster-io/dagster/blob/master/CHANGES.md) - [Commits](https://github.com/dagster-io/dagster/commits) --- updated-dependencies: - dependency-name: dagster-postgres dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…/dagster-postgres-gte-0.21.5-and-lt-0.21.8 Update dagster-postgres requirement from <0.21.7,>=0.21.5 to >=0.21.5,<0.21.8
…/black-gte-23-and-lt-23.12 Update black requirement from <23.11,>=23 to >=23,<23.12
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.4 → v0.1.5](astral-sh/ruff-pre-commit@v0.1.4...v0.1.5) - [github.com/psf/black-pre-commit-mirror: 23.10.1 → 23.11.0](psf/black-pre-commit-mirror@23.10.1...23.11.0) - [github.com/pre-commit/mirrors-prettier: v3.0.3 → v3.1.0](pre-commit/mirrors-prettier@v3.0.3...v3.1.0)
…te-config [pre-commit.ci] pre-commit autoupdate
…ive-dbs-test Merging this in since Ben approved it... in the hopes that we'll get a fully functional deployment tonight.
# Set up reproducible Python environments with conda-lock (#2901) * Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Switch to using conda-forge version of recordlinkage v0.16 * Update conda-lock.yml now that all dependencies are available on conda-forge * Consolidate conda env files under environments/ dir * Add a GitHub action to relock dependencies * Quote the pip install command * Remove pip install of pudl from environment.yml * Rename workflow * Only build lockfile from pyproject.toml, don't install extras. * Just install conda-lock, not pudl, before running conda-lock. * install conda-lock with pip * Move all remaining dev-environment.yml deps to pyproject.toml * Add other platforms; make draft PR against dev. * Comment out dev base branch for now. * Remove pandas extras and recordlinkage deps from pyproject.toml * Use conda-lock --micromamba rather than --mamba * Don't specify grpcio, or specific recordlinkage version * Render platform-specific environment files in github action * Fix paths relative to environments directory * Add some comment notes to workflow * Render environment for Read The Docs. * Use environment not explicit rendered lockfile * Add readthedocs specific sphinx extension * Don't render explicit conda env for RTD since it can't read it. * Build linux-aarch64 lockfile. Use conda-lock.yml in workflows. * Comment out non-working linux-aarch64 platform for now. * Switch to using rendered lockfiles. * Remove deprecated environment files * Switch to using a micromamba docker image * Install git into the docker image. * Use micromamba and unrendered multi-platform lockfile. * Add main category to micromamba environment creation. * Use conda-lock not base as env name * Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Add linux-aarch64 platform back into conda-lock settings. ## Move tox.ini and update-lockfile logic to Makefile - Remove deprecated API_KEY_EIA envvar - Add .gitignore in new environments directory - Create Makefile and migrat tox.ini logic into it - Replace spaces in Makefile with tabs - Install pudl via pip install --no-deps to avoid contaminating the locked environment - Move pip install and coverage XML logic from GHA workflow into the Makefile - Increase the minimum versions of some dependencies. - Move update-lockfile GHA workflow logic into Makefile - Attempt to run slow tests in parallel using "wait" to prevent premature exit of the shell ## Adjust Dockerfile & deployment use micromamba * Use micromamba not conda in Dockerfile CMD, also use pip install --no-deps * Use micromamba not conda in command passed to build container * Use default mambauser rather than catalyst in docker container * Remove --no-capture-output which isn't supported by micromamba. Is this a problem? * Remove uninterpolated vars in .env and more --no-capture-output * Separate ETL and pytest commands. * Stop trying to run tests in parallel. Sigh. * Add google cloud sdk to conda environment. * Install Google Cloud SDK from conda-forge. * Add back in the making of required directories. Oops. * Attempt to have micromamba run pass through output * Use prettier to standardize formatting of lockfiles. * Add dagster (server startup) target to Makefile * Update conda lockfile and rerender environment files * Attempt to trigger update-lockfile when pyproject.toml is changed. * Remove non-required upper bounds on dependency versions. * Use correct branch name for update-lockfile trigger. * Fix incorrect nbformat minimum version * Update Makefile in response to PR feedback ## Set up triggers to automate updating of conda lockfiles * Remove dagster-postgres version to avoid PyPI conda-forge conflict. * Break up and distribute the nuke target. * Resolve issues with pandera dependency. Reorganize makefile. * Add triggers and commit/PR for workflow_dispatch, pull_request, schedule and set appropriate GITHUB_REF values for each case. * Use push instead of pull_request to trigger on path. This avoids re-locking the dependencies every single time you push to a PR that had a change to pyproject.toml *somewhere* in it. * Also trigger based on path if .github/workflows/update-lockfile.yml changes. * Update conda-lock.yml and rendered conda environment files. ## Consolidate optional dependencies; test nuke; parallelize tests * Move previous dev, test, and datasette optional dependencies into the required dependencies to simplify application installation. * Test make nuke; parallelize --live-dbs tests * Move prettier into conda-only dependencies * Update conda-lock.yml and rendered conda environment files. * Remove action test file hashlog * Remove merge markers. * Remove transitive astroid dependency that's now correctly included in solve. * Use the real immature library version of dagster-postgres (0.21.6) rather than the accidentally packaged 1.5.6 version found in conda. We'll need to keep an eye out for when dagster-postgres graduates to the stable versioning and update it. This is a bit of a mess because of some broken automation in the conda packaging for dagster which has now been fixed. * Update "make pudl" to remove the old PUDL DB and reinitialize with alembic, rather than writing to the DB that already exists. * Fixed some groupby.agg() deprecation warnings. * Fix dagster-postgres version (again). * Update username in path to settings file * Avoid bugs in ferc_to_sqlite --clobber; don't use cache_dir for pip install. * Make FERC extraction output removal more specific. * Bump numpy and numba minimum versions. * Bump black version in pre-commit * Bump ruff pre-commit hook version ## Update docs to reflect swapping Make in for Tox, application packaging. * Rename tox-pytest and update-lockfile workflows. * Make scheduled PRs against dev rather than conda-lockfile branch. * Update to pyarrow 14 and grpcio 1.59 b/c security * Update release notes * Add CI targets in Makefile. Docs cleanup. * Update conda-lock.yml and rendered conda environment files. --------- Co-authored-by: zaneselvans <zaneselvans@users.noreply.github.com>
…release Add EIA860 2022 final release data
* Add EIA860 2022 final release data * Add unmapped utility and plant ids and update mcoe row count * Add EIA861 2022 final release data * Update conda-lock.yml and rendered conda environment files. --------- Co-authored-by: aesharpe <aesharpe@users.noreply.github.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3054 +/- ##
=====================================
Coverage 88.7% 88.7%
=====================================
Files 90 90
Lines 10988 10994 +6
=====================================
+ Hits 9752 9758 +6
Misses 1236 1236 ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.