Add check to ensure locked dependencies have source distributions available. #14742

reivilibre · 2022-12-28T14:28:27Z

It'd be nice if it was obvious when the Poetry lockfile has not been updated in a safe way.
Crucially, all dependencies need an sdist or else we won't be able to install them everywhere.

Follows: #14718

Base: develop

Original commit schedule, with full messages:

Add a script to check the lockfile for dependencies without sdists.
Add dependency on tomli
Add workflow to ensure locked deps have sdists

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

DMRobertson

PTAL at

Permissions on the action
Unexpected guff in the lockfile

the other comments are just me thinking out loud.

poetry.lock

pyproject.toml

.github/workflows/poetry_lockfile.yaml

DMRobertson · 2023-01-09T12:35:43Z

scripts-dev/check_locked_deps_have_sdists.py

+        # or a `files` inline table in each [[package]]
+        packages_to_assets = {
+            package["name"]: package["files"] for package in lockfile_content["package"]
+        }


Seems to be part of the poetry 1.3 lockfile format, see python-poetry/poetry#6393

Note that I'm about to require the new lockfile format in #14860

Might one day incorporate other checks like #14742

clokep · 2023-01-17T15:43:07Z

Crucially, all dependencies need an sdist or else we won't be able to install them everywhere.

Is it correct that "everywhere" means:

If you use a Python distribution that doesn't have wheels, e.g. Pyston or brand new Pythons (Python 3.11)
If you run on hardware that doesn't have wheels for some things (ARM? Raspberry Pi?)

I'm sure there are other examples too.

Might one day incorporate other checks like #14742

* Upgrade to new lockfile format Now requires poetry >= 1.2.2 to read and poetry >= 1.3.0 to write. Cheat sheet: ``` poetry --version poetry show > scratch/before pipx upgrade poetry poetry --version poetry show > scratch/after diff scratch{before,after} && echo "no change!" ``` * Use Poetry 1.3.2 when reading or writing lockfile * Remove unneeded(?) poetry dep for cibuildwheel * Update docs * Remove redundant call to setup-python * Remove outdated comments related to Poetry 1.x * Remove outdated docs line was fixed in #13082 * Minor improvements to poetry cheat sheet * Invoke setup-python-poetry with explicit version Not sure about this. It's hardcoding versions everywhere. * Changelog * Check the lockfile is version 2.0 Might one day incorporate other checks like #14742 * Typo fixes, thanks Sean Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

reivilibre · 2023-02-13T16:00:42Z

Crucially, all dependencies need an sdist or else we won't be able to install them everywhere.

Is it correct that "everywhere" means:
* If you use a Python distribution that doesn't have wheels, e.g. Pyston or brand new Pythons (Python 3.11)

* If you run on hardware that doesn't have wheels for some things (ARM? Raspberry Pi?)
I'm sure there are other examples too.

Yeah, both of those would apply. Note that 'Brand new Pythons' can still be OK with wheels that predate them if the wheels are targetted at ABI3, but different CPU architectures is a notable one. (Also, some distributions might refuse to use, or be unable to use, wheels.)

This partially reverts commit 4bb82b4.

Probably cargo culted from another workflow

DMRobertson · 2023-02-13T18:06:09Z

.github/workflows/poetry_lockfile.yaml

+on:
+  push:
+    branches: ["develop", "release-*"]
+    paths:
+      - poetry.lock
+  pull_request:
+    paths:
+      - poetry.lock


Nice that we only trigger this if needed.

The downside is that it can't be fed into the tests-done job to tell github it's safe to merge this PR. But I think we can live with that.

It's frustrating how the default behaviour isn't just to make sure all the tests pass, but ah well

@thezaidbintariq

Synapse 1.78.0 (2023-02-28) =========================== Bugfixes -------- - Fix a bug introduced in Synapse 1.76 where 5s delays would occasionally occur in deployments using workers. ([\#15150](matrix-org/synapse#15150)) Synapse 1.78.0rc1 (2023-02-21) ============================== Features -------- - Implement the experimental `exact_event_match` push rule condition from [MSC3758](matrix-org/matrix-spec-proposals#3758). ([\#14964](matrix-org/synapse#14964)) - Add account data to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.78/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\#14969](matrix-org/synapse#14969)) - Implement [MSC3873](matrix-org/matrix-spec-proposals#3873) to disambiguate push rule keys with dots in them. ([\#15004](matrix-org/synapse#15004)) - Allow Synapse to use a specific Redis [logical database](https://redis.io/commands/select/) in worker-mode deployments. ([\#15034](matrix-org/synapse#15034)) - Tag opentracing spans for federation requests with the name of the worker serving the request. ([\#15042](matrix-org/synapse#15042)) - Implement the experimental `exact_event_property_contains` push rule condition from [MSC3966](matrix-org/matrix-spec-proposals#3966). ([\#15045](matrix-org/synapse#15045)) - Remove spurious `dont_notify` action from the defaults for the `.m.rule.reaction` pushrule. ([\#15073](matrix-org/synapse#15073)) - Update the error code returned when user sends a duplicate annotation. ([\#15075](matrix-org/synapse#15075)) Bugfixes -------- - Prevent clients from reporting nonexistent events. ([\#13779](matrix-org/synapse#13779)) - Return spec-compliant JSON errors when unknown endpoints are requested. ([\#14605](matrix-org/synapse#14605)) - Fix a long-standing bug where the room aliases returned could be corrupted. ([\#15038](matrix-org/synapse#15038)) - Fix a bug introduced in Synapse 1.76.0 where partially-joined rooms could not be deleted using the [purge room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api). ([\#15068](matrix-org/synapse#15068)) - Fix a long-standing bug where federated joins would fail if the first server in the list of servers to try is not in the room. ([\#15074](matrix-org/synapse#15074)) - Fix a bug introduced in Synapse v1.74.0 where searching with colons when using ICU for search term tokenisation would fail with an error. ([\#15079](matrix-org/synapse#15079)) - Reduce the likelihood of a rare race condition where rejoining a restricted room over federation would fail. ([\#15080](matrix-org/synapse#15080)) - Fix a bug introduced in Synapse 1.76 where workers would fail to start if the `health` listener was configured. ([\#15096](matrix-org/synapse#15096)) - Fix a bug introduced in Synapse 1.75 where the [portdb script](https://matrix-org.github.io/synapse/release-v1.78/postgres.html#porting-from-sqlite) would fail to run after a room had been faster-joined. ([\#15108](matrix-org/synapse#15108)) Improved Documentation ---------------------- - Document how to start Synapse with Poetry. Contributed by @thezaidbintariq. ([\#14892](matrix-org/synapse#14892), [\#15022](matrix-org/synapse#15022)) - Update delegation documentation to clarify that SRV DNS delegation does not eliminate all needs to serve files from .well-known locations. Contributed by @williamkray. ([\#14959](matrix-org/synapse#14959)) - Fix a mistake in registration_shared_secret_path docs. ([\#15078](matrix-org/synapse#15078)) - Refer to a more recent blog post on the [Database Maintenance Tools](https://matrix-org.github.io/synapse/latest/usage/administration/database_maintenance_tools.html) page. Contributed by @jahway603. ([\#15083](matrix-org/synapse#15083)) Internal Changes ---------------- - Re-type hint some collections as read-only. ([\#13755](matrix-org/synapse#13755)) - Faster joins: don't stall when another user joins during a partial-state room resync. ([\#14606](matrix-org/synapse#14606)) - Add a class `UnpersistedEventContext` to allow for the batching up of storing state groups. ([\#14675](matrix-org/synapse#14675)) - Add a check to ensure that locked dependencies have source distributions available. ([\#14742](matrix-org/synapse#14742)) - Tweak comment on `_is_local_room_accessible` as part of room visibility in `/hierarchy` to clarify the condition for a room being visible. ([\#14834](matrix-org/synapse#14834)) - Prevent `WARNING: there is already a transaction in progress` lines appearing in PostgreSQL's logs on some occasions. ([\#14840](matrix-org/synapse#14840)) - Use `StrCollection` to avoid potential bugs with `Collection[str]`. ([\#14929](matrix-org/synapse#14929)) - Improve performance of `/sync` in a few situations. ([\#14973](matrix-org/synapse#14973)) - Limit concurrent event creation for a room to avoid state resolution when sending bursts of events to a local room. ([\#14977](matrix-org/synapse#14977)) - Skip calculating unread push actions in /sync when enable_push is false. ([\#14980](matrix-org/synapse#14980)) - Add a schema dump symlinks inside `contrib`, to make it easier for IDEs to interrogate Synapse's database schema. ([\#14982](matrix-org/synapse#14982)) - Improve type hints. ([\#15008](matrix-org/synapse#15008), [\#15026](matrix-org/synapse#15026), [\#15027](matrix-org/synapse#15027), [\#15028](matrix-org/synapse#15028), [\#15031](matrix-org/synapse#15031), [\#15035](matrix-org/synapse#15035), [\#15052](matrix-org/synapse#15052), [\#15072](matrix-org/synapse#15072), [\#15084](matrix-org/synapse#15084)) - Update [MSC3952](matrix-org/matrix-spec-proposals#3952) support based on changes to the MSC. ([\#15037](matrix-org/synapse#15037)) - Avoid mutating a cached value in `get_user_devices_from_cache`. ([\#15040](matrix-org/synapse#15040)) - Fix a rare exception in logs on start up. ([\#15041](matrix-org/synapse#15041)) - Update pyo3-log to v0.8.1. ([\#15043](matrix-org/synapse#15043)) - Avoid mutating cached values in `_generate_sync_entry_for_account_data`. ([\#15047](matrix-org/synapse#15047)) - Refactor arguments of `try_unbind_threepid` and `_try_unbind_threepid_with_id_server` to not use dictionaries. ([\#15053](matrix-org/synapse#15053)) - Merge debug logging from the hotfixes branch. ([\#15054](matrix-org/synapse#15054)) - Faster joins: omit device list updates originating from partial state rooms in /sync responses without lazy loading of members enabled. ([\#15069](matrix-org/synapse#15069)) - Fix clashing database transaction name. ([\#15070](matrix-org/synapse#15070)) - Upper-bound frozendict dependency. This works around us being unable to test installing our wheels against Python 3.11 in CI. ([\#15114](matrix-org/synapse#15114)) - Tweak logging for when a worker waits for its view of a replication stream to catch up. ([\#15120](matrix-org/synapse#15120)) <details><summary>Locked dependency updates</summary> - Bump bleach from 5.0.1 to 6.0.0. ([\#15059](matrix-org/synapse#15059)) - Bump cryptography from 38.0.4 to 39.0.1. ([\#15020](matrix-org/synapse#15020)) - Bump ruff version from 0.0.230 to 0.0.237. ([\#15033](matrix-org/synapse#15033)) - Bump dtolnay/rust-toolchain from 9cd00a88a73addc8617065438eff914dd08d0955 to 25dc93b901a87e864900a8aec6c12e9aa794c0c3. ([\#15060](matrix-org/synapse#15060)) - Bump systemd-python from 234 to 235. ([\#15061](matrix-org/synapse#15061)) - Bump serde_json from 1.0.92 to 1.0.93. ([\#15062](matrix-org/synapse#15062)) - Bump types-requests from 2.28.11.8 to 2.28.11.12. ([\#15063](matrix-org/synapse#15063)) - Bump types-pillow from 9.4.0.5 to 9.4.0.10. ([\#15064](matrix-org/synapse#15064)) - Bump sentry-sdk from 1.13.0 to 1.15.0. ([\#15065](matrix-org/synapse#15065)) - Bump types-jsonschema from 4.17.0.3 to 4.17.0.5. ([\#15099](matrix-org/synapse#15099)) - Bump types-bleach from 5.0.3.1 to 6.0.0.0. ([\#15100](matrix-org/synapse#15100)) - Bump dtolnay/rust-toolchain from 25dc93b901a87e864900a8aec6c12e9aa794c0c3 to e12eda571dc9a5ee5d58eecf4738ec291c66f295. ([\#15101](matrix-org/synapse#15101)) - Bump dawidd6/action-download-artifact from 2.24.3 to 2.25.0. ([\#15102](matrix-org/synapse#15102)) - Bump types-pillow from 9.4.0.10 to 9.4.0.13. ([\#15104](matrix-org/synapse#15104)) - Bump types-setuptools from 67.1.0.0 to 67.3.0.1. ([\#15105](matrix-org/synapse#15105)) </details>

reivilibre added 2 commits December 28, 2022 14:09

Add a script to check the lockfile for dependencies without sdists.

0aa18a2

Add dependency on tomli

4bb82b4

reivilibre force-pushed the rei/lockdeps_have_sdists branch from 363dd84 to a6c1870 Compare December 28, 2022 15:28

reivilibre added 3 commits December 28, 2022 15:39

Add workflow to ensure locked deps have sdists

e1148cc

Newsfile

4644a90

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Support the other style of lockfile

3b2d6e2

reivilibre force-pushed the rei/lockdeps_have_sdists branch from a6c1870 to 3b2d6e2 Compare December 28, 2022 15:39

reivilibre marked this pull request as ready for review December 28, 2022 17:16

reivilibre requested a review from a team as a code owner December 28, 2022 17:16

DMRobertson suggested changes Jan 9, 2023

View reviewed changes

DMRobertson pushed a commit that referenced this pull request Jan 17, 2023

Check the lockfile is version 2.0

26dc5d1

Might one day incorporate other checks like #14742

DMRobertson pushed a commit that referenced this pull request Jan 17, 2023

Check the lockfile is version 2.0

f871c9c

Might one day incorporate other checks like #14742

DMRobertson pushed a commit that referenced this pull request Jan 17, 2023

Check the lockfile is version 2.0

7ab56ae

Might one day incorporate other checks like #14742

DMRobertson pushed a commit that referenced this pull request Jan 17, 2023

Check the lockfile is version 2.0

b6fddbe

Might one day incorporate other checks like #14742

DMRobertson pushed a commit that referenced this pull request Jan 17, 2023

Check the lockfile is version 2.0

9382733

Might one day incorporate other checks like #14742

reivilibre added 5 commits February 13, 2023 16:02

Revert lockfile changes from "Add dependency on tomli"

170abb7

This partially reverts commit 4bb82b4.

Merge branch 'develop' into rei/lockdeps_have_sdists

c02c919

Update the lockfile content hash only (needed to clear poetry caches)

33bab13

Remove write privileges

44847bf

Probably cargo culted from another workflow

Only bother with Poetry 1.3+ lockfiles

2e75a31

reivilibre requested a review from DMRobertson February 13, 2023 18:02

DMRobertson reviewed Feb 13, 2023

View reviewed changes

DMRobertson approved these changes Feb 13, 2023

View reviewed changes

reivilibre merged commit c0bf4c3 into develop Feb 13, 2023

reivilibre deleted the rei/lockdeps_have_sdists branch February 13, 2023 18:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add check to ensure locked dependencies have source distributions available. #14742

Add check to ensure locked dependencies have source distributions available. #14742

reivilibre commented Dec 28, 2022

DMRobertson left a comment

DMRobertson Jan 9, 2023

DMRobertson Jan 17, 2023 •

edited

clokep commented Jan 17, 2023

reivilibre commented Feb 13, 2023

DMRobertson Feb 13, 2023

reivilibre Feb 13, 2023

Add check to ensure locked dependencies have source distributions available. #14742

Add check to ensure locked dependencies have source distributions available. #14742

Conversation

reivilibre commented Dec 28, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

DMRobertson Jan 9, 2023

Choose a reason for hiding this comment

DMRobertson Jan 17, 2023 • edited

Choose a reason for hiding this comment

clokep commented Jan 17, 2023

reivilibre commented Feb 13, 2023

DMRobertson Feb 13, 2023

Choose a reason for hiding this comment

reivilibre Feb 13, 2023

Choose a reason for hiding this comment

DMRobertson Jan 17, 2023 •

edited