fix: add `libc.so` fallback for musl systems to the ctypes impl by mgorny · Pull Request #725 · psycopg/psycopg

mgorny · 2024-01-27T14:33:00Z

Add a fallback to libc.so library name to fix loading the ctypes implementation on musl systems. On musl, find_library("c") does not work (the problem has been reported to CPython in 2014, and has not been resolved yet), causing the module to fail on assert libcname. Instead, add a fallback to using libc.so and let ctypes raise an exception if such a library does not exist.

thesamesam · 2024-01-28T08:54:02Z

For completeness, the Gentoo bug is https://bugs.gentoo.org/923003 and the upstream CPython bug is python/cpython#65821.

dvarrazzo · 2024-01-28T12:43:12Z

Hello,

does it mean that the pure Python version of psycopg just doesn't work on musl? I would like to see it tested in CI.

The change you provided has broken the tests. Are you able to provide something working?

mgorny · 2024-01-28T16:53:15Z

does it mean that the pure Python version of psycopg just doesn't work on musl?

Yes, or at least on vanilla musl+CPython. It is possible that some distributions are patching musl and/or CPython to make ctypes.util.find_library("c") work.

I would like to see it tested in CI.

I don't think GitHub Actions support any musl-backed setup.

The change you provided has broken the tests. Are you able to provide something working?

Could you be more specific? Give that master is broken right now and how many different workflows are being run, it's hard for me to find the specific job that I broke.

I'll update the code per the suggestion.

thesamesam · 2024-01-28T16:56:11Z

I would like to see it tested in CI.

I don't think GitHub Actions support any musl-backed setup.

I think you can ask Github Actions to use an arbitrary Docker container, but then some action-stuff like the Python setup may not work, see e.g. https://github.com/Jannik2099/pms-utils/blob/main/.github/workflows/alpine.yml.

But Alpine patches their musl, so this bug wouldn't have shown up there anyway. We'd need to do a Gentoo image. If @dvarrazzo would like me to spend time on that, I can look into it. But it won't be super fast either (I can try my best to make it not-slow, but I can't guarantee it'll only be <5 minutes or something).

dvarrazzo · 2024-01-28T18:57:26Z

Give[n] that master is broken right now

Ah, I didn't notice that the world is on fire. Master was working as of yesterday. I'll check what's up with it.

dvarrazzo · 2024-01-28T18:58:54Z

But Alpine patches their musl, so this bug wouldn't have shown up there anyway.

I understand, I identified musl == alpine but of course they are two different things.

Nevermind then, I don't want to add other element to the test matrix, it's already slow enough as it is.

Add a fallback to `libc.so` library name to fix loading the ctypes implementation on musl systems. On musl, `find_library("c")` does not work (the problem has been reported to CPython in 2014, and has not been resolved yet), causing the module to fail on `assert libcname`. Instead, add a fallback to using `libc.so` and let ctypes raise an exception if such a library does not exist.

dvarrazzo · 2024-01-29T00:28:35Z

I have rebased your branch after fixing master, and added a release note entry. As soon as tests pass I will merge your PR and will release 3.1.18 including this fix in the next few days.

Thank you for your contribution!

mgorny · 2024-01-29T05:09:09Z

Thanks a lot!

Bumps [psycopg](https://github.com/psycopg/psycopg) from 3.1.15 to 3.1.18. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psycopg/psycopg/blob/master/docs/news.rst">psycopg's changelog</a>.</em></p> <blockquote> <p>.. currentmodule:: psycopg</p> <p>.. index:: single: Release notes single: News</p> <h1><code>psycopg</code> release notes</h1> <h2>Future releases</h2> <p>Psycopg 3.2 (unreleased) ^^^^^^^^^^^^^^^^^^^^^^^^</p> <ul> <li>Add support for integer, floating point, boolean <code>NumPy scalar types</code>__ (:ticket:<code>[#332](https://github.com/psycopg/psycopg/issues/332)</code>).</li> <li>Add <code>!timeout</code> and <code>!stop_after</code> parameters to <code>Connection.notifies()</code> (:ticket:<code>340</code>).</li> <li>Add :ref:<code>raw-query-cursors</code> to execute queries using placeholders in PostgreSQL format (<code>$1</code>, <code>$2</code>...) (:ticket:<code>[#560](https://github.com/psycopg/psycopg/issues/560)</code>).</li> <li>Add <code>~rows.scalar_row</code> to return scalar values from a query (:ticket:<code>[#723](https://github.com/psycopg/psycopg/issues/723)</code>).</li> <li>Add <code>~Connection.set_autocommit()</code> on sync connections, and similar transaction control methods available on the async connections.</li> <li>Add support for libpq functions to close prepared statements and portals introduced in libpq v17 (:ticket:<code>[#603](https://github.com/psycopg/psycopg/issues/603)</code>).</li> <li>The <code>!context</code> parameter of <code>sql</code> objects <code>~sql.Composable.as_string()</code> and <code>~sql.Composable.as_bytes()</code> methods is now optional (:ticket:<code>[#716](https://github.com/psycopg/psycopg/issues/716)</code>).</li> <li>Disable receiving more than one result on the same cursor in pipeline mode, to iterate through <code>~Cursor.nextset()</code>. The behaviour was different than in non-pipeline mode and not totally reliable (:ticket:<code>[#604](https://github.com/psycopg/psycopg/issues/604)</code>). The <code>Cursor</code> now only preserves the results set of the last <code>~Cursor.execute()</code>, consistently with non-pipeline mode.</li> </ul> <p>.. __: <a href="https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types">https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types</a></p> <h2>Current release</h2> <p>Psycopg 3.1.18 ^^^^^^^^^^^^^^</p> <ul> <li>Fix possible deadlock on pipeline exit (:ticket:<code>[#685](https://github.com/psycopg/psycopg/issues/685)</code>).</li> <li>Fix overflow loading large intervals in C module (:ticket:<code>[#719](https://github.com/psycopg/psycopg/issues/719)</code>).</li> <li>Fix compatibility with musl libc distributions affected by <code>CPython issue [#65821](https://github.com/psycopg/psycopg/issues/65821)</code>__ (:ticket:<code>[#725](https://github.com/psycopg/psycopg/issues/725)</code>).</li> </ul> <p>.. __: <a href="https://redirect.github.com/python/cpython/issues/65821">python/cpython#65821</a></p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psycopg/psycopg/commit/8585a23fcd7bcf75193adbc10d3005752ba8f15f"><code>8585a23</code></a> chore: bump psycopg package version to 3.1.18</li> <li><a href="https://github.com/psycopg/psycopg/commit/ab646b70c82aafe6004064a40a3ba358142999a3"><code>ab646b7</code></a> fix(c): drop spurious loop break in pipeline_communicate</li> <li><a href="https://github.com/psycopg/psycopg/commit/bebfe97f934c9136e4db52709ac0fb4dd9cae64d"><code>bebfe97</code></a> chore: bump cibuildwheel version</li> <li><a href="https://github.com/psycopg/psycopg/commit/89394a6f36d42d308a8e672e9b5deef8e76254ae"><code>89394a6</code></a> chore: bump checkout action to v4</li> <li><a href="https://github.com/psycopg/psycopg/commit/ed579e51ca9b44af148e55d345e312f58ce12a6f"><code>ed579e5</code></a> docs: fix tickets format</li> <li><a href="https://github.com/psycopg/psycopg/commit/d4a4e8e1447de3446f614a29a8274ef7c4d03d64"><code>d4a4e8e</code></a> Merge branch 'musl-ctypes' into maint-3.1</li> <li><a href="https://github.com/psycopg/psycopg/commit/8bc51e6812cfaedebdd7afff7c86be301d5fbf66"><code>8bc51e6</code></a> docs: mention musl-ctypes workaround in news file</li> <li><a href="https://github.com/psycopg/psycopg/commit/afb040a800b2667a07dc441e8cdb94e55a0dcf65"><code>afb040a</code></a> fix: add <code>libc.so</code> fallback for musl systems to the ctypes impl</li> <li><a href="https://github.com/psycopg/psycopg/commit/06ef0d92109a63fa1a7630804a3a26af0e0a39c9"><code>06ef0d9</code></a> test: drop ineffective marker on fixture</li> <li><a href="https://github.com/psycopg/psycopg/commit/b955118e523c84f5f702d93fd74288ce51ff61db"><code>b955118</code></a> Merge branch 'fix-interval-overflow' into maint-3.1</li> <li>Additional commits viewable in <a href="https://github.com/psycopg/psycopg/compare/3.1.15...3.1.18">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=psycopg&package-manager=pip&previous-version=3.1.15&new-version=3.1.18)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>

Bumps [psycopg](https://github.com/psycopg/psycopg) from 3.1.15 to 3.1.18. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psycopg/psycopg/blob/master/docs/news.rst">psycopg's changelog</a>.</em></p> <blockquote> <p>.. currentmodule:: psycopg</p> <p>.. index:: single: Release notes single: News</p> <h1><code>psycopg</code> release notes</h1> <h2>Future releases</h2> <p>Psycopg 3.2 (unreleased) ^^^^^^^^^^^^^^^^^^^^^^^^</p> <ul> <li>Add support for integer, floating point, boolean <code>NumPy scalar types</code>__ (:ticket:<code>[microsoft#332](https://github.com/psycopg/psycopg/issues/332)</code>).</li> <li>Add <code>!timeout</code> and <code>!stop_after</code> parameters to <code>Connection.notifies()</code> (:ticket:<code>340</code>).</li> <li>Add :ref:<code>raw-query-cursors</code> to execute queries using placeholders in PostgreSQL format (<code>$1</code>, <code>$2</code>...) (:ticket:<code>[microsoft#560](https://github.com/psycopg/psycopg/issues/560)</code>).</li> <li>Add <code>~rows.scalar_row</code> to return scalar values from a query (:ticket:<code>[microsoft#723](https://github.com/psycopg/psycopg/issues/723)</code>).</li> <li>Add <code>~Connection.set_autocommit()</code> on sync connections, and similar transaction control methods available on the async connections.</li> <li>Add support for libpq functions to close prepared statements and portals introduced in libpq v17 (:ticket:<code>[microsoft#603](https://github.com/psycopg/psycopg/issues/603)</code>).</li> <li>The <code>!context</code> parameter of <code>sql</code> objects <code>~sql.Composable.as_string()</code> and <code>~sql.Composable.as_bytes()</code> methods is now optional (:ticket:<code>[microsoft#716](https://github.com/psycopg/psycopg/issues/716)</code>).</li> <li>Disable receiving more than one result on the same cursor in pipeline mode, to iterate through <code>~Cursor.nextset()</code>. The behaviour was different than in non-pipeline mode and not totally reliable (:ticket:<code>[microsoft#604](https://github.com/psycopg/psycopg/issues/604)</code>). The <code>Cursor</code> now only preserves the results set of the last <code>~Cursor.execute()</code>, consistently with non-pipeline mode.</li> </ul> <p>.. __: <a href="https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types">https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types</a></p> <h2>Current release</h2> <p>Psycopg 3.1.18 ^^^^^^^^^^^^^^</p> <ul> <li>Fix possible deadlock on pipeline exit (:ticket:<code>[microsoft#685](https://github.com/psycopg/psycopg/issues/685)</code>).</li> <li>Fix overflow loading large intervals in C module (:ticket:<code>[microsoft#719](https://github.com/psycopg/psycopg/issues/719)</code>).</li> <li>Fix compatibility with musl libc distributions affected by <code>CPython issue [#65821](https://github.com/psycopg/psycopg/issues/65821)</code>__ (:ticket:<code>[microsoft#725](https://github.com/psycopg/psycopg/issues/725)</code>).</li> </ul> <p>.. __: <a href="https://redirect.github.com/python/cpython/issues/65821">python/cpython#65821</a></p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psycopg/psycopg/commit/8585a23fcd7bcf75193adbc10d3005752ba8f15f"><code>8585a23</code></a> chore: bump psycopg package version to 3.1.18</li> <li><a href="https://github.com/psycopg/psycopg/commit/ab646b70c82aafe6004064a40a3ba358142999a3"><code>ab646b7</code></a> fix(c): drop spurious loop break in pipeline_communicate</li> <li><a href="https://github.com/psycopg/psycopg/commit/bebfe97f934c9136e4db52709ac0fb4dd9cae64d"><code>bebfe97</code></a> chore: bump cibuildwheel version</li> <li><a href="https://github.com/psycopg/psycopg/commit/89394a6f36d42d308a8e672e9b5deef8e76254ae"><code>89394a6</code></a> chore: bump checkout action to v4</li> <li><a href="https://github.com/psycopg/psycopg/commit/ed579e51ca9b44af148e55d345e312f58ce12a6f"><code>ed579e5</code></a> docs: fix tickets format</li> <li><a href="https://github.com/psycopg/psycopg/commit/d4a4e8e1447de3446f614a29a8274ef7c4d03d64"><code>d4a4e8e</code></a> Merge branch 'musl-ctypes' into maint-3.1</li> <li><a href="https://github.com/psycopg/psycopg/commit/8bc51e6812cfaedebdd7afff7c86be301d5fbf66"><code>8bc51e6</code></a> docs: mention musl-ctypes workaround in news file</li> <li><a href="https://github.com/psycopg/psycopg/commit/afb040a800b2667a07dc441e8cdb94e55a0dcf65"><code>afb040a</code></a> fix: add <code>libc.so</code> fallback for musl systems to the ctypes impl</li> <li><a href="https://github.com/psycopg/psycopg/commit/06ef0d92109a63fa1a7630804a3a26af0e0a39c9"><code>06ef0d9</code></a> test: drop ineffective marker on fixture</li> <li><a href="https://github.com/psycopg/psycopg/commit/b955118e523c84f5f702d93fd74288ce51ff61db"><code>b955118</code></a> Merge branch 'fix-interval-overflow' into maint-3.1</li> <li>Additional commits viewable in <a href="https://github.com/psycopg/psycopg/compare/3.1.15...3.1.18">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=psycopg&package-manager=pip&previous-version=3.1.15&new-version=3.1.18)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>

dvarrazzo requested changes Jan 28, 2024

View reviewed changes

Comment thread psycopg/psycopg/pq/_pq_ctypes.py Outdated

mgorny force-pushed the musl-ctypes branch from eeb662b to 8402731 Compare January 28, 2024 16:57

dvarrazzo force-pushed the musl-ctypes branch from 8402731 to 46381be Compare January 29, 2024 00:21

docs: mention musl-ctypes workaround in news file

0b6aacc

dvarrazzo merged commit 26afb15 into psycopg:master Jan 29, 2024

mgorny deleted the musl-ctypes branch January 29, 2024 05:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: add `libc.so` fallback for musl systems to the ctypes impl#725

fix: add `libc.so` fallback for musl systems to the ctypes impl#725
dvarrazzo merged 2 commits into
psycopg:masterfrom
mgorny:musl-ctypes

mgorny commented Jan 27, 2024

Uh oh!

thesamesam commented Jan 28, 2024

Uh oh!

dvarrazzo commented Jan 28, 2024

Uh oh!

Uh oh!

mgorny commented Jan 28, 2024

Uh oh!

thesamesam commented Jan 28, 2024 •

edited

Loading

Uh oh!

dvarrazzo commented Jan 28, 2024 •

edited

Loading

Uh oh!

dvarrazzo commented Jan 28, 2024

Uh oh!

dvarrazzo commented Jan 29, 2024

Uh oh!

mgorny commented Jan 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mgorny commented Jan 27, 2024

Uh oh!

thesamesam commented Jan 28, 2024

Uh oh!

dvarrazzo commented Jan 28, 2024

Uh oh!

Uh oh!

mgorny commented Jan 28, 2024

Uh oh!

thesamesam commented Jan 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dvarrazzo commented Jan 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dvarrazzo commented Jan 28, 2024

Uh oh!

dvarrazzo commented Jan 29, 2024

Uh oh!

mgorny commented Jan 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thesamesam commented Jan 28, 2024 •

edited

Loading

dvarrazzo commented Jan 28, 2024 •

edited

Loading