add a 'size' parameter to Cursor.stream() by dlax · Pull Request #794 · psycopg/psycopg

dlax · 2024-04-17T10:22:12Z

Based on #793

dvarrazzo · 2024-04-17T21:00:58Z

Heads up: I gave a forced push to the branch because github didn't detect that #793 was merged and the MR still showed two commits.

dvarrazzo

Just some nits, but otherwise looks good, thank you 🙂

psycopg/psycopg/_cursor_base.py

psycopg/psycopg/rows.py

psycopg/psycopg/cursor_async.py

docs/api/cursors.rst

dvarrazzo · 2024-06-12T17:27:23Z

I've rebased this to master and about to merge it.

This triggers results retrieval by chunks from the server, if > 1, leveraging the "chunked rows mode" from libpq 17. A new has_stream_chunked() capability is added.

Bumps [psycopg](https://github.com/psycopg/psycopg) from 3.1.19 to 3.2.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psycopg/psycopg/blob/master/docs/news.rst">psycopg's changelog</a>.</em></p> <blockquote> <p>.. currentmodule:: psycopg</p> <p>.. index:: single: Release notes single: News</p> <h1><code>psycopg</code> release notes</h1> <h2>Current release</h2> <p>Psycopg 3.2.1 ^^^^^^^^^^^^^</p> <ul> <li>Fix packaging metadata breaking <code>[c]</code>, <code>[binary]</code> dependencies (:ticket:<code>[#853](https://github.com/psycopg/psycopg/issues/853)</code>).</li> </ul> <h2>Psycopg 3.2</h2> <p>.. rubric:: New top-level features</p> <ul> <li>Add support for integer, floating point, boolean <code>NumPy scalar types</code>__ (:ticket:<code>[#332](https://github.com/psycopg/psycopg/issues/332)</code>).</li> <li>Add <code>!timeout</code> and <code>!stop_after</code> parameters to <code>Connection.notifies()</code> (:ticket:<code>340</code>).</li> <li>Allow dumpers to return <code>!None</code>, to be converted to NULL (:ticket:<code>[#377](https://github.com/psycopg/psycopg/issues/377)</code>).</li> <li>Add :ref:<code>raw-query-cursors</code> to execute queries using placeholders in PostgreSQL format (<code>$1</code>, <code>$2</code>...) (🎟️<code>[#560](psycopg/psycopg#560), [#839](https://github.com/psycopg/psycopg/issues/839)</code>).</li> <li>Add <code>capabilities</code> object to :ref:<code>inspect the libpq capabilities <capabilities></code> (🎫<code>[#772](https://github.com/psycopg/psycopg/issues/772)</code>).</li> <li>Add <code>~rows.scalar_row</code> to return scalar values from a query (:ticket:<code>[#723](https://github.com/psycopg/psycopg/issues/723)</code>).</li> <li>Add <code>~Connection.cancel_safe()</code> for encrypted and non-blocking cancellation when using libpq v17. Use such method internally to implement <code>!KeyboardInterrupt</code> and <code>~cursor.copy</code> termination (:ticket:<code>[#754](https://github.com/psycopg/psycopg/issues/754)</code>).</li> <li>The <code>!context</code> parameter of <code>sql</code> objects <code>~sql.Composable.as_string()</code> and <code>~sql.Composable.as_bytes()</code> methods is now optional (:ticket:<code>[#716](https://github.com/psycopg/psycopg/issues/716)</code>).</li> <li>Add <code>~Connection.set_autocommit()</code> on sync connections, and similar transaction control methods available on the async connections.</li> <li>Add a <code>size</code> parameter to <code>~Cursor.stream()</code> to enable results retrieval in chunks instead of row-by-row (:ticket:<code>[#794](https://github.com/psycopg/psycopg/issues/794)</code>).</li> </ul> <p>.. rubric:: New libpq wrapper features</p> <ul> <li>Add support for libpq functions to close prepared statements and portals introduced in libpq v17 (:ticket:<code>[#603](https://github.com/psycopg/psycopg/issues/603)</code>).</li> <li>Add support for libpq encrypted and non-blocking query cancellation functions introduced in libpq v17 (:ticket:<code>[#754](https://github.com/psycopg/psycopg/issues/754)</code>).</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psycopg/psycopg/commit/bb47d3944d1c65d9baf83808696aba1b2dfed9af"><code>bb47d39</code></a> chore: bump psycopg package version to 3.2.1</li> <li><a href="https://github.com/psycopg/psycopg/commit/55490a2d6b8344e3b49f6004c8e7afdc6d40c5fb"><code>55490a2</code></a> fix: fix versions in packaging metadata</li> <li><a href="https://github.com/psycopg/psycopg/commit/1cbc42a6759823e18d273e739cc64d7f994f53b2"><code>1cbc42a</code></a> docs: fix title level of major releases</li> <li><a href="https://github.com/psycopg/psycopg/commit/06a6e5e213577fca0fa1c6ca62def8fa5803663a"><code>06a6e5e</code></a> docs: mention dropping Python 3.7 in psycopg 3.2 release</li> <li><a href="https://github.com/psycopg/psycopg/commit/ea3735dccc5dfd20eb4ba5c4a3c52ec76d9aa6d0"><code>ea3735d</code></a> docs: better organization of the 3.2 release notes</li> <li><a href="https://github.com/psycopg/psycopg/commit/896eee2363d0bc25cc31ccf4189b295bb61deb40"><code>896eee2</code></a> chore: bump psycopg package version to 3.2.0</li> <li><a href="https://github.com/psycopg/psycopg/commit/2e2f4d7dd31cac67b790c293de70133373173ebb"><code>2e2f4d7</code></a> chore: bump psycopg package version to 3.1.20</li> <li><a href="https://github.com/psycopg/psycopg/commit/7369d3bb0280851832a395d6b3084de7b520e3a0"><code>7369d3b</code></a> Merge pull request <a href="https://redirect.github.com/psycopg/psycopg/issues/846">#846</a> from eli-schwartz/tomllib</li> <li><a href="https://github.com/psycopg/psycopg/commit/6672c708f1813978461aff6e5e240a96773ff873"><code>6672c70</code></a> style: shorter line in pyproject.toml</li> <li><a href="https://github.com/psycopg/psycopg/commit/a517bb4579d2b8caa4ddc1e81294d6026c8974d6"><code>a517bb4</code></a> build: avoid installing tomli on recent python</li> <li>Additional commits viewable in <a href="https://github.com/psycopg/psycopg/compare/3.1.19...3.2.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=psycopg&package-manager=pip&previous-version=3.1.19&new-version=3.2.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [psycopg](https://github.com/psycopg/psycopg) from 3.1.19 to 3.2.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psycopg/psycopg/blob/master/docs/news.rst">psycopg's changelog</a>.</em></p> <blockquote> <p>.. currentmodule:: psycopg</p> <p>.. index:: single: Release notes single: News</p> <h1><code>psycopg</code> release notes</h1> <h2>Current release</h2> <p>Psycopg 3.2.1 ^^^^^^^^^^^^^</p> <ul> <li>Fix packaging metadata breaking <code>[c]</code>, <code>[binary]</code> dependencies (:ticket:<code>[microsoft#853](https://github.com/psycopg/psycopg/issues/853)</code>).</li> </ul> <h2>Psycopg 3.2</h2> <p>.. rubric:: New top-level features</p> <ul> <li>Add support for integer, floating point, boolean <code>NumPy scalar types</code>__ (:ticket:<code>[microsoft#332](https://github.com/psycopg/psycopg/issues/332)</code>).</li> <li>Add <code>!timeout</code> and <code>!stop_after</code> parameters to <code>Connection.notifies()</code> (:ticket:<code>340</code>).</li> <li>Allow dumpers to return <code>!None</code>, to be converted to NULL (:ticket:<code>[microsoft#377](https://github.com/psycopg/psycopg/issues/377)</code>).</li> <li>Add :ref:<code>raw-query-cursors</code> to execute queries using placeholders in PostgreSQL format (<code>$1</code>, <code>$2</code>...) (🎟️<code>[microsoft#560](psycopg/psycopg#560), [microsoft#839](https://github.com/psycopg/psycopg/issues/839)</code>).</li> <li>Add <code>capabilities</code> object to :ref:<code>inspect the libpq capabilities <capabilities></code> (🎫<code>[microsoft#772](https://github.com/psycopg/psycopg/issues/772)</code>).</li> <li>Add <code>~rows.scalar_row</code> to return scalar values from a query (:ticket:<code>[microsoft#723](https://github.com/psycopg/psycopg/issues/723)</code>).</li> <li>Add <code>~Connection.cancel_safe()</code> for encrypted and non-blocking cancellation when using libpq v17. Use such method internally to implement <code>!KeyboardInterrupt</code> and <code>~cursor.copy</code> termination (:ticket:<code>[microsoft#754](https://github.com/psycopg/psycopg/issues/754)</code>).</li> <li>The <code>!context</code> parameter of <code>sql</code> objects <code>~sql.Composable.as_string()</code> and <code>~sql.Composable.as_bytes()</code> methods is now optional (:ticket:<code>[microsoft#716](https://github.com/psycopg/psycopg/issues/716)</code>).</li> <li>Add <code>~Connection.set_autocommit()</code> on sync connections, and similar transaction control methods available on the async connections.</li> <li>Add a <code>size</code> parameter to <code>~Cursor.stream()</code> to enable results retrieval in chunks instead of row-by-row (:ticket:<code>[microsoft#794](https://github.com/psycopg/psycopg/issues/794)</code>).</li> </ul> <p>.. rubric:: New libpq wrapper features</p> <ul> <li>Add support for libpq functions to close prepared statements and portals introduced in libpq v17 (:ticket:<code>[microsoft#603](https://github.com/psycopg/psycopg/issues/603)</code>).</li> <li>Add support for libpq encrypted and non-blocking query cancellation functions introduced in libpq v17 (:ticket:<code>[microsoft#754](https://github.com/psycopg/psycopg/issues/754)</code>).</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psycopg/psycopg/commit/bb47d3944d1c65d9baf83808696aba1b2dfed9af"><code>bb47d39</code></a> chore: bump psycopg package version to 3.2.1</li> <li><a href="https://github.com/psycopg/psycopg/commit/55490a2d6b8344e3b49f6004c8e7afdc6d40c5fb"><code>55490a2</code></a> fix: fix versions in packaging metadata</li> <li><a href="https://github.com/psycopg/psycopg/commit/1cbc42a6759823e18d273e739cc64d7f994f53b2"><code>1cbc42a</code></a> docs: fix title level of major releases</li> <li><a href="https://github.com/psycopg/psycopg/commit/06a6e5e213577fca0fa1c6ca62def8fa5803663a"><code>06a6e5e</code></a> docs: mention dropping Python 3.7 in psycopg 3.2 release</li> <li><a href="https://github.com/psycopg/psycopg/commit/ea3735dccc5dfd20eb4ba5c4a3c52ec76d9aa6d0"><code>ea3735d</code></a> docs: better organization of the 3.2 release notes</li> <li><a href="https://github.com/psycopg/psycopg/commit/896eee2363d0bc25cc31ccf4189b295bb61deb40"><code>896eee2</code></a> chore: bump psycopg package version to 3.2.0</li> <li><a href="https://github.com/psycopg/psycopg/commit/2e2f4d7dd31cac67b790c293de70133373173ebb"><code>2e2f4d7</code></a> chore: bump psycopg package version to 3.1.20</li> <li><a href="https://github.com/psycopg/psycopg/commit/7369d3bb0280851832a395d6b3084de7b520e3a0"><code>7369d3b</code></a> Merge pull request <a href="https://redirect.github.com/psycopg/psycopg/issues/846">#846</a> from eli-schwartz/tomllib</li> <li><a href="https://github.com/psycopg/psycopg/commit/6672c708f1813978461aff6e5e240a96773ff873"><code>6672c70</code></a> style: shorter line in pyproject.toml</li> <li><a href="https://github.com/psycopg/psycopg/commit/a517bb4579d2b8caa4ddc1e81294d6026c8974d6"><code>a517bb4</code></a> build: avoid installing tomli on recent python</li> <li>Additional commits viewable in <a href="https://github.com/psycopg/psycopg/compare/3.1.19...3.2.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=psycopg&package-manager=pip&previous-version=3.1.19&new-version=3.2.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

dlax force-pushed the cursor-stream-chunks branch from 430b5e5 to 09f6494 Compare April 17, 2024 11:31

dlax marked this pull request as ready for review April 17, 2024 12:03

dlax force-pushed the cursor-stream-chunks branch 4 times, most recently from 27ff63c to dc7ac28 Compare April 17, 2024 12:29

dvarrazzo force-pushed the cursor-stream-chunks branch from dc7ac28 to 318e6bb Compare April 17, 2024 20:59

dvarrazzo requested changes Apr 17, 2024

View reviewed changes

psycopg/psycopg/_cursor_base.py Outdated Show resolved Hide resolved

psycopg/psycopg/rows.py Outdated Show resolved Hide resolved

psycopg/psycopg/cursor_async.py Outdated Show resolved Hide resolved

docs/api/cursors.rst Show resolved Hide resolved

dvarrazzo mentioned this pull request Apr 17, 2024

mark files generated by async_to_sync script as such in git #795

Closed

dlax force-pushed the cursor-stream-chunks branch 2 times, most recently from baa3797 to 7902650 Compare April 19, 2024 07:55

dvarrazzo force-pushed the cursor-stream-chunks branch from 7902650 to edae20f Compare May 18, 2024 21:03

dvarrazzo added this to the 3.2 milestone May 18, 2024

dvarrazzo force-pushed the cursor-stream-chunks branch 2 times, most recently from b7ffccf to 1a6cff9 Compare June 12, 2024 17:26

feat: add a size parameter to Cursor.stream()

9346151

This triggers results retrieval by chunks from the server, if > 1, leveraging the "chunked rows mode" from libpq 17. A new has_stream_chunked() capability is added.

dvarrazzo force-pushed the cursor-stream-chunks branch from 1a6cff9 to 9346151 Compare June 13, 2024 14:05

dvarrazzo merged commit ab84aaf into psycopg:master Jun 13, 2024

dlax deleted the cursor-stream-chunks branch June 14, 2024 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add a 'size' parameter to Cursor.stream()#794

add a 'size' parameter to Cursor.stream()#794
dvarrazzo merged 1 commit intopsycopg:masterfrom
dlax:cursor-stream-chunks

dlax commented Apr 17, 2024

Uh oh!

dvarrazzo commented Apr 17, 2024

Uh oh!

dvarrazzo left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dvarrazzo commented Jun 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

dlax commented Apr 17, 2024

Uh oh!

dvarrazzo commented Apr 17, 2024

Uh oh!

dvarrazzo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dvarrazzo commented Jun 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants