Return an iterator on the cursor object to allow streaming of records with minimal memory usage. #350
Description
Is your feature request related to a problem? Please describe.
From discussion on this thread: https://www.reddit.com/r/dataengineering/comments/1o6oqbr/jupyter_notebooks_with_the_microsoft_python/
This is a request to return an iterator on cursor that does not put rows into a list, the iterator returns individual rows. It would work like a generator where each request would return a row from SQLFetch().
Describe the solution you'd like
The code to use it would look like this:
with connect(getenv("SQL_CONNECTION_STRING")) as connection: # type: ignore with connection.cursor("customers_query") as cursor: cursor.execute(SQL_QUERY_ORDERS_BY_CUSTOMER) for row in cursor: # do something with row
A natural use case for this would be pass the iterator to the BCP functionality we are working on to copy data between tables as quickly as possible with minimal memory usage.
Describe alternatives you've considered
You can do this with fetchmany() for small size batches and probably even fetchone() but neither returns a direct iterator.
Additional context
Also based on comments here: https://www.reddit.com/r/SQLServer/comments/1p41v8d/comment/nqk4puj/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button