Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Psycopg2 can't work with huge redshift datatables #553
If you use a client-side cursor psycopg will fetch itersize items. In this example you are fetching 10 records in the sql and 100 in psycopg, which may or may not different. For extra control use a named cursor and
If the query is allocating excessive memory on the server is not psycopg bug. Psycopg is only opening a cursor on the server; what the server does is outside its control.
The rest of the discussion is polluted by sqlalchemy syntax: I don't know what sqlalchemy tells psycopg to do.
If you have a bug with psycopg not doing what it should with a named cursor open a self-contained bug. I don't see a bug here: I see a combination of not knowing what psycopg operation sqlalchemy does and not knowing what happens on the server.
referenced this issue
May 9, 2017
Btw, please be polite with words u are using. I know how cursors should work. @dvarrazzo With postgres it works, but if i use client side cursor with redshift, psycopg start allocating everything in memory instead of providing stream (cursor) object. Could u imagine that 160GB table is allocating in memory. Either my app crashes or database server, depending on the cursor type. I wouldn't ask if the default behaviour would work. I provided snippet that uses JDBC driver for comparison. It works like a charm, but my app is based on Python stack. I tried to use server side cursor just to compare. I tried both raw driver connections and ORM based, nothing helps (only for postgres).