FetchMany MemoryError

[tristanwietsma]

I'm using an iterator with fetchmany to stream a large result out as CSV:

class ODBCContext:

     def __init__(self, odbc_connection_string):
        self.conn_str = odbc_connection_string

    def __enter__(self):
        self.connection = pyodbc.connect(self.conn_str)
        self.cursor = self.connection.cursor()
        return self.cursor

    def __exit__(self, error_type, error_value, traceback):
        self.cursor.close()
        self.connection.close()

def ResultIter(cursor, size=100):
    i = 0
    while True:
        part = cursor.fetchmany(size)
        i += len(part)
        print "fetched:", i
        if not part:
            print "break"
            break
        yield part

def StreamingCSV(connection_string, query, header=None):
        if header:
            yield string.join([header],',') + '\n'

        with ODBCContext(connection_string) as cursor:
            cursor.execute(query)
            for part in ResultIter(cursor):
                for row in part:
                    line = string.join([str(r) for r in row],',') + '\n'
                    yield line

For a large query, I can only get 3641 rows out of ~4300 before fetchmany throws a MemoryError.

I've tried a variety of array sizes, as well as fetchone. The error is consistent.

I've also monitored memory usage for the process and I'm not entirely convinced there is a legitimate memory error: physical memory remains stable around 25 megs, virtual at 2.5 gb. At no point do I get any telltale signs of a machine running out of memory.

Any thoughts?

[tristanwietsma]

Seems to be working fine for pypyodbc.

[mkleehammer]

I have pyodbc in a number of 24x7 servers and do not see any evidence of leaks. Is this still occurring? I'm going to close this, but please reopen if you found anything.

[mkleehammer]

I made 4.0.15 yesterday to fix that. Please try that version.

…

On Mar 16, 2017, 7:46 AM -0500, ZaM-Rai ***@***.***>, wrote: I'm getting this error after about 24 hours of running my DB. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub (#25 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/AABrHXSpN5rab8BY1g9WZHlPTvt1ZJNCks5rmS8UgaJpZM4CJQKZ).