Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MemoryError with Version 4.0. #223

Closed
Sargul opened this issue Apr 5, 2017 · 4 comments
Closed

MemoryError with Version 4.0. #223

Sargul opened this issue Apr 5, 2017 · 4 comments

Comments

@Sargul
Copy link

Sargul commented Apr 5, 2017

Hi,

we recently tried to switch to pyodbc 4.0.16 from 3.x and ran into troubles with a connection to IBM DB2

We use a db2 version 10.1 client and the unixodbc drivers.

This configuration has been working with version pyodbc 3.x for many months but now fails whenever we read a NULL value from a char or varchar field in the database

Sample program:

import pyodbc
connection = pyodbc.connect(
            'DSN={0};UID={1};PWD={2}'.format("MONSTATS", "rpri122", "Feb15Feb"),
            autocommit=True,
            ansi=True,
            timeout=120
        )
cursor = connection.cursor()
try:
   cursor.execute("drop table test")
except Exception as ex:
   pass
cursor.execute("CREATE TABLE TEST (COL1 INTEGER, COL2 VARCHAR(10))")
cursor.execute("INSERT INTO TEST VALUES(1,'test')")
print("sel col1")
cursor.execute("SELECT col1 FROM TEST").fetchall()
print("sel col2")
cursor.execute("SELECT col2 FROM TEST").fetchall()
cursor.execute("INSERT INTO TEST VALUES(null,null)")
print("sel col1")
cursor.execute("SELECT col1 FROM TEST").fetchall()
print("sel col2")
cursor.execute("SELECT col2 FROM TEST").fetchall()
print("done")

On the execute

SELECT col2 FROM TEST

we get a Segmentation fault or memory error as soon as a null value is present. The kind of error differs from script to script.

$test.py
sel col1
sel col2
sel col1
sel col2
Segmentation fault

Any ideas what might cause this. The IBM DB2 driver works fine with other ODBC Tools.

Kind Regards

Charly
@cyrixsimon
Copy link

Hi,

a backtrace can be found at the bottom of this post.

I think the returned values from the driver are not determined correctly.

ReadVarColumn: SQLGetData avail=4096 --> ret=0 cbData=-1
Segmentation fault

This should be handled here (getdata.cpp >>> ReadVarColumn) ...

if (ret == SQL_SUCCESS && cbData < 0)
{
            // HACK: FreeTDS 0.91 on OS/X returns -4 for NULL data instead of SQL_NULL_DATA
            // (-1).  I've traced into the code and it appears to be the result of assigning -1
            // to a SQLLEN.  We are going to treat all negative values as NULL.
            ret = SQL_NULL_DATA;
            cbData = 0;
}

... but in this case not correctly for the null value.

BACKTRACE

#0 0x00007ffff6c62e1e in __memcpy_sse2_unaligned () from /lib64/libc.so.6
#1 0x00007ffff7922f4d in PyBytes_FromStringAndSize (str=str@entry=0x77a130 "\230L\367\366\377\177", size=) at Objects/bytesobject.c:123
#2 0x00007ffff7a370ec in do_mkvalue (flags=, p_va=, p_format=) at Python/modsupport.c:378
#3 do_mktuple (p_format=p_format@entry=0x7fffffffd3e0, p_va=p_va@entry=0x7fffffffd3e8, endchar=endchar@entry=0, n=5, flags=flags@entry=1) at Python/modsupport.c:196
#4 0x00007ffff7a386b4 in va_build_value (format=, va=va@entry=0x7fffffffd428, flags=flags@entry=1) at Python/modsupport.c:482
#5 0x00007ffff7a3883a in _Py_VaBuildValue_SizeT (format=, va=va@entry=0x7fffffffd428) at Python/modsupport.c:461
#6 0x00007ffff7915df1 in _PyObject_CallFunction_SizeT (callable=0x7ffff7d51c00 <_PyExc_UnicodeDecodeError>, format=format@entry=0x7ffff7ab4ff9 "sy#nns") at Objects/abstract.c:2543
#7 0x00007ffff7936b71 in PyUnicodeDecodeError_Create (encoding=, object=, length=, start=, end=end@entry=314,
reason=reason@entry=0x7ffff7abf3df "illegal UTF-16 surrogate") at Objects/exceptions.c:2036
#8 0x00007ffff79c37df in make_decode_exception (reason=0x7ffff7abf3df "illegal UTF-16 surrogate", endpos=314, startpos=, length=, input=, encoding=,
exceptionObject=0x7fffffffd660) at Objects/unicodeobject.c:4218
#9 unicode_decode_call_errorhandler_writer (errors=, errorHandler=0x7fffffffd650, encoding=, reason=0x7ffff7abf3df "illegal UTF-16 surrogate", input=0x7fffffffd600, inend=0x7fffffffd640,
startinpos=0x7fffffffd610, endinpos=0x7fffffffd620, exceptionObject=0x7fffffffd660, inptr=0x7fffffffd630, writer=0x7fffffffd670) at Objects/unicodeobject.c:4374
#10 0x00007ffff79c90fd in PyUnicode_DecodeUTF16Stateful (s=, size=, errors=0x7ffff65c040c "strict", byteorder=, consumed=0x0) at Objects/unicodeobject.c:5668
#11 0x00007ffff65c0299 in TextBufferToObject (enc=..., pbData=, cbData=) at src/textenc.cpp:100
#12 0x00007ffff65bec02 in GetText (cur=cur@entry=0x7ffff481c030, iCol=iCol@entry=0) at src/getdata.cpp:267
#13 0x00007ffff65bf2cb in GetData (cur=cur@entry=0x7ffff481c030, iCol=iCol@entry=0) at src/getdata.cpp:808
#14 0x00007ffff65b5b19 in Cursor_fetch (cur=0x7ffff481c030) at src/cursor.cpp:1002
#15 0x00007ffff65b60fb in Cursor_fetchone (self=, args=) at src/cursor.cpp:1109
#16 0x00007ffff796dd5b in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff59533a8, args=args@entry=0x692788, nargs=0, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:192
#17 0x00007ffff796e015 in _PyCFunction_FastCallKeywords (func=func@entry=0x7ffff59533a8, stack=stack@entry=0x692788, nargs=, kwnames=kwnames@entry=0x0) at Objects/methodobject.c:295
#18 0x00007ffff7a062ea in call_function (pp_stack=pp_stack@entry=0x7fffffffda80, oparg=oparg@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:4798
#19 0x00007ffff7a099ec in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3284
#20 0x00007ffff7a05eee in _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffff6b46ae0, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x8, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name=name@entry=0x0, qualname=qualname@entry=0x0)
at Python/ceval.c:4128
#21 0x00007ffff7a06513 in PyEval_EvalCodeEx (_co=_co@entry=0x7ffff6b46ae0, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0,
kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:4149
#22 0x00007ffff7a0655b in PyEval_EvalCode (co=co@entry=0x7ffff6b46ae0, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120) at Python/ceval.c:695
#23 0x00007ffff7a40de2 in run_mod (arena=0x7ffff6ba4078, flags=0x7fffffffdd90, locals=0x7ffff6b8d120, globals=0x7ffff6b8d120, filename=0x7ffff67d95a8, mod=0x63f098) at Python/pythonrun.c:980
#24 PyRun_FileExFlags (fp=fp@entry=0x6925f0, filename_str=filename_str@entry=0x7ffff6a44898 "test.py", start=start@entry=257, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120, closeit=closeit@entry=1,
flags=flags@entry=0x7fffffffdd90) at Python/pythonrun.c:933
#25 0x00007ffff7a40f47 in PyRun_SimpleFileExFlags (fp=fp@entry=0x6925f0, filename=, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdd90) at Python/pythonrun.c:396
#26 0x00007ffff7a413e3 in PyRun_AnyFileExFlags (fp=fp@entry=0x6925f0, filename=, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdd90) at Python/pythonrun.c:80
#27 0x00007ffff7a5cf1d in run_file (p_cf=0x7fffffffdd90, filename=0x608270 L"test.py", fp=0x6925f0) at Modules/main.c:338
#28 Py_Main (argc=argc@entry=2, argv=argv@entry=0x603010) at Modules/main.c:809
#29 0x0000000000400bdc in main (argc=2, argv=) at ./Programs/python.c:69
(gdb) where
#0 0x00007ffff6c62e1e in __memcpy_sse2_unaligned () from /lib64/libc.so.6
#1 0x00007ffff7922f4d in PyBytes_FromStringAndSize (str=str@entry=0x77a130 "\230L\367\366\377\177", size=) at Objects/bytesobject.c:123
#2 0x00007ffff7a370ec in do_mkvalue (flags=, p_va=, p_format=) at Python/modsupport.c:378
#3 do_mktuple (p_format=p_format@entry=0x7fffffffd3e0, p_va=p_va@entry=0x7fffffffd3e8, endchar=endchar@entry=0, n=5, flags=flags@entry=1) at Python/modsupport.c:196
#4 0x00007ffff7a386b4 in va_build_value (format=, va=va@entry=0x7fffffffd428, flags=flags@entry=1) at Python/modsupport.c:482
#5 0x00007ffff7a3883a in _Py_VaBuildValue_SizeT (format=, va=va@entry=0x7fffffffd428) at Python/modsupport.c:461
#6 0x00007ffff7915df1 in _PyObject_CallFunction_SizeT (callable=0x7ffff7d51c00 <_PyExc_UnicodeDecodeError>, format=format@entry=0x7ffff7ab4ff9 "sy#nns") at Objects/abstract.c:2543
#7 0x00007ffff7936b71 in PyUnicodeDecodeError_Create (encoding=, object=, length=, start=, end=end@entry=314,
reason=reason@entry=0x7ffff7abf3df "illegal UTF-16 surrogate") at Objects/exceptions.c:2036
#8 0x00007ffff79c37df in make_decode_exception (reason=0x7ffff7abf3df "illegal UTF-16 surrogate", endpos=314, startpos=, length=, input=, encoding=,
exceptionObject=0x7fffffffd660) at Objects/unicodeobject.c:4218
#9 unicode_decode_call_errorhandler_writer (errors=, errorHandler=0x7fffffffd650, encoding=, reason=0x7ffff7abf3df "illegal UTF-16 surrogate", input=0x7fffffffd600, inend=0x7fffffffd640,
startinpos=0x7fffffffd610, endinpos=0x7fffffffd620, exceptionObject=0x7fffffffd660, inptr=0x7fffffffd630, writer=0x7fffffffd670) at Objects/unicodeobject.c:4374
#10 0x00007ffff79c90fd in PyUnicode_DecodeUTF16Stateful (s=, size=, errors=0x7ffff65c040c "strict", byteorder=, consumed=0x0) at Objects/unicodeobject.c:5668
#11 0x00007ffff65c0299 in TextBufferToObject (enc=..., pbData=, cbData=) at src/textenc.cpp:100
#12 0x00007ffff65bec02 in GetText (cur=cur@entry=0x7ffff481c030, iCol=iCol@entry=0) at src/getdata.cpp:267
#13 0x00007ffff65bf2cb in GetData (cur=cur@entry=0x7ffff481c030, iCol=iCol@entry=0) at src/getdata.cpp:808
#14 0x00007ffff65b5b19 in Cursor_fetch (cur=0x7ffff481c030) at src/cursor.cpp:1002
#15 0x00007ffff65b60fb in Cursor_fetchone (self=, args=) at src/cursor.cpp:1109
#16 0x00007ffff796dd5b in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff59533a8, args=args@entry=0x692788, nargs=0, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:192
#17 0x00007ffff796e015 in _PyCFunction_FastCallKeywords (func=func@entry=0x7ffff59533a8, stack=stack@entry=0x692788, nargs=, kwnames=kwnames@entry=0x0) at Objects/methodobject.c:295
#18 0x00007ffff7a062ea in call_function (pp_stack=pp_stack@entry=0x7fffffffda80, oparg=oparg@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:4798
#19 0x00007ffff7a099ec in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3284
#20 0x00007ffff7a05eee in _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffff6b46ae0, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x8, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name=name@entry=0x0, qualname=qualname@entry=0x0)
at Python/ceval.c:4128
#21 0x00007ffff7a06513 in PyEval_EvalCodeEx (_co=_co@entry=0x7ffff6b46ae0, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0,
kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:4149
#22 0x00007ffff7a0655b in PyEval_EvalCode (co=co@entry=0x7ffff6b46ae0, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120) at Python/ceval.c:695
#23 0x00007ffff7a40de2 in run_mod (arena=0x7ffff6ba4078, flags=0x7fffffffdd90, locals=0x7ffff6b8d120, globals=0x7ffff6b8d120, filename=0x7ffff67d95a8, mod=0x63f098) at Python/pythonrun.c:980
#24 PyRun_FileExFlags (fp=fp@entry=0x6925f0, filename_str=filename_str@entry=0x7ffff6a44898 "test.py", start=start@entry=257, globals=globals@entry=0x7ffff6b8d120, locals=locals@entry=0x7ffff6b8d120, closeit=closeit@entry=1,
flags=flags@entry=0x7fffffffdd90) at Python/pythonrun.c:933
#25 0x00007ffff7a40f47 in PyRun_SimpleFileExFlags (fp=fp@entry=0x6925f0, filename=, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdd90) at Python/pythonrun.c:396
#26 0x00007ffff7a413e3 in PyRun_AnyFileExFlags (fp=fp@entry=0x6925f0, filename=, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdd90) at Python/pythonrun.c:80
#27 0x00007ffff7a5cf1d in run_file (p_cf=0x7fffffffdd90, filename=0x608270 L"test.py", fp=0x6925f0) at Modules/main.c:338
#28 Py_Main (argc=argc@entry=2, argv=argv@entry=0x603010) at Modules/main.c:809
#29 0x0000000000400bdc in main (argc=2, argv=) at ./Programs/python.c:69

Regards
Simon

@cyrixsimon
Copy link

cyrixsimon commented Apr 11, 2017 •
edited

Hi,

I have add a cast to int of cbData in the ReadVarColumn function of getdata.cpp. This solves the null value handling problem but there are still memory errors on data values.

if (ret == SQL_SUCCESS && (int)cbData < 0)
{
            // HACK: FreeTDS 0.91 on OS/X returns -4 for NULL data instead of SQL_NULL_DATA
            // (-1).  I've traced into the code and it appears to be the result of assigning -1
            // to a SQLLEN.  We are going to treat all negative values as NULL.
            ret = SQL_NULL_DATA;
            cbData = 0;
}

Regards
Simon

cyrixsimon pushed a commit to cyrixsimon/pyodbc that referenced this issue Apr 11, 2017
fix for issue mkleehammer#223 (memory error with db2 driver)
0ff4e1a
@Sargul
Copy link
Author

Sargul commented Apr 26, 2017

Is there any chance that the commit of @cyrixsimon will be reviews as this would fix our issues

mkleehammer added a commit that referenced this issue Apr 28, 2017
@mkleehammer
Merge pull request #226 from cyrixsimon/master
ca3df0d 
fix for issue #223 (memory error with db2 driver)
@mkleehammer
Copy link
Owner

I've merged it in but have not made a build yet. I was hoping to fix a few things before. Is that a problem or do you need a prebuilt, official version to continue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mkleehammer @Sargul @gordthompson @cyrixsimon