Add SQL_WCHART_CONVERT for UnixODBC.

[maparent]

The simple test in testunicode.py fails otherwise, on Ubuntu 64 i686, with a 64 bit driver. (Virtuoso)
This is not yet final code, I will package it better once I hear confirmation that it does not break other configurations, or create tests otherwise.
I'm sending it as a pull request in this state to raise the issue.

[mkleehammer]

Thanks so much. Take a look at the "u4" branch and see if it works for you:

https://github.com/mkleehammer/pyodbc/tree/u4

To make sure it is only enabled when Unicode is 4-bytes, I've used the array class to specifically look at the Unicode character size. I then enable it using define_macros but only in the Linux / Unix area.

I wasn't exactly sure what to do with the test code and I don't have Virtuoso right now, but it would be great to setup a unit test file for it. In the meantime, can you test the u4 branch against your install?

Thanks again,
Michael Kleehammer

[okbob]

Hello

I am testing this driver with Vertica (with last commit ddcaaed) and I have no success with simply strings:

# -*- coding: utf-8 -*-

import pyodbc
from optparse import OptionParser

vdb_config = 'DSN=verticaDSN'

cn = pyodbc.connect(vdb_config);

print cn.getinfo(pyodbc.SQL_DRIVER_NAME)
print cn.getinfo(pyodbc.SQL_DRIVER_VER)

c = cn.cursor()

c.execute("select * from bubu")

for r in c:
    print r
    print ">>> %s, %s, %s <<<", r.a, r.b, r.c

Result:

verticaodbcw.so
06.01.0200
(1L, u'\U4a4f4841', datetime.date(2013, 10, 15))
>>> %s, %s, %s <<< 1  2013-10-15
(2L, u'\U647a614e\U6b657261', datetime.date(2013, 10, 16))
>>> %s, %s, %s <<< 2 ???? 2013-10-16

I table bubu:

vertica=> select * from bubu;
 a |    b     |     c      
---+----------+------------
 1 | AHOJ     | 2013-10-15
 2 | Nazdarek | 2013-10-16
(2 rows)

I hope so ODBC unixODBC-2.2.14-12.el6_3.x86_64 is ok (isql shows correct result).

[maparent]

Thank you very much for the data point.
Let me ask a few questions:
Is column B a VARCHAR or NVARCHAR?
Could you put some international data in the column so we can see how it is treated?
It seems that \U4a4f4841 corresponds exactly to "AHOJ".
It makes me wonder if your column is really unicode, or actually UTF-8, latin-1 or some other encoding. Or, of course, sent as such.
Can I also ask you to check the width of wchar_t, SQLWCHAR and Py_UNICODE_SIZE?
(You should see PYODBC_UNICODE_WIDTH fly by in the compile, that corresponds to SQLWCHAR.)
It seems your machine is 64 bits. If you can tell us distro that may help.
Thank you,
Marc-Antoine

Le 2013-10-15 à 06:12, Pavel Stehule notifications@github.com a écrit :

Hello

I am testing this driver with Vertica (with last commit ddcaaed) and I have no success with simply strings:

-- coding: utf-8 --

import pyodbc
from optparse import OptionParser

vdb_config = 'DSN=verticaDSN'

cn = pyodbc.connect(vdb_config);

print cn.getinfo(pyodbc.SQL_DRIVER_NAME)
print cn.getinfo(pyodbc.SQL_DRIVER_VER)

c = cn.cursor()

c.execute("select * from bubu")

for r in c:
print r
print ">>> %s, %s, %s <<<", r.a, r.b, r.c
Result:

verticaodbcw.so
06.01.0200
(1L, u'\U4a4f4841', datetime.date(2013, 10, 15))

%s, %s, %s <<< 1 2013-10-15
(2L, u'\U647a614e\U6b657261', datetime.date(2013, 10, 16))
%s, %s, %s <<< 2 ???? 2013-10-16
I table bubu:

vertica=> select * from bubu;
a | b | c
---+----------+------------
1 | AHOJ | 2013-10-15
2 | Nazdarek | 2013-10-16
(2 rows)
I hope so ODBC unixODBC-2.2.14-12.el6_3.x86_64 is ok (isql shows correct result).

—
Reply to this email directly or view it on GitHub.

[okbob]

It is varchar - Vertica doesn't supports nvarchars - if I understand, it use UTF8

Yes, my environment is 64bit - RHEL.

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DPYODBC_VERSION=3.0.8-beta06 -DPYODBC_UNICODE_WIDTH=4 -DSQL_WCHART_CONVERT=1 -I/usr/include/python2.6 -c /root/pyodbc/src/pyodbcdbg.cpp -o build/temp.linux-x86_64-2.6/root/pyodbc/src/pyodbcdbg.o -Wno-write-strings

[mkleehammer]

Now that I look at this again I'm not sure it is a good idea. The unixODBC authors discourage changing flags like using wchar_t size, etc. Can you try 3.0.10 and see if it helps.

A better solution might be to allow an encoding to be set on a connection such as UTF8. I also might be able to optimize it a bit in Python 3.4+ where we might actually have 1-byte strings internally.

[mkleehammer]

I have rewritten Unicode support in the v4 branch.