copy_from ValueError "column list to long"

[psycoteam]

Originally submitted by: Dolf Andringa

Hi All,

I am using psycopg2 happyly to import data from csv files from other databases into a postgres database for analysis. Today I ran into a ValueError raised by cursor.copy_from with the message "column list too long".
After digging around some in the source I saw in cursor_type.h in _psyco_curs_copy_columns the if statement
if (offset + collen > DEFAULT_COPYBUFF - 2) {
Py_DECREF(col);
Py_DECREF(coliter);
PyErr_SetString(PyExc_ValueError, "column list too long");
return -1;
}

Why is psycopg2 only allowing a maximum number of characters in the column list? This doesn't seem to make much sense to me unless I am missing something. I was told by the people in the #postgresql channel on irc.freenode.net that this is not due to a limitation in postgresql itself. Is that correct?
Is there any way to allow a longer column length somehow?
Cheers,

Dolf.

[dvarrazzo]

Does it mean the sum of the length of the column names is more than 8k chars? How long is that?

Probably the author thought it would have been a safe limit... We should use a dynamically allocated buffer.

[dvarrazzo]

Fixed in my devel. To be released in 2.4.3.