lib: Increase max size of SQL statements #1124
Conversation
Increases length of SQL statements in db.execute and elsewhere (DB_SQL_MAX) from 1024 * pow(2, 3) to 1024 * pow(2, 6). Longer SQL statements are needed for long attribute values, but also for tables with a lot of columns, esp. when this table needs to be dropped and re-created during v.db.dropcolumn (which is the original motivation for this change). The new value is much higher than the current one and the SQL in the v.db.dropcolumn case above. If every occurence of DB_SQL_MAX in the code (77) would be allocated at the same time, it would be still under 5 MiB.
|
Out of curiosity: Which error message appears at time without this charge in case of a table with many columns and buffer being exceeded? |
|
I had these issues last year when trying to drop columns from a table with many many columns after using i.segment.stats. Here's the ticket I had opened back then: https://trac.osgeo.org/grass/ticket/3919 |
The one I got was what @veroandreo reported, so something like: Obviously, this suboptimal, mostly unhelpful, and it can still can occur even when db.execute is used directly. It is only far less likely with this change. Ideally, there would be no limit (or only the backend's limit if any). |
|
I have updated the description to be clear about the new size and that the ideal solution is something else. |
|
From Trac Ticket #3919:
This PR changes that to 65536, but it is possible it was not length issue, but rather compilation issue at that point. I struggled with that myself (whether I need to re-compile db.execute, lib or both). |
|
The OSGeo4W tests passed before and the Ubuntu tests are passing, so I intent to merge this, although 65536 bytes goes over some of the low limits discussed in bug-coreutils: thread stack size (but for that even 16384 would be too much since it is the whole stack size on some systems which would make even our current value 8192 barely acceptable). |
|
A rather old PR, I suggest to rebase and wait for the GH checks. I approve this PR, but there could be a new issue for dynamic allocation for the size of the SQL statement passed. |
Increases length of SQL statements in db.execute and elsewhere (DB_SQL_MAX) from 1024 * pow(2, 3) to 1024 * pow(2, 6). Longer SQL statements are needed for long attribute values, but also for tables with a lot of columns, esp. when this table needs to be dropped and re-created during v.db.dropcolumn (which is the original motivation for this change). The new value is much higher than the current one and the SQL statement I had problem with in the v.db.dropcolumn case above. However, since this is likely to be bigger than a path or other things we have fixed sizes for, it seems it deserves more. One char array would now take 0.0625 MiB which may be a problem if rest of the program allocates a lot of other things on stack, but likely fine because there is usually only few instances DB_SQL_MAX array in the program. If every occurrence of DB_SQL_MAX in the code (77) would be allocated at the same time (which is not the case), it would be under 5 MiB which still fits into glibc stack size (7.4 MB). The stack size issue may appear on some special platforms where 65536 bytes is a significant part of the stack size limit, but in general, for these platforms even 16384 can be too much since 65536 bytes might be the whole stack size or more on some systems (HP-UX 11 has 16 KB). This makes even our current value of 8192 bytes barely acceptable there, so increasing it is probably not an issue because the code doesn't work there already. A real, ideal fix would be having no limit and using dynamically allocated arrays for all SQL statements.
Increases length of SQL statements in db.execute and elsewhere (DB_SQL_MAX) from 1024 * pow(2, 3) to 1024 * pow(2, 6). Longer SQL statements are needed for long attribute values, but also for tables with a lot of columns, esp. when this table needs to be dropped and re-created during v.db.dropcolumn (which is the original motivation for this change). The new value is much higher than the current one and the SQL statement I had problem with in the v.db.dropcolumn case above. However, since this is likely to be bigger than a path or other things we have fixed sizes for, it seems it deserves more. One char array would now take 0.0625 MiB which may be a problem if rest of the program allocates a lot of other things on stack, but likely fine because there is usually only few instances DB_SQL_MAX array in the program. If every occurrence of DB_SQL_MAX in the code (77) would be allocated at the same time (which is not the case), it would be under 5 MiB which still fits into glibc stack size (7.4 MB). The stack size issue may appear on some special platforms where 65536 bytes is a significant part of the stack size limit, but in general, for these platforms even 16384 can be too much since 65536 bytes might be the whole stack size or more on some systems (HP-UX 11 has 16 KB). This makes even our current value of 8192 bytes barely acceptable there, so increasing it is probably not an issue because the code doesn't work there already. A real, ideal fix would be having no limit and using dynamically allocated arrays for all SQL statements.
Increases length of SQL statements in db.execute and elsewhere (DB_SQL_MAX) from 1024 * pow(2, 3) to 1024 * pow(2, 6). Longer SQL statements are needed for long attribute values, but also for tables with a lot of columns, esp. when this table needs to be dropped and re-created during v.db.dropcolumn (which is the original motivation for this change). The new value is much higher than the current one and the SQL statement I had problem with in the v.db.dropcolumn case above. However, since this is likely to be bigger than a path or other things we have fixed sizes for, it seems it deserves more. One char array would now take 0.0625 MiB which may be a problem if rest of the program allocates a lot of other things on stack, but likely fine because there is usually only few instances DB_SQL_MAX array in the program. If every occurrence of DB_SQL_MAX in the code (77) would be allocated at the same time (which is not the case), it would be under 5 MiB which still fits into glibc stack size (7.4 MB). The stack size issue may appear on some special platforms where 65536 bytes is a significant part of the stack size limit, but in general, for these platforms even 16384 can be too much since 65536 bytes might be the whole stack size or more on some systems (HP-UX 11 has 16 KB). This makes even our current value of 8192 bytes barely acceptable there, so increasing it is probably not an issue because the code doesn't work there already. A real, ideal fix would be having no limit and using dynamically allocated arrays for all SQL statements.
Increases length of SQL statements in db.execute and elsewhere (DB_SQL_MAX)
from 1024 * pow(2, 3) to 1024 * pow(2, 6). Longer SQL statements are needed
for long attribute values, but also for tables with a lot of columns,
esp. when this table needs to be dropped and re-created during v.db.dropcolumn
(which is the original motivation for this change).
The new value is much higher than the current one and the SQL statement I had problem with in the v.db.dropcolumn case above. However, since this is likely to be bigger than a path or other things we have fixed sizes for, it seems it deserves more.
One char array would now take 0.0625 MiB which may be a problem if rest of the program allocates a lot of other things on stack, but likely fine because there is usually only few instances DB_SQL_MAX array in the program. If every occurrence of DB_SQL_MAX in the code (77) would be allocated at the same time (which is not the case), it would be under 5 MiB which still fits into glibc stack size.
The real, ideal fix would be no limit and dynamically allocated arrays for the SQL statement, but that's a bigger change than this PR.