[sqlite3] Do not truncate results of user functions and aggregates on the first NUL

[erlend-aasland]

BPO	44822
Nosy	@serhiy-storchaka, @miss-islington, @erlend-aasland
PRs	bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks #27588 [3.10] bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks (GH-27588) #27611 [3.9] bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks (GH-27588) #27639

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/erlend-aasland'
closed_at = <Date 2021-08-06.21:08:39.691>
created_at = <Date 2021-08-04.08:29:38.206>
labels = ['extension-modules', 'type-bug', '3.9', '3.10', '3.11']
title = '[sqlite3] Do not truncate results of user functions and aggregates on the first NUL'
updated_at = <Date 2021-08-06.21:08:39.690>
user = 'https://github.com/erlend-aasland'

bugs.python.org fields:

activity = <Date 2021-08-06.21:08:39.690>
actor = 'erlendaasland'
assignee = 'erlendaasland'
closed = True
closed_date = <Date 2021-08-06.21:08:39.691>
closer = 'erlendaasland'
components = ['Extension Modules']
creation = <Date 2021-08-04.08:29:38.206>
creator = 'erlendaasland'
dependencies = []
files = []
hgrepos = []
issue_num = 44822
keywords = ['patch']
message_count = 8.0
messages = ['398865', '398869', '398873', '398881', '398971', '399115', '399146', '399149']
nosy_count = 3.0
nosy_names = ['serhiy.storchaka', 'miss-islington', 'erlendaasland']
pr_nums = ['27588', '27611', '27639']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44822'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[erlend-aasland]

The third argument to sqlite3_result_text() is the length of the string passed as the second argument. Currently, we pass -1, so SQLite has to invoke strlen() to compute the length of the passed string. Suggesting to use PyUnicode_AsUTF8AndSize() iso. PyUnicode_AsUTF8() and pass the string size to avoid the superfluous strlen().

See also:

• https://sqlite.org/c3ref/result_blob.html

[serhiy-storchaka]

The difference between specifying negative and non-negative third argument of sqlite3_result_text() is that in the latter case the result can contain embedded NUL characters.

Could you please add a test for string containing embedded NUL?

Letting SQLite compute string length would work incorrect in case of embedded NULs, so I think that we should raise explicit error if sz > INT_MAX.

[erlend-aasland]

Could you please add a test for string containing embedded NUL?

Of course, thanks for the heads up.

I think that we should raise explicit error if sz > INT_MAX.

Yes, I thought about this. I do agree that raising OverflowError would be better. Thanks.

[erlend-aasland]

Serhiy, I've updated the PR, if you want to take a look. Thanks for your feedback.

[serhiy-storchaka]

New changeset 8f010dc by Erlend Egeberg Aasland in branch 'main':
bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks (GH-27588)
8f010dc

[miss-islington]

New changeset 2b1e713 by Miss Islington (bot) in branch '3.10':
bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks (GH-27588)
2b1e713

[serhiy-storchaka]

New changeset c352412 by Erlend Egeberg Aasland in branch '3.9':
[3.9] bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks (GH-27588). (GH-27639)
c352412

[erlend-aasland]

Thanks, Serhiy!