String truncation exception on UPPER/LOWER functions, UTF8 database and some multibyte characters [CORE5255]

[firebird-automations]

Submitted by: Олег Короткий (олег короткий)

Votes: 2

select upper('ɐ') from rdb$database
-----------------
arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 2, actual 2.

select lower('ɐ') from rdb$database
-----------------
LOWER
ɐ

select lower('Ⱥ') from rdb$database
-----------------
arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 2, actual 2.

select upper('Ⱥ') from rdb$database
-----------------
UPPER
Ⱥ

[firebird-automations]

Commented by: Олег Короткий (олег короткий)

Also CONTAINING function bug:
select iif('ɱ' containing 'а',1,0) from rdb$database
------------------
arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 2, actual 2.

[firebird-automations]

Commented by: @dyemanov

What was the connection charset in these tests?

[firebird-automations]

Modified by: @dyemanov

security: Developers [ 10012 ] =>

[firebird-automations]

Commented by: Олег Короткий (олег короткий)

UTF8

[firebird-automations]

Modified by: @dyemanov

assignee: Adriano dos Santos Fernandes [ asfernandes ]

[firebird-automations]

Commented by: Олег Короткий (олег короткий)

2.5 is not affected

[firebird-automations]

Commented by: Олег Короткий (олег короткий)

LIKE is not affected, only CONTAINING

[firebird-automations]

Commented by: Олег Короткий (олег короткий)

My small collection of bad chars:
ɐ
ɱ
ɑ
Ⱥ

[firebird-automations]

Commented by: Franz Bartlechner (haba)

Same problem here!
Problem occures on Firebird Version 3.0.7 and nightly builds of Firebird 4.
Charset UTF8 (Everywere. DB created with UTF8, Tablefields created with UTF8 and Clientconnetion with UTF8)
No Problem on Firebird Version 2.5.9.
The Exception 'arithmetic exception, numeric overflow, or string truncation. string right truncation. expected length X, actual X." occurs also if a table field contains one of the Characters ɐ, ɱ, ɑ (... more chars) and a select with a
containing or upper is done on the table.

Reproduce the Exception:

CREATE TABLE TEST (TEST VARCHAR(10) CHARACTER SET UTF8 NOT NULL COLLATE UTF8);
INSERT INTO TEST (TEST) VALUES ('xxɐxx');
SELECT * FROM TEST WHERE TEST CONTAINING 'AAA'; (EXCEPTION HERE)
OR
SELECT * FROM TEST where UPPER(TEST) like '%AAA%'; (EXCEPTION HERE)

[firebird-automations]

Commented by: Franz Bartlechner (haba)

Found another char:
ɥ

[firebird-automations]

Commented by: Franz Bartlechner (haba)

I iterate through many/all chars and found out the following problematic:
Ⱥ U+023A
Ⱦ U+023E
ȿ U+023F
ɀ U+0240
ɐ U+0250
ɑ U+0251
ɒ U+0252
ɜ U+025C
ɡ U+0261
ɥ U+0265
ɦ U+0266
ɪ U+026A
ɫ U+026B
ɬ U+026C
ɱ U+0271
ɽ U+027D
ʇ U+0287
ʝ U+029D
ʞ U+029E

[asfernandes]

Commented by: Олег Короткий (олег короткий)

Also CONTAINING function bug:
select iif('ɱ' containing 'а',1,0) from rdb$database

arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 2, actual 2.

The problem with UPPER/LOWER is being fixed as part of this report.

Problem with CONTAINING is left for #6851.

[mrotteveel]

@asfernandes Will this also be fixed for 4.0.1?

[asfernandes]

@asfernandes Will this also be fixed for 4.0.1?

Yes. I've put now the missing label.

[pavel-zotov]

::: test details :::
CONTAINING still does not work.
Sent example to Adriano, waiting for reply.

[asfernandes]

::: test details :::
CONTAINING still does not work.
Sent example to Adriano, waiting for reply.

@pavel-zotov, as I've commented:

The problem with UPPER/LOWER is being fixed as part of this report.

Problem with CONTAINING is left for #6851.

[pavel-zotov]

sorry for noise, I was inattentive. Test for this issue will check all except statements with containing.

[pavel-zotov]

::: test details :::

Test verifies work of predicates, functions for string handling and comparison.
This is performed for VARCHAR and BLOB datatypes, but currently BLOB datatype has a problem:
invocation of rsa_encrypt() leads to 'malformed string' error.
Because of this, such code for BLOBs temporary is commented out.
Sent letter to Alex et al, 14-jun-2021 20:03.