Can't read a NONE-Array with WIN1251 chars in UNICODE_FSS connection charset [CORE2500]

[firebird-automations]

Submitted by: @ibprovider

Attachments:
test_txt_array__utf8.wsf

Hello

1. Metadata: Text array with NONE charset
2. Insert to this array a WIN1251 text
3. Try select this array in UNICODE_FSS connection charset

And get the error

Test case
1. Download IBProvider (TRIAL or FREE)
2. Build project from TestCode\ActiveX\IBP\oledb_test\builds\vc8\ibp_oledb_test_vc8.sln
3. Create in ANY database the next structures

CREATE GENERATOR GEN_ID_TBL_CS__NONE;

CREATE TABLE TBL_CS__NONE ( TEST_ID T_INTEGER NOT NULL, COL_BLOB BLOB SUB_TYPE TEXT CHARACTER SET NONE, CHAR__1 CHAR(1) CHARACTER SET NONE, VARCHAR__1 VARCHAR(1) CHARACTER SET NONE, CHAR__8 CHAR(8) CHARACTER SET NONE, VARCHAR__8 VARCHAR(8) CHARACTER SET NONE, CHAR__32 CHAR(32) CHARACTER SET NONE, VARCHAR__32 VARCHAR(32) CHARACTER SET NONE, CHAR_ARRAY__8 CHAR(8) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__8 VARCHAR(8) [0:2] CHARACTER SET NONE, CHAR_ARRAY__32 CHAR(32) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__32 VARCHAR(32) [0:2] CHARACTER SET NONE, CONSTRAINT PK_TBL_CHAR__NONE PRIMARY KEY (TEST_ID) );

SET TERM !! ;
CREATE TRIGGER BI_TBL_CS__NONE_TEST_ID FOR TBL_CS__NONE BEFORE INSERT AS BEGIN IF(NEW.TEST_ID IS NULL)THEN NEW.TEST_ID=GEN_ID(GEN_ID_TBL_CS__NONE,1); END!!
SET TERM ; !!

4. Execute test case "array.002.unicode.write_array_str.TBL_CS__NONE.VARCHAR_ARRAY__8.ctype__WIN1251.chars__WIN1251"

---------
[BUG FIX]

file: cvt.cpp
function: transliterate

if \(\(charset1 \!= charset2\) &&
	\(charset1 \!= ttype\_none\) && // <\-\-\-\-  append this condition
	\(charset2 \!= ttype\_none\) &&
	\(charset1 \!= ttype\_binary\) &&
	\(charset2 \!= ttype\_binary\) &&
	\(charset1 \!= ttype\_dynamic\) && \(charset2 \!= ttype\_dynamic\)\)

[firebird-automations]

Modified by: @ibprovider

description: Hello

1. Metadata: Text array with NONE charset
2. Insert to this array a WIN1251 text
3. Try select this array in UNICODE_FSS connection charset

And get the error

Test case
1. Download IBProvider (TRIAL or FREE)
2. Build project from TestCode\ActiveX\IBP\oledb_test\builds\vc8\ibp_oledb_test_vc8.sln
3. Create in ANY database a next structures

CREATE GENERATOR GEN_ID_TBL_CS__NONE;

CREATE TABLE TBL_CS__NONE ( TEST_ID T_INTEGER NOT NULL, COL_BLOB BLOB SUB_TYPE TEXT CHARACTER SET NONE, CHAR__1 CHAR(1) CHARACTER SET NONE, VARCHAR__1 VARCHAR(1) CHARACTER SET NONE, CHAR__8 CHAR(8) CHARACTER SET NONE, VARCHAR__8 VARCHAR(8) CHARACTER SET NONE, CHAR__32 CHAR(32) CHARACTER SET NONE, VARCHAR__32 VARCHAR(32) CHARACTER SET NONE, CHAR_ARRAY__8 CHAR(8) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__8 VARCHAR(8) [0:2] CHARACTER SET NONE, CHAR_ARRAY__32 CHAR(32) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__32 VARCHAR(32) [0:2] CHARACTER SET NONE, CONSTRAINT PK_TBL_CHAR__NONE PRIMARY KEY (TEST_ID) );

SET TERM !! ;
CREATE TRIGGER BI_TBL_CS__NONE_TEST_ID FOR TBL_CS__NONE BEFORE INSERT AS BEGIN IF(NEW.TEST_ID IS NULL)THEN NEW.TEST_ID=GEN_ID(GEN_ID_TBL_CS__NONE,1); END!!
SET TERM ; !!

4. Execute test case "array.002.unicode.write_array_str.TBL_CS__NONE.VARCHAR_ARRAY__8.ctype__WIN1251.chars__WIN1251"

---------
[BUG FIX]

file: cvt.cpp
function: transliterate

if \(\(charset1 \!= charset2\) &&
	\(charset1 \!= ttype\_none\) && // <\-\-\-\-  append this condition
	\(charset2 \!= ttype\_none\) &&
	\(charset1 \!= ttype\_binary\) &&
	\(charset2 \!= ttype\_binary\) &&
	\(charset1 \!= ttype\_dynamic\) && \(charset2 \!= ttype\_dynamic\)\)

=>

Hello

1. Metadata: Text array with NONE charset
2. Insert to this array a WIN1251 text
3. Try select this array in UNICODE_FSS connection charset

And get the error

Test case
1. Download IBProvider (TRIAL or FREE)
2. Build project from TestCode\ActiveX\IBP\oledb_test\builds\vc8\ibp_oledb_test_vc8.sln
3. Create in ANY database the next structures

CREATE GENERATOR GEN_ID_TBL_CS__NONE;

CREATE TABLE TBL_CS__NONE ( TEST_ID T_INTEGER NOT NULL, COL_BLOB BLOB SUB_TYPE TEXT CHARACTER SET NONE, CHAR__1 CHAR(1) CHARACTER SET NONE, VARCHAR__1 VARCHAR(1) CHARACTER SET NONE, CHAR__8 CHAR(8) CHARACTER SET NONE, VARCHAR__8 VARCHAR(8) CHARACTER SET NONE, CHAR__32 CHAR(32) CHARACTER SET NONE, VARCHAR__32 VARCHAR(32) CHARACTER SET NONE, CHAR_ARRAY__8 CHAR(8) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__8 VARCHAR(8) [0:2] CHARACTER SET NONE, CHAR_ARRAY__32 CHAR(32) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__32 VARCHAR(32) [0:2] CHARACTER SET NONE, CONSTRAINT PK_TBL_CHAR__NONE PRIMARY KEY (TEST_ID) );

SET TERM !! ;
CREATE TRIGGER BI_TBL_CS__NONE_TEST_ID FOR TBL_CS__NONE BEFORE INSERT AS BEGIN IF(NEW.TEST_ID IS NULL)THEN NEW.TEST_ID=GEN_ID(GEN_ID_TBL_CS__NONE,1); END!!
SET TERM ; !!

4. Execute test case "array.002.unicode.write_array_str.TBL_CS__NONE.VARCHAR_ARRAY__8.ctype__WIN1251.chars__WIN1251"

---------
[BUG FIX]

file: cvt.cpp
function: transliterate

if \(\(charset1 \!= charset2\) &&
	\(charset1 \!= ttype\_none\) && // <\-\-\-\-  append this condition
	\(charset2 \!= ttype\_none\) &&
	\(charset1 \!= ttype\_binary\) &&
	\(charset2 \!= ttype\_binary\) &&
	\(charset1 \!= ttype\_dynamic\) && \(charset2 \!= ttype\_dynamic\)\)

[firebird-automations]

Modified by: @dyemanov

assignee: Adriano dos Santos Fernandes [ asfernandes ]

[firebird-automations]

Commented by: @ibprovider

Hello, again

This is separate small test case on VBScript (UTF-8 File)

0. Download and install last IBProvider (TRIAL)
1. Create metadata in ANY FB-database
2. Change c_db_location constant
3. Run cmd.exe
4. Execute test_txt_array__utf8.wsf

Test:
- connect to database with NONE charset
- create array with win1251 text
- connect to database with WIN1251 charset
- read array. OK
- connect to database with UNICODE_FSS charset
- read array. FAIL

For use debugger for this script, execute "test_txt_array__utf8.wsf //X"

[firebird-automations]

Modified by: @ibprovider

Attachment: test_txt_array__utf8.wsf [ 11456 ]

[firebird-automations]

Commented by: @asfernandes

The proposed fix is a band-aid which would certainly have others consequences.

The real problem is that array slices works with blr_text, blr_varying and blr_cstring, and not with the blr_*2 versions. So it ignores the charset and use the connection charset.

In your case, the target, which should be using NONE charset is marked with UNICODE_FSS charset. It then fails verifying malformed characters in the UNICODE_FSS descriptor. This is not the mistake, it's intentional. It just happens that something fail early.

[firebird-automations]

Commented by: @ibprovider

>The proposed fix is a band-aid which would certainly have others consequences.

Yes, I know this.

I think, this condition is wrongly from begin. Because, at equal situation, I use other rules (one for all datatypes: Text BLOB, Text Array, Text Column)

With something like next:

[return data to user]

Step1:
Transformation of SourceCharsetID and TargetCharsetID to StableCharsetID:

For Source (ColumnCharset):
NONE->NONE
OCTETS->OCTETS
DYNAMIC->CONNECTION_CHARSET
Other->Other

For Target (ConnectionCharset):
if(TARGET==NONE || SOURCE==NONE || SOURCE==OCTETS)
return SOURCE_CHARSET

OCTETS-> Generate Error // Can't use OCTETS for Connection Charset

Other->Other

Step2:
Decision (Need translation or not)

if(StableSourceCharsetID!=StableTargetCharsetID)
{
return true; //Need translate between codepage
}

return false; //Equal codepages

---
"user" do not know about internal specific of array implementation. He uses CHAR/VARCHAR as for simple columns

And wait similar behavior for text transformation

---
PS. Adriano, I ask you fix this problem, because ARRAYS - my favorite data types :)

[firebird-automations]

Commented by: @ibprovider

Hi

I think, you can close this ticked - I found the decision :)

need:
1. use the "isc_put_slice", "isc_get_slice"
2. pass text data through "isc_blr_text2" and "isc_blr_varying2" datatypes

This is works for all FB/IB versions

[firebird-automations]

Modified by: @asfernandes

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Won't Fix [ 2 ]

[firebird-automations]

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]