Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operations when using "SET DECFLOAT BIND BIGINT,n" with result of 11+ digits, fail with "Decimal float invalid operation" [CORE6181] #6426

Closed
firebird-issue-importer opened this issue Nov 2, 2019 · 10 comments

Comments

@firebird-issue-importer
Copy link

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Submitted by: @mrotteveel

Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Expected result is: 12345678.901

SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Expected result is: 12345678.900

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot).

Commits: 1827c56

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: @mrotteveel

description: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('123456789.0' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18-19 digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes.

=>

Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18-19 digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes.

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: @mrotteveel

description: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18-19 digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes.

=>

Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes.

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: @mrotteveel

description: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes.

=>

Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot).

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: @mrotteveel

description: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot).

=>

Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.

The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
1234567.890

SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Expected result is: 12345678.901

SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;

             CAST

=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

Expected result is: 12345678.900

It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.

Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database

will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).

As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot).

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: Sean Leyne (seanleyne)

summary: Decfloat bind BIGINT,n with resulting value with 11 digits or more fail with Decimal float invalid operation. An indeterminant error occurred during an operation. => Operations using Decfloat bind BIGINT,n with result of 11+ digits, fail with "Decimal float invalid operation"

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 2, 2019

Modified by: Sean Leyne (seanleyne)

summary: Operations using Decfloat bind BIGINT,n with result of 11+ digits, fail with "Decimal float invalid operation" => Operations when using "SET DECFLOAT BIND BIGINT,n" with result of 11+ digits, fail with "Decimal float invalid operation"

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 8, 2019

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 4.0 Beta 2 [ 10888 ]

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 9, 2019

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Done successfully

@firebird-issue-importer
Copy link
Author

@firebird-issue-importer firebird-issue-importer commented Nov 9, 2019

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants