{{ message }}

# Invalid return type for functions with INT128 /Numeric(38)/ argument [CORE6344] #6585

Closed
opened this issue Jun 24, 2020 · 12 comments
Closed

# Invalid return type for functions with INT128 /Numeric(38)/ argument [CORE6344]#6585

opened this issue Jun 24, 2020 · 12 comments
Assignees
Labels

### firebird-issue-importer commented Jun 24, 2020

 Submitted by: @AlexPeshkoff Some builtin functions, using int128 argument, should return same type value (CEIL, FLOOR). Others (like SQRT, LOG, EXP) should return decfloat(34) and use it for internal calculations. Currently all of them return and use internally double precision. ====== Test Details ====== See: functional\datatypes\int128-math-functions.fbt functional\datatypes\int128-agregate-functions.fbt NOTE-1: currently (06-jun-2020) there are several issues related to TRUNC() and ROUND() results. See .fbt for details. Test can be adjusted later after additional fixes. NOTE-2: Test for windowed functions deferred, see CORE6357 The text was updated successfully, but these errors were encountered:

### firebird-issue-importer commented Jun 24, 2020

 Modified by: @AlexPeshkoff assignee: Alexander Peshkov [ alexpeshkoff ]

### firebird-issue-importer commented Jun 24, 2020

 Commented by: @asfernandes Because this change will make existing code (not updated with any direct usage of the new types) use them, I think it requires a compatibility option for client libraries not supporting the new types returned.

### firebird-issue-importer commented Jun 25, 2020

 Commented by: @AlexPeshkoff It exists in B2, DataTypeCompatibility in firebird.conf

2) select round(170141183460469231731687303715884105727,1) from rdb$database; # Second statement issues: # Statement failed, SQLSTATE = 22003 arithmetic exception, numeric overflow, or string truncation -numeric value is out of range 3) select round(-170141183460469231731687303715884105728,0) from rdb$database;

Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.

4) select round(-170141183460469231731687303715884105728,1) from rdb\$database;

Statement failed, SQLSTATE = 22003
arithmetic exception, numeric overflow, or string truncation
-numeric value is out of range

### firebird-issue-importer commented Jul 6, 2020

 Modified by: @pavel-zotov status: Open [ 1 ] => Open [ 1 ] QA Status: No test => Covered by another test(s) Test Details: See: functional\datatypes\int128-math-functions.fbt NOTE: currently (06-jul-2020) there are several issues related to TRUNC() and ROUND() results. See .fbt for details. Test can be adjusted later after additional fixes.

### firebird-issue-importer commented Jul 6, 2020

 Modified by: @pavel-zotov status: Open [ 1 ] => Open [ 1 ] Test Details: See: functional\datatypes\int128-math-functions.fbt NOTE: currently (06-jul-2020) there are several issues related to TRUNC() and ROUND() results. See .fbt for details. Test can be adjusted later after additional fixes. => See: functional\datatypes\int128-math-functions.fbt functional\datatypes\int128-agregate-functions.fbt NOTE: currently (06-jul-2020) there are several issues related to TRUNC() and ROUND() results. See .fbt for details. Test can be adjusted later after additional fixes.

### firebird-issue-importer commented Jul 6, 2020

 Modified by: @pavel-zotov status: Open [ 1 ] => Open [ 1 ] Test Details: See: functional\datatypes\int128-math-functions.fbt functional\datatypes\int128-agregate-functions.fbt NOTE: currently (06-jul-2020) there are several issues related to TRUNC() and ROUND() results. See .fbt for details. Test can be adjusted later after additional fixes. => See: functional\datatypes\int128-math-functions.fbt functional\datatypes\int128-agregate-functions.fbt NOTE-1: currently (06-jun-2020) there are several issues related to TRUNC() and ROUND() results. See .fbt for details. Test can be adjusted later after additional fixes. NOTE-2: Test for windowed functions deferred, see CORE6357

### firebird-issue-importer commented Jul 8, 2020

 Commented by: @AlexPeshkoff A cos() function always converts it's argument to double precision - therefore it's OK as long as argument's precision does not overflow double's precision. With values larger than it cos() as implemented in firebird becomes a kind of random generator, which is bad, but absolutely not related with this ticket.

### firebird-issue-importer commented Jul 9, 2020

Commented by: @pavel-zotov

> A cos() function always converts it's argument to double precision - therefore it's OK as long as argument's precision does not overflow double's precision

COS() is periodical function, its full period 2*pi().

Using DECFLOAT34 datatype it is easy to get values for such numbers as 2^63 by emulating MOD() and substitute literal value for PI (first 39 digits) instead of call FB math function PI().

set sqlda_display on;
-- Value of PI from Wolfram: 3.14159265358979323846264338327950288419716939937510582097494459230781640628620899862803482534211706798214808651328...
recreate table test(x decfloat);
insert into test(x) values(9223372036854775807);

select
,cos( x - floor( x / 3.141592653589793238462643383279502884197/2) * 3.141592653589793238462643383279502884197 * 2 ) as ALMOST_EXACT_COSINE
,cos( x - floor(x / pi() / 2) * pi() * 2 ) as TOTALLY_WRONG_COSINE
from test;

# Output:

ALMOST_EXACT_COSINE 0.8477880073480192
TOTALLY_WRONG_COSINE 0.2317799904467769

PS.

0xFF.
What about MOD() ? Why it was not implemented for full support of DECFLOAT datatype ?

### firebird-issue-importer commented Jul 10, 2020

 Commented by: @AlexPeshkoff I slightly remember school trigonometry :) To work well for all decfloat values pi should have number of digits comparable with max power of decfloat, i.e. abot 6000. Even for decimal/numeric max scale (127) existing in firebird high precision libraries are not enough. Also I'm sure that if user needs to calculate trigonometric results for such huge values that means something is wrong, may be with his brain :D 0xff. MOD is also not supported for double precision, can you explain what means mod for *floating point* values?

### firebird-issue-importer commented Jul 13, 2020

 Modified by: @AlexPeshkoff status: Open [ 1 ] => Resolved [ 5 ] resolution: Fixed [ 1 ] Fix Version: 4.0 RC 1 [ 10930 ]