Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple mathematics gone wrong [CORE2849] #3235

Closed
firebird-issue-importer opened this issue Feb 5, 2010 · 8 comments
Closed

Simple mathematics gone wrong [CORE2849] #3235

firebird-issue-importer opened this issue Feb 5, 2010 · 8 comments

Comments

@firebird-issue-importer
Copy link

firebird-issue-importer commented Feb 5, 2010

Submitted by: Hilmar Brodner (syn)

We have stumbled across this on a regular basis. Strangely enough I haven't found a bug report on this here although this should be a rather common and quite critical problem:

SELECT -- aim is to get 13,3% of 113 = 15,02900
-- With CAST
CAST((113 * (13.3 / 100)) AS NUMERIC(15,5)) AS "Wrong_1",
CAST((113 * (13.3 / 100.0)) AS NUMERIC(15,5)) AS "Wrong_2",
CAST((113 * 13.3 / 100) AS NUMERIC(15,5)) AS "Wrong_3",

    \-\- Without CAST 
     \(113 \* \(13\.3 / 100\)\)   AS "Wrong\_4",
     \(113 \* \(13\.3 / 100\.0\)\) AS "Wrong\_5",
     \(113 \* 13\.3 / 100\)  AS "Wrong\_6",

     \-\- Correct
     CAST\(\(113 \* \(13\.3 / \(100 \+ 0\.00000\)\)\) AS NUMERIC\(15,5\)\) AS "Ok\_1",
     CAST\(\(113 \* \(13\.3 / 100\.00000\)\) AS NUMERIC\(15,5\)\) AS "Ok\_2",
     CAST\(113 \* \(13\.3 / \(100 \+ 0\.00000\)\) AS NUMERIC\(15,5\)\) AS "Ok\_3",
     CAST\(113 \* \(13\.3 \* 0\.01\) AS NUMERIC\(15,5\)\) AS "Ok\_4",
     CAST\(\(113 \* 13\.3 \* 0\.01\) AS NUMERIC\(15,5\)\) AS "Ok\_5",
     \(113 \* 13\.3 \* 0\.01\) AS "Ok\_6"

FROM RDB$DATABASE

The problem basically results from FB not adding the necessary amount of decimal digits, thus continuing to calculate with the initial amount, although the division should have added further digits.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Commented by: @dyemanov

The rules for numerics are that the result of NUMERIC(A, X) / NUMERIC(B, Y) has the scale of exactly X + Y.
So everything you need is: 113 * (13.3 / 100.00). Or cast the every argument to the necessary scale. Or use double precision instead.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Commented by: @hvlad

Can't resist :

Many users never read documentation.
Strangely enough I haven't found a bug report on this here although this should be a rather common and quite critical problem (c)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Commented by: @helebor

The SQL engine delivers just what you ask for. This is a support question, not a bug report. Post your examples to the firebird-support list and ask why your expressions are not getting the results you expect.

Read up about integer division, scale, precision and numeric types in standard SQL and you will start to understand why SQL arithmetic doesn't work like your pocket calculator.

Hint: Supply enough places of decimal in your operands to provide the scale you want in your result and to avoid being caught by integer division. (In SQL, 10/3 returns 3, which is correct. 5/3 returns 1, which is also correct.)

Try
CAST((113.00* (13.300 / 100)) AS NUMERIC(15,5))

Please don't post bug reports until it is confirmed, via firebird-support first, and then in firebird-devel, that a bug exists.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Won't Fix [ 2 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Commented by: Hilmar Brodner (syn)

Sorry, but in my eyes it's still a bug:

Doing a SELECT with absolute values and getting different results depending on the way the calculation is written does not sound right to me. Maybe I'm expecting too much, but why does FB convert 13.3 to a NUMERIC(x.1) and not a double precision?

As for the documentation:
Please point me to the right place. I'm unable to find anything(!) on this matter. The only thing coming close to this ist something from Helen regarding the behaviour of Numerics in UNION-Selects in the FB 2.0 release notes...

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Commented by: @dyemanov

13.3 is a NUMERIC(x, 1) literal because the SQL specification declares it being so. 13.3e0 is a double precision literal though.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Commented by: @hvlad

Hilmar,

read IB6 beta docs, Language Reference (available at http://www.ibphoenix.com/downloads/60LangRef.zip)

page 22 - Multiplication and Division

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Feb 5, 2010

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant