Error in Field Calculator Results, when using the 'division' operator

[qgib]

Author Name: Chinedu Ogbodo (Chinedu Ogbodo)
Original Redmine Issue: 11066
Affected QGIS version: 2.4.0

---

I created an attribute table for a point shapefile.
The attribute table had 3 columns, apart from the ‘id’ column.
Then, I tried to create a 4th column from two of the already existing columns, using field calculator.
The result of the division operator didn’t look good at all.
In the 4th column, it created a wrong result.
The results of dividing one column by the next, was really wrong. But when I tested the ‘addition’, ‘Subtraction’ and ‘Multiplication’ operator , they checked out fine, except ‘division’.
I observed that the results didn’t display correctly the decimal portion of the output.
I had set the precision level to 4.

For example, 3 / 7 gives 0.0000 (which is wrong) instead of 0. 4285714285714
In other words, it appears to throw away the decimal portions, instead of honour the precision level that was entered. And I think it is a fundamental flaw, which should be fixed.

[qgib]

Author Name: Matthias Kuhn (@m-kuhn)

---

Hi Chinedu,

The calculator works with the format, that parts of the expression are. That means, when you are doing an operation based on two integers (whole numbers), the result will be the same format. That also means, that you just have to change one of the two numbers to a decimal number to get more precise results. In your case this means using:

3.0 / 7

or

3 / 7.0

---

• resolution was changed from to wontfix
• status_id was changed from Open to Closed