Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

floating point error in round function in Ubuntu #25958

Closed
qgib opened this issue Feb 8, 2018 · 3 comments
Closed

floating point error in round function in Ubuntu #25958

qgib opened this issue Feb 8, 2018 · 3 comments
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Expressions Related to the QGIS expression engine or specific expression functions

Comments

@qgib
Copy link
Contributor

qgib commented Feb 8, 2018

Author Name: Michel Stuyts (@mstuyts)
Original Redmine Issue: 18062
Affected QGIS version: master
Redmine category:expressions


The following procedure gives a floating point error in the results in Ubuntu Linux

  1. Open QGIS 2.99 in Ubuntu Linux 16.04 LTS
  2. Create a new shapefile for lines
  3. Start editing this shapefile and create a line
  4. Stop editing the shapefile and save the edits
  5. Go to the layer styling panel and select "Show labels for this layer"
  6. For "Label with" add round(123.456789,2) as value and click "Apply"
  7. The label now shows 123.459999999999
  8. Open attribute table
  9. Open field calculator
  10. Select Create new field, with output field type Real, output field length 10 and Precision 10 (that's the highest possible value). As a expression enter round(123.456789,2).
  11. The output preview says 123.459999999999
  12. The result in the table says 123.4600000000

On Windows 10, the result is always the correct 123.46.

@qgib
Copy link
Contributor Author

qgib commented Feb 8, 2018

Author Name: Kory Roberts (Kory Roberts)


Looks similar to these already reported bugs:
#25756
#25903

@qgib qgib closed this as completed Feb 9, 2018
@qgib
Copy link
Contributor Author

qgib commented Feb 9, 2018

Author Name: Nyall Dawson (@nyalldawson)


This is an issue with the underlying Qt 5.6 library, and is not something we can fix in QGIS (believe us, we tried HARD). The ONLY solution is to use a build with a newer version of Qt (5.7 and above are fine) - that may mean updating your distro.


  • resolution was changed from to up/downstream
  • status_id was changed from Open to Closed

@qgib
Copy link
Contributor Author

qgib commented Feb 12, 2018

Author Name: Michel Stuyts (@mstuyts)


Since I just want to show it as a label and don't want to calculate with it, for now I'll just use the following custom function in the Function Editor:

@qgsfunction(args='auto', group='Custom')
def str_round(value1, value2, feature, parent):
    return str(round(value1,value2))

This will show the correct result as a string.

@qgib qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! Expressions Related to the QGIS expression engine or specific expression functions labels May 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Expressions Related to the QGIS expression engine or specific expression functions
Projects
None yet
Development

No branches or pull requests

1 participant