Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Processing: precision loss on decimal number #29134

Closed
qgib opened this issue Feb 20, 2019 · 2 comments
Closed

Processing: precision loss on decimal number #29134

qgib opened this issue Feb 20, 2019 · 2 comments
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Processing Relating to QGIS Processing framework or individual Processing algorithms

Comments

@qgib
Copy link
Contributor

qgib commented Feb 20, 2019

Author Name: Tobias Wendorff (Tobias Wendorff)
Original Redmine Issue: 21316
Affected QGIS version: 3.5(master)
Redmine category:processing/core
Assignee: Nyall Dawson


Bug report

  1. create a new Temporary Scratch Layer
  2. add some fields
    @name: one
    type: decimal number (real)
    provider type: double
  • length: 32
  • precision: 30@

@name: two
type: decimal number (numeric)
provider type: numeric

  • length: 20
  • precision: 18@

@name: three
type: decimal number (decimal)
provider type: decimal

  • length: 20
  • precision: 18@

@name: four
type: decimal number (real)
provider type: real

  • length: internally set to -1
  • precision: internally set to -1@

@name: five
type: decimal number (double)
provider type: double precision

  • length: internally set to -1
  • precision: internally set to -1@
  1. add a new feature
  2. add the value "0.01234567890123456789" into each field (length 22, precision 20)
  3. save & check results
    @- one : 0.012345678901234568430878013601
  • two : 0.012345678901234567
  • three: 0.012345678901234567
  • four : 0.012345678901234568
  • five : 0.012345678901234568@
  1. When running "Refactor Fields", "Extract by Attribute" or others in Processing, the result is:
    @- one : 0.012345678901234568430878013601
  • two : 0.012345678901234568
  • three: 0.012345678901234568
  • four : 0.01235 ["source field" shows, precision gets smashed to 5]
  • five : 0.01235 ["source field" shows, precision gets smashed to 5]@
  1. Expected results

The output field should have the same precision as the input field.

@qgib
Copy link
Contributor Author

qgib commented Feb 20, 2019

Author Name: Nyall Dawson (@nyalldawson)


  • assigned_to_id was configured as Nyall Dawson
  • status_id was changed from Open to In Progress

@qgib
Copy link
Contributor Author

qgib commented Feb 20, 2019

Author Name: Nyall Dawson (@nyalldawson)


Applied in changeset f3cb348.


  • done_ratio was changed from 0 to 100
  • status_id was changed from In Progress to Closed

@qgib qgib closed this as completed Feb 20, 2019
@qgib qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! Processing Relating to QGIS Processing framework or individual Processing algorithms labels May 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Processing Relating to QGIS Processing framework or individual Processing algorithms
Projects
None yet
Development

No branches or pull requests

1 participant