Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Round brush coordinates #1641

Merged
merged 3 commits into from Mar 31, 2017
Merged

Round brush coordinates #1641

merged 3 commits into from Mar 31, 2017

Conversation

wch
Copy link
Collaborator

@wch wch commented Mar 30, 2017

This fixes #1634.

@wch wch added the review label Mar 30, 2017
@wch wch changed the title Round brush coordinates to 5 digits. Round brush coordinates to 5 digits Mar 30, 2017
@wch wch force-pushed the fix-plot-rounding branch 6 times, most recently from 7e7e9a2 to 7d82e80 Compare March 30, 2017 22:12
@wch wch changed the title Round brush coordinates to 5 digits Round brush coordinates Mar 30, 2017
@wch
Copy link
Collaborator Author

wch commented Mar 31, 2017

For future reference, here's a comparison of different methods of rounding numbers in Javascript:

// Round to a specified number of digits after the decimal. 0 means round to
// integer value. Negative values mean round to digits before decimal.
function roundDecimal(x, digits = 0) {
  const n = Math.pow(10, digits);
  return Math.round(x * n) / n;
}

// Round to a specified number of significant digits.
function roundSignif(x, digits = 1) {
  if (digits < 1)
    throw "Significant digits must be 1 or greater";

  const scaleFactor = Math.floor(Math.log10(Math.abs(x)));
  return roundDecimal(x, digits - scaleFactor - 1);
}


for (var precision = 1; precision < 15; precision++) {
  for (var i = 0; i < 10000; i++) {
    let val = Math.pow((Math.random() / Math.random()), 10*Math.random());

    let v1 = roundSignif(val, precision);
    let v2 = parseFloat(val.toPrecision(precision));
    if (v1 !== v2) {
      console.error("Inconsistency found after " + i + " iterations at precision " + precision + ":\n" +
        val + "\n" + v1 + "\n" + v2);
      break;
    }
  }
}

(Also at https://gist.github.com/wch/9d27ddcac1d81101b4708403a096cd31)

Using parseFloat(val.toPrecision(precision)) seems to be less prone to rounding error than other methods that use Math.round().

@wch wch merged commit 9804a79 into master Mar 31, 2017
@wch wch deleted the fix-plot-rounding branch March 31, 2017 15:46
@wch wch removed the review label Mar 31, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

108-module-output very flickery on IE11
1 participant