New var creation with right bit shift shows incorrect result using same initializer #1336
Replies: 26 comments
-
Posted at 2019-07-25 by maze1980 Where or what is the issue exactly? It's over 200 lines of code. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-25 by AkosLukacs Ok, so summarised the
, but different in Espruino, right? My guess is rounding is somewhat messed up, the interesting part:
So it's not exactly 14991360, but a bit smaller. In your
If you round it, you get what you expect:
That doesn't mean I have an explanation, probably a different behaviour with bitwise operations on fractions in Espruino. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-25 by AkosLukacs Yepp, it's rounding and bitwise operations, simplest repro is 🥁🥁 🥁🥁🥁:
Results:
Edit: with @robin's numbers:
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-25 by Robin Thr 2019.07.25 Thank you @AkosLukacs for the speedy reply. Thanks for testing on different OS and browser. At least I can be assured I'm not going insane. ;-) (yet) Always amazes me how each of us tackle problems. I'll take time to study the effort you put in. Still not solved though. After I posted and gave it some thought, I wondered if a floating point issue might be at hand. But if that be true, then Line L43 L44 in correct.js
would mean that the assignment of the same value is being corrupted. This would imply that the floating point value, of an identical value, is changing after the first assignment!!! |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-25 by AkosLukacs Nope no magic corruption:) Only different decimal precision Look at code at line 57-L60:
Going back from line 60-59-58: To demonstrate it (start with Add It will print |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-25 by Robin
Yes and now you have proved my point. We are masking bits with identical hex values. In either case the answer should be identical, as we are dealing with bits. Thats the reason for performing a mask operation, to detect which bit.
That case is using a floating point value for input. The example provided used the same integer equivalent of the hex notation. I left out the We have the node.js test, the Firefox browser test and my Chrome browser test, that produce correct identical results in these three examples. In this instance on one device using a different mechanism, we get different conflicting results. We shouldn't. I'm reading up at mozilla . . . > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators "Bitwise operators treat their operands as a sequence of 32 bits (zeroes and ones), rather than as decimal, hexadecimal, or octal numbers." "The operands of all bitwise operators are converted to signed 32-bit integers in two's complement format, except for zero-fill right shift which results in an unsigned 32-bit integer. " As these are numbers, it shouldn't matter if it is decimal or in hex notation. The browser tests show this. (a thought - aren't integers converted to floating point under the hood? hmmmm) I did notice that the mozilla site uses integer constants for their flags, rather than hex notation as I did. Could this be a difference between C/C++ and Javascript? Testing that . . . |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by AkosLukacs The point is -IMO - there is something wrong with floating point calculation. The bit-mask mismatch is a result of the floating point error. The result of calculation To demonstrate it, ran this in
Results are identical in non-Espruino environments:
Oh, btw just bit-shifting indeed wraps around in browser & node:
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by maze1980 Offtopic: Try 0.1+0.2 in nodejs(v10.15.1) and in Espruino (2v04.6). That's Javascript, and you should always keep in mind how numbers are represented internally. Ontopic (Espruino 2v04.6)
and one fix is to manually round the pow result:
Another fix is to use 524288 instead of Math.pow(2,19) for this calculation. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by Robin Fri 2019.07.26 Good evening AkosLukacs and maze1980 as I sit down for my morning coffee. Thanks for cranking out those calcs @AkosLukacs.
Yes, I agree. I remember removing the parseInt() wrapper as the integer result was not the same as the corresponding one compared to the E notation. So, how serious is a non compatible F.P. calculation engine or what acceptable error range? I haven't found any specification docs on the site yet. Thanks @maze1980 for the intermediate steps as a possible work around. I also, plucked all the toString(16) toString(2) parseInt() etc from the multitude of lines created trying to isolate this issue. The Mozilla docs indicate that for bit manipulation, 2's compliment is used. It is also indicated that only 1's and 0's are used, rather than numbers. So big question, what is the proper way to get from user input to only 1's and 0's? The original C++ line to convert:
C/C++ uses 64 bits in their shift calculations. But the divide sign bothers me in the above line. Still searching for docs on 2's compliment and using division. Doesn't F.P. get used (yes?) under the hood when dividing? If yes, then the above, although works on Arduino, seems rather pointless. Why go to the trouble of left shifting, maintaining the 1's and 0's only to introduce rounding error with F.P. division? Sadly absolutely no docs or examples with that source. (maybe the author realized that the lower LSB weren't really needed?) Since we are limited to 32 bits for bitwise calcs in Javascript, maybe an answer might be to somehow mask off the MSB just before left shift into no mans land, and shift that bit into a separate var container nineteen times, thus preserving 1's and 0's up to 64 bits? But, then how to properly handle the division? The author of the Arduino source must have had a reason to left shift nineteen times, rather than just do the calculation in F.P. wouldn't you agree? Is anyone able to confirm that only integer values, for both operands and flags, (maybe not hex either) should be used in bitwise calcs? In the mean time until the F.P. issue can be looked at, will just use constants to prove out the initial code conversion, but will need a solution for down the road in a ~month or so. Time for more research. . . . EDIT: I wonder if inline 'C' using E.compiledC will improve things? Just did a lookup for the Math.round() function
Just looking around and found the comment at L291 for Math.pow() VERY interesting. Maybe this is why the F.P. calcs are off? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by AkosLukacs
AFAIK all math operations are floating point operations in javascript & Espruino. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by Robin
I can't take the credit for that, @maze1980 did the groundwork in #9. (unless of course you are referring to comment at L291)
So original assumption correct, what is the best way to supply the operand to be able to perform this stunt in Javascript? All docs I've read so far indicate use of integers. But aren't integers a representation of what is store using F.P.? hmmmmm
But doesn't the divide convert from integer to F.P. for the calculation, introducing rounding error as you so kindly pointed out?
Part of a massive Arduino script I want to see if it possible to run using EspruinoWiFi's, so that those guys don't have all the fun!! ;-) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by Robin Fri 2019.07.26 While I am unable to verify if the methods are acceptable, these authors have solutions: It appears (x).toString(2); may be acceptable as is parseInt('100001', 2); https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions Good beginner article Division of a 2's complement value appears to be iterative and not a conversion to floating point. Nicely done visual on 2's complement division So, that might explain why division of an extremely large number may follow a bit shift. Still, we are stuck with just 32bits in Javascript.
Still stuck. This solution needs a workaround for nineteen left shifts. Original value is one third of max safe integer for bit manipulation. Shifting left will force MSB bits to drop off, as we have way larger than 32 bits. There would be 52 after the shift. See #1 hilight in red bit pattern. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-26 by maze1980 In JavaScript there is only "var" to define are a variable, and if it happens to be a number stored/converted to floating point for math operations. In C you have different data types, there are float and double for floating point numbers, and of course byte, integer, uint64_t etc. A byte is stored using 8 bits. And if you store the number 3 in your byte and divide it by the result is 1, not 1.5. The same is true for uint64_t. Compiled C should work with uint64_t. I wouldn't go this way and stick with JavaScript for simplicity. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-27 by Robin Sat 2019.07.27
Sort of agree with you but, . . . as @AkosLukacs and yourself have pointed out, and with the note in the link from #10 that I found, there is an issue (fewer identical upper MSB values-see your #9) with the floating point values compared with other means. This severely (in my need) affects accuracy where the LSB bits will be needed. My suggestion for an attempt with inline 'C' might bypass what the Javascript Math library does, which is dependent on floating point. Don't know if the inline 'C' functions might use the identical floating point mechanism at the hardware level. i.e. Math.pow() is the library layer on top of the Espruino Javascript layer on top of inline 'C' asm on top of the physical hardware layer itself.
Just thought of a possible hacky solution using the original Arduino script. As my design will require the end user to input one of only 20 to 50 valid values using this C++ function conversion, I could possibly write a small Arduino code file and build a table of those values that it puts out. Then using an array or switch case code block in Javascript, extract the hex bits already converted that way. This would however, make for headaches for future user update requests of the finished deployed module. Naaaahh, as a purist, I'll wait and see if we can find a Javascript solution. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-27 by AkosLukacs If you provide a link to a datasheet, description and the original source, we may be able to figure out what it wants to do, and help with a re-implementation. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-27 by maze1980 A re-implementation from the original C line
can be re-written to C code using floating point numbers
and then converted to JavaScript:
(0.016384 is equal to Math.pow(2, 19) / 32000000) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-27 by AkosLukacs I meant the original intention of the code, so maybe we can figure out a way to do it without floating point operations and rounds. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-28 by Robin Sat 2019.07.27 midnight Two topics to follow, Topic 1 Burning the midnight oil here it's 3am, but learned a ton! Thank you @maze1980 for yet another angle to approach. But thought about the solution on a rather long walk-about. There are four areas to observe in that process. See #10 2's compliment is used in original equation for bit shifting - just ones and zeros. Floating point numbers are made up of a mantissa and characteristic, both with round off error suspectability. Typecasting introduces yet another form of rounding. @AkosLukacs pointed out in his worked examples the rounding error after six digits (base 10) of accuracy. #3 and #4 This reveals the limitation of the floating point engine Espruino uses, after around 5-6 digits. The original equation required pow(2,19) - ~pow(2,~7.?) is approx ~pow(2,~11.?) is approx 0xFFx??? where the error creeps in. This means we need accuracy better than So, although the maths actually do work in that example,
it's the combination of rounding error that allows it to happen. It appears that attempting to use floating point to solve a 2's compliment equation is not the way to go as the need is, to shift ones and zeros and not a representation of a number using a mantissa and characteristic. The accuracy is just not there. And, sadly the kicker: 'Arithmetic with floats, doubles and uint64 may not work since it requires functions that won't be compiled in.' Fourth link below has a nice table for comparrison Table of bits float - 32 bits Nice visual
Sec 5 Nice pictorial Sec 6 Round off error in mantissa
Type casting in C |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-28 by maze1980 While I'm too lazy to test the result in JavaScript using a 64 bit math module vs. normal numbers, but in C it is like this:
If you want to see more than "Hello World", "Done" change
For me it doesn't look like there's any need to shift ones and zeros. Think about it. @AkosLukacs: |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-28 by Robin Sun 2019.07.28
@AkosLukacs perfomed the how proof in #3 and #4 I explained the why in #19
There isn't any bit masking in that example. It is a division proof using type casting in C. Will work on the I2C slave example you posted: Thank you for that suggestion |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-28 by Robin Sun 2019.07.28
@AkosLukacs, the source wasn't needed as I only requested a snippet of how Arduino masked bits. Too many over-lapping projects here. Was working on a tutorial for bitwise operators. It does appear that the source sent me is a kludge copy of that or a similar file. Source that isn't commented is always suspect. Seeing the same no comments there (link #20) also, now puts the kludge chunk sent me in question. So the formula does appear to have some physics/mathematical relationship for others to have used it. Bigger fish to fry. Attempting to rework a rather large source is the big show stopper right now. Found this as an alternate means using additional memory: Then on to having two devices talk back-n-forth, which I started here: Using a formula to mask bits seems iffy anyway. What if down the road there becomes an EspruinoWiFi Deluxe using a different processor and different/unique floating point engine that is different from the existing one. Any attempt to port on to the new device, would result in errors there also. Then, immediately I'd get the blame for crappy code. A constants table makes more sense. We'll wait for the manufacturer datasheet as you so pointed out. Using named indexes would allow the ability to 'screen' a user request. So, this appears to be the better solution, or an array of constants. Thank you for your assistance with the detective work. It has been an exciting exchange and a fantastic learning experience. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-28 by AkosLukacs Ah, now I got it. That's why I asked @robin what is this about. And reading the latest comment, turns out it's just a piece of random code... Sooooooo, originally Semtech gives users a formula in chapter 4.1.4! you don't need 64 bit integer, or more precise FP, just do the math, and multiply with 0.016384! as @maze1980 already pointed out. And now 915e6 makes sense as well: 915MHz frequency. Quick aside: a good compiler probably can optimize away the bit shift and division, so the programmer just leaves it there instead of manually calculating / simplifying the formula. And most likely easier to follow the code. Attachments: |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-29 by @gfwilliams Ok, so I'm a bit late to this as I had a long weekend, but it seems like this is basically solved now thanks to @AkosLukacs and @maze1980. It's unfortunate that Espruino gets very slightly different answers when doing some floating point maths, but if you're dependent on the exact value of a floating point number after multiplying and dividing it, it's probably a sign that the code should be working some other way. The differences in Espruino vs browser may actually be related to this bug: espruino/Espruino#1573 So the maths itself may well be identical, but the conversion of the actual string into a floating point number (and back again) may not be exactly the same as the desktop browsers do it. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-29 by maze1980 @robin: You didn't understood the example, #3 and #4 show something completely different. But I'm sure you'll find a way to get the right numbers somehow. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-29 by Robin Mon 2019.07.29
Thank you @gfwilliams, that confirms (in summary) my closing comment in #22 and that will be the approach I shall use.
Actually, the equation helped @AkosLukacs locate the inconsistency as in the link to bug 1573 that you pointed to. It was code I just copied across for demonstration. Wasn't even needed as I was using constants. If there is a conversion issue (bug 1573) in question for constants, that could explain more than what our weekend exchange uncovered. I'm after masking bits. The thread got side tracked with what the equation uncovered. Maybe there are two unique conflicting themes here. > 'but the conversion ... into a ... (and back again) may not be exactly the same' So that I may understand how floating point calculations are made under the hood, where is the better place to start on it's inclusion in Espruino? Is it done with Log tables? Where would they be defined? It just surprises me that the inconsistency compared with other techniques actually exists. I'd hate to think that bit masking in unreliable . . . . |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-29 by @gfwilliams
I pointed you to However conversion to/from Strings is done with Espruino's own functions (which happens in
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-07-25 by Robin
Thr 2019.07.25
This is a rather strange anomally, and not what was originally tackled.
Foundation:
Was in the process of converting this C++ script into Javascript:
C/C++ uses a long int 64 bits, so shifting 19 places will work, and it appears that the uint8_t typecast is being used to strip leading and trailing bytes.
Kept getting incorrect result. Had a hunch that left shifing 19 places might be an issue
Down a rabbit hole, . . . but the wrong one!
After manually creating binary strings
0001101101000100111001010110000000000000000000000000 / 0001111010000100100000000000
determined that the bits were getting sent to no mans land during left shift, as expected.
Verified how Javascript vars work:
"The MAX_SAFE_INTEGER constant has a value of 9007199254740991 (9,007,199,254,740,991 or ~9 quadrillion). The reasoning behind that number is that JavaScript uses double-precision floating-point format numbers as specified in IEEE 754 and can only safely represent numbers between -(253 - 1) and 253 - 1."
"They are 64-bit floating point values, the largest exact integral value is 253-1, or 9007199254740991 . In ES6, this is defined as Number.MAX_SAFE_INTEGER. Note that the bitwise operators and shift operators operate on 32-bit ints, so in that case, the max safe integer is 231-1, or 2147483647."
So the data should fit inside that value, as we are one third of the max value. Realized that as left shifting is nothing more than doubling, tried the same using
Math.pow()
Find that rabbit hole.
Process:
Wrote many string representation formatting functions to show the decimal values shown using the WebIDE debugger as both formatted hex and formatted binary to determine if it was in fact a shift issue.
It didn't appear to be, . . . until . . .
Spent two days attempting to understand why a previously working/tested format function all of a sudden started giving incorrect output. In a final act of desperation, started whacking out everything that wasn't needed from the 700+ lines of code. Still had the incorrect output.
Decided to make Html scripting pages to test. They tested fine. So where was the issue?
Found the rabbit hole.
Observation:
I made a copy of the original data in a new var assignment. An Viola! Found what was causing what appeared to be a bit shift issue. Still not quite sure, as the debugger doesn't reveal anything new on the right-hand editor side of the WebIDE. But it appears to be the assignment of an existing var!
The offending lines. They appear to be valid to me. See comment at top of source file for commented out lines. Files contain identical source, just more comments in 'incorrect' version.
Google's closure compiler shows zero errors
And with the two lines commented in:
Hoisting doesn't appear to be an issue, nor does redefining:
"If you re-declare a JavaScript variable, it will not lose its value."
Works in Chrome, either .htm files demonstrate this.
Spending as much time on this as I have, I would really like to understand where in the source this culprit exists. I'll try a few attempts at renaming the var declaration and assignment, to see if that may be part of the issue.
The two lines commented out are indicated in each source file at page top.
Attachments:
Beta Was this translation helpful? Give feedback.
All reactions