Big number #1784

Closed
Joris-van-der-Wel opened this Issue Sep 28, 2011 · 18 comments

10 participants

@Joris-van-der-Wel

I often run into javascript's lack of 64 bit support.

There are a few 64 bit / big number modules on NPM. And there has been some talk about it on the ecmascript technical committee.
http://wiki.ecmascript.org/doku.php?id=proposals:numbers&s=decimal
http://wiki.ecmascript.org/doku.php?id=harmony:binary_data_discussion&s=int64

In my opinion a big number implementation should be part of the core node.js, for a few reasons:

  • efficiency, arithmetic could be done in C (I am not sure if a node module could do this)
  • easier to implement, there are plenty of C libraries providing arbitrary arithmetic, it would just be a matter of exposing an API. Such as http://gmplib.org/
  • to make sure modules big numbers in their API do it in a consistent way
  • the issue comes up in almost all my node.js projects (I often work with databases or RPC to C applications)
  • I can imagine people expecting something like this to be supported by default

I am willing to spend time on this, if something can be agreed upon.

Thanks

@felixge

-1 from me. I would love to see good support for this, but the node core seems like the wrong core for it. For now npm modules seem to be the correct stop-gap solution until JS as a language addresses this problem.

@bobrik

Working with really big numbers with gmp is faster, but if you want to use gmp or something like this for every number in your code, there will be performance problems.

Using C modules is fast enough if you really need it.

@Joris-van-der-Wel

@felixge That was my original preference. My issue is that I do not see this being part of ecmascript and V8 anytime soon. As far as I can google, there are unfinished proposals for things like 64 bit integers and decimal types, but not arbitrary precision. I personally only need 64 bit, but even this seems years away.

@mikeal
Node.js Foundation member

I'm +1 on adding a bignum object and -1 on using the ECMA-Harmony spec directly.

I'd rather migrate off our bignum when it's implemented in v8 than deal with any inconsistencies we might have in our implementation compared to theirs.

This is one of the reasons why Mozilla and webkit namespace draft versions of new spec implementations in their browsers.

@tj
tj commented Sep 28, 2011

v8 has src/bignum.cc, not sure if we can just use that or not

@trevnorris

IMHO, should be handled by library.

@TooTallNate

This is already handled in a number of different ways by multiple npm modules. Additionally, I'm not sure how node-core could make the situation any better without extending the language. Therefore: closing.

@seishun

The problem with modules is that you cannot (easily) work with them from C++. For example, you write a library that generates or unpacks a 64-bit integer and needs to pass it in a JS object.

protobuf is one example of such a library - currently, it represents 64-bit integers as decimal strings, which is fine for most cases, but creates problems when you need to do bitwise operations on them.

Ideally, an int64 type could be implemented similarly to Buffers - stored internally as an actual 64-bit integer and provide convenience methods for conversion to/from various JS representations, such as a decimal/hex string or a Buffer.

@trevnorris

@seishun, I'm not sure I understand why something like this couldn't be implemented in a module? Like you said, you can't pass back a 64-bit integer, but you could pass back a Buffer containing them. One possible solution would be to implement your own ArrayBufferView that wraps around a Buffer and internally performs 64-bit integer operations. Take a look how this is done in src/v8_typed_array.cc.

I imagine the syntax would look something like the following:

var buff = new Buffer(0xF8),
    view = new Int64Array(buff);
view.toHex(<idx>);
view.fromHex(<value>);
@TooTallNate

My ref module has read and write Int64 functions as well. Used in conjunction with ref-array, you have your new Int64Array(buf) class.

@seishun

@trevnorris, I think you misunderstood the problem. Sure, I can pass back a buffer, but working with a buffer is unwieldy in most cases, so I pass back a string, which is the lesser of two evils. As for implementing an ArrayBufferView, I'm not sure how it could work as a generic solution. Do you suggest every module that needs to return an int64 to implement ArrayBufferView?

@TooTallNate, I'm aware of the ref module's read/write int64 functions, and I use them when I need a buffer, rather than a string, representation of a 64-bit int. But like I said in the previous post, these kinds of conversions are inconvenient. Every modern language has a type for raw data and for 64-bit integers. Node.js has filled in the first blank (instead of forcing users to work with binary strings) and I don't see the reason why it shouldn't be the same for the second one.

@TooTallNate

@seishun We use the V8 vm as-is without modifications. Supporting 64-bit integers in the JavaScript language would require changes to the language itself and we're not in the business of doing that. Once ES6 (or ES7) defines 64-bit integers and V8 implementes them, then node.js will support them natively. Until then, one of these userland solutions is your only option.

@seishun

@TooTallNate, sorry if I wasn't clear. My suggestion is not to extend the existing Number type, but to provide a separate class (let's call it Int64 here) that would wrap a long long (similarly to how Buffer wraps a char*) and expose itself to JS as an object. One could, for example, construct it from a string (var myInt = new Int64('76561197970532372');) and have convenient access to various operations with it, e.g. var hex = myInt.toString(16);.

Of course, this could be done in a JS module that would store the int64 as, for example, a buffer. But the advantage of having this as core Node.js is that the Int64 class would be accessible from a C/C++ addon, allowing one to expose a long long to JS with something like Int64::New(76561197970532372LL).

I hope I'm not hugely misunderstanding something basic about Node.js here.

@TooTallNate

But the advantage of having this as core Node.js is that the Int64 class would be accessible from a C/C++ addon

Meh, this is easy enough already just using Buffer instances IMO (warning, untested, also doesn't verify length of the input Buffer):

Local<Value> buf = args[0];
char *raw = Buffer::Data(buf);
int64_t *val = reinterpret_cast<int64_t *>(raw);
// now you can read/write to *val
@seishun

sigh I'm not arguing that there is no way to pass an int64 from C/C++ to JS - buffer and string are two possible ways. I'm arguing that both of these representations are hacky, a pain to convert between and lacking.

Sure, a third-party module could solve all of the above. But then what should I do if I'm writing a module that needs to return an int64? If I return a string and the user uses that module, we'll have a needless conversion to a string and back. If I return a buffer, the user will be inconvenienced if all they want is to print out the value to the console.

One might argue that this is not enough of a reason to include 64-bit integers in core Node.js. But Node.js has done a great job filling in the blanks of JavaScript - providing file system access, sockets, a binary data container etc. I don't see how 64-bit integers don't belong with these.

@trevnorris

@seishun Could you give some examples how you expect the API? If you're looking for the following:

var myInt64 = <int64_value>;

Then you'd be asking to extend v8, not node. You did mention adding an Int64 class. This is already being defined. So the specification is still being worked out. If someone wants to create a module with this functionality, then feel free.

I have to disagree with your observation that the "problem with modules is that you cannot (easily) work with them from C++". While it's not documented, you can create and overwrite any amount of functionality within a module. For example in the oef module I overwrite the default OnFatalError function that node sets.

Basically, this is definitely something that can be done in userland. And until the spec is completed, it isn't going to be built into node.

@vicary

+1 for waiting the ES6 and V8's native support. Currently I am shifting calculation works into database, and stay with strings in nodejs at the moment.

Any historical hints about how fast Google will catch up with the standards?

@bnoordhuis
Node.js Foundation member

Any historical hints about how fast Google will catch up with the standards?

Reasonably fast, on the order of months. A new feature is usually hidden behind command line flags until the feature shows up in browsers (cf. Harmony proxies).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment