Support fixed-point math

[stonemonk]

You want the most precision possible when dealing with a users money. In particular, methods for placing orders and doing deposits/withdrawals should accept decimal.Decimal for the relavant arguments, return values and should be used internally for any calculations they perform.

[kroitor]

Thanks, for your suggestion! We agree completely! As you may know, this library is under active development. Switching to fixed point math is on our roadmap. Related issue: #393.

By the way, your PRs and contributions are welcome! We will be happy if you could help us with that. But please note that apart from Python we support a few other languages as well. So, we have to make it portable. Therefore this is a little more involved than just switching to decimal.Decimal (more about it).

We hope to deliver it soon. Will let you know as soon as fixed-point math is integrated into it.
Stay tuned!

[stonemonk]

So I was proposing that CCXT switches from python float (which being backed by C double type, is 64bit on nearly every platform) to decimal.Decimal which is 128bit and would be suffient precision for working with currencies and is a more natural way of working with inherently floating point amounts (compared to trying to convert to fixed point).

I take it that you have a preference for using fixed point math, but this seems like it could be challenging if any calculations are required and because different symbols may require a different base unit. For BTC based symbols, you could use https://en.bitcoin.it/wiki/Satoshi_(unit) ints for example, but then what about symbols based on ETH or USD?

Wondering about your general thoughts about how to approach this topic. I dunno about the transpiling and other langs, but to use decimal.Decimal in the native python version would be pretty simple to switch to.

[kroitor]

I take it that you have a preference for using fixed point math, but this seems like it could be challenging if any calculations are required and because different symbols may require a different base unit. For BTC based symbols, you could use https://en.bitcoin.it/wiki/Satoshi_(unit) ints for example, but then what about symbols based on ETH or USD?

We avoid rate conversions inside the library, therefore we don't really care about the base unit. I think we should approach it mathematically, not financially. Therefore a fixed-point solution seems to fit our needs better than big integers. Whatever way we resolve this, we still believe that the fix should be independent of any particular currency.

I dunno about the transpiling and other langs, but to use decimal.Decimal in the native python version would be pretty simple to switch to.

Not really, because there's a lot of exchanges (up to a half) that do return fp-numbers in JSON... So we will have to convert from float to decimal.Decimal, and that in itself introduces some problems due to float-rounding and precision issues.

[stonemonk]

Therefore a fixed-point solution seems to fit our needs better than big integers.

I don't understand this, "fixed-point" and "big integer" mean the same thing to me. Fixed-point normally refers to use of integer math by working with numbers at a certain scale. e.g. multipling all prices and quantities by 10 million, so you can represent it as a big integer and thus avoid using floating points and thereby avoiding float precision issues. I was suggesting that this would be difficult because you might not be able to find an appropriate base scale for all currencies (and for the quanitity/amount field for orders!)

Personally I would stick to a floating point solution, but just upgrade the datatype from a 64bit float to 128bit decimal.Decimal.

Not really, because there's a lot of exchanges (up to a half) that do return fp-numbers in JSON... So we will have to convert from float to decimal.Decimal, and that in itself introduces some problems due to float-rounding and precision issues.

You can work around this easily. The standard JSON parser has a way of customizing the float handling, specifically:

response.json(parse_float=lambda f: f)

This tells it to not try to parse unquoted floats as float, but rather just return it as a string. Then you can just call float(str_dec_field) or decimal.Decimal(str_dec_field) to preserve the original precision. Or if you want fixed point math, something like this:

scale = decimal.Decimal("100000000")
int_field = int(decimal.Decimal(str_dec_field) * scale)

If you wanted, you could also do that with string manipulation (detect and shift the decimal point character in the string, then convert directly to int) to avoid the use of decimal.Decimal completely. But again I would urge you to stick with floating point math and just upgrade the precision by using a 128bit floating point type.

[kroitor]

I don't understand this, "fixed-point" and "big integer" are the same to me.

I may be using the wrong term here, but what I mean under "fixed-point" is being able to do the following string-wise operation:

"0.0000000000000000000000000000000000000000000000000000000001"
x
"1234567891234567891234567891234567891234567891234567891234.0"

So we don't have to choose the base unit or to convert to integers or whatever. The only thing we need is an arbitrary-precision arithmetics util. We don't really care whether it stores those values as integers inside the util or not, the only thing we care about is getting adequate string representations of arbitrary-precision floats and having the ability to do basic arithmetics with those representations. To me, the underlying mechanics looks like something that isn't based on integers solely. We don't need BigInts, we want BigNumbers.

Fixed-point normally refers to use of integer math by working with numbers at a certain scale. e.g. multipling all prices and quantities by 10 million, so you can represent it as a big integer and thus avoid using floating points and thereby avoiding float precision issues.

We believe, that mathematically, having a solution for arbitrary-precision numbers, we don't need to choose the scaler. This really should be currency-agnostic. The added complexity for arithmetics will definitely slow us down, so our main concern here is the efficiency of the math-util in question.

[stonemonk]

Ahh I see, I hadn't even considered that option. Well that would certainly work if you can find a suitable library. But using 128bit decimals would also be sufficient and is a built in native for python so it seems like a more natural choice. I guess JS and PHP do not have 128bit float support built in though.

Maybe this is a suitable JS lib which could be transpiled?: https://github.com/MikeMcl/big.js/

It might be best to always use strings for CCXT API input/output as to hide the underlying details, thus allowing the user to chose a suitable representation in their own code.

You mention efficiency, but I don't think its a concern as you only really want to use it when dealing with a users money (create orders, fetch orders, fetch balances). For tickers, candles, etc there is no real reason to use higher precision.

[stonemonk]

Just want to say that I was incorrect when I said that Python's built in decimal.Decimal type was 128bit (I was thinking about the mongo Decimal type).

decimal.Decimal is in fact arbitrary precision (defaults to 28 decimal places but is adjustable), and would thus be a more reasonable choice for the Python version of CCXT than using a third party library.

[kroitor]

@stonemonk , we have a concern on decimal.Decimal, because from Py2 docs it looks like it uses a global context in a dirty manner. If there's no other way to specify rounding/truncation rules, that global context might be a showstopper:

What do you think of the above?

[stonemonk]

I am not working in Py2, but I don't think it's a problem. Any body else's code using decimal should be setting precision already prior to use. But if you are concerned about this you can always save the original precision first then restore it afterwards.

If the quantitize() method works in Py2, you might be better off using the default context precision of 28 decimals and only quantitizing down when needed..

[kroitor]

@stonemonk looks like we can use decimal.Decimal in both Pythons with default precision + quantize, it should cover all values, hopefully. Thx!

[Bomper]

Have you considered using BigNum for the JavaScript library? Floats are dangerous (precision loss) and strings lead to concatenation instead of addition (e.g. order.size + 10 may be "3010" if order.size was "30").

[kroitor]

@Bomper yes, in fact, all languages are subject to that, so we are preparing a generalized solution to this issue. Will let you know upon progress.

[gaardiolor]

Would realy like this to be implemented too, using float all my calculations go wrong. Now I have to cast al floats to decimals, and back to float again when trading...

[gaardiolor]

python module simplejson supports decimal import / exports. Easy workaround:

import ccxt
import simplejson as json

e = ccxt.bittrex()
decimal_order_book = json.loads(json.dumps(e.fetch_order_book('ETH/BTC')), use_decimal=True)

@kroitor this should be pretty easy to implement in ccxt (at least for python sync / async) ?

• use simplejson instead of json
• make use_decimal configurable

[kroitor]

@gaardiolor if I do this without changing all the code that works with precision for order submission, withdrawal submission, not only the parsing, but also all other aspects of the library, this would mostly break it unusable. There isn't an "easy one-liner" to fix for this, otherwise I would have done it three times already. It requires a more thorough design, and this is the core code of the lib, therefore it has to be designed with double care. We need to have base methods for arbitrary precision arithmetics first.

[gaardiolor]

OK, totally trust your judgement of course, I'm pretty new to programming. Awesome piece of work this library.

[damian123]

Floats should never be used for money but ccxt forces it, will this task be finished?

[frosty00]

@damian123 you can use exchange.number = str or exchange.number = decimal.Decimal it is almost finished

[timolson]

Note that some *_exchanges_* send (and use!) floats, and the conversion to e.g. Decimal still does not guarantee correct accounting in such cases.

…

On Thu, Mar 31, 2022 at 4:52 PM Carlo Revelli ***@***.***> wrote: @damian123 <https://github.com/damian123> you can use exchange.number = str or exchange.number = decimal.Decimal it is almost finished — Reply to this email directly, view it on GitHub <#522 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM6KWZH4VDWGRQMMXFXA6DVCYGB7ANCNFSM4EDABUAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pietrodn]

In what sense do some exchanges use float? The REST APIs use JSON numbers, that can be (de)serialized as Decimal, maintaining precision.

The only issue would be if they used binary-encoded IEEE 754 numbers, but I don’t know if any exchange that does that…

[kroitor]

@pietrodn it's more about the performance of native JSON parsers like JSON.parse() vs a custom parser that would be required to treat JSON numbers as text-strings, which is why we just quote long numbers and feed them into the native parser as quoted strings.

[timolson]

The REST APIs use JSON numbers, that can be (de)serialized as Numeric,

maintaining precision. A JSON number is either an integer or a float, so there is the possibility of roundoff error if there is any use of JSON Number anywhere at all, from the exchange, to the transmitted packet, to ccxt. Any cast to float will poison the data with inaccuracy. My point was that there are cases when this isn't even ccxt's fault. IIRC Bittrex, for example, was computing their fees using floats last time I checked. If you try to match their calculations locally, there's a dependence on the rounding mode of IEEE floats :clownface: If the exchange sends the price as a float, for example, you might end up with 1234.5599999999999999998 instead of 1234.56, for example. You can't fix the exchanges but you can use ccxt's metadata about market precisions to round prices, amounts, etc. to the correct values.

…

On Thu, Mar 31, 2022 at 5:58 PM Igor Kroitor ***@***.***> wrote: @pietrodn <https://github.com/pietrodn> it's more about the performance of native JSON parsers like JSON.parse() vs a custom parser that would be required to treat JSON numbers as text-strings, which is why we just quote long numbers and feed them into the native parser as quoted strings. — Reply to this email directly, view it on GitHub <#522 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM6KW47FP424OYCGNYEDXTVCYNXFANCNFSM4EDABUAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[frosty00]

@timolson the exchanges sends a json string that contains numbers. At this point the numbers are still strings, we simply feed those strings to the .number constructor without parsing it as a float first, hope that clears things up for you!

[gshpychka]

@damian123 you can use exchange.number = str or exchange.number = decimal.Decimal it is almost finished

I've tried it, but I'm getting an error when trying to create an order with Decimal price and size:

File "/usr/local/lib/python3.9/site-packages/ccxt/okcoin.py", line 2071, in create_order
    request['price'] = self.price_to_precision(symbol, price)
  File "/usr/local/lib/python3.9/site-packages/ccxt/base/exchange.py", line 2952, in price_to_precision
    return self.decimal_to_precision(price, ROUND, market['precision']['price'], self.precisionMode, self.paddingMode)
  File "/usr/local/lib/python3.9/site-packages/ccxt/base/decimal_to_precision.py", line 39, in decimal_to_precision
    assert(isinstance(precision, float) or isinstance(precision, numbers.Integral))
AssertionError

@frosty00 passing this function without the assertions to exchange seems to work. Maybe a good idea to remove them, or account for Decimal?

[gshpychka]

@kroitor would it make sense to adjust that check to support Decimal?

[gshpychka]

@ttodua what's required to progress this issue? Right now the error above is the only thing that's keeping me with using Decimal in Python.

[samgermain]

@ttodua what's required to progress this issue? Right now the error above is the only thing that's keeping me with using Decimal in Python.

I'll push a PR later today that does a lot of the changes, and after that, for all the exchanges that you want to use, I would make sure that all the arithmetic on prices uses methods like Precise.stringAdd, Precise.stringMul... instead of +,* ...

[samgermain]

Changes like the ones in this PR need to be made across all exchanges. I can get all the one that use Math. pretty easily by using a search, but catching all the +, -, * / could be a bit tricky and might require some Beta use

[gshpychka]

@samgermain not sure I understand completely - assigning Decimal to exchange.number already means that number operations will be fixed-point, doesn't it? It seems to work fine, but the error I posted above is what's still broken.

[gshpychka]

@samgermain can you explain?

[samgermain]

@samgermain can you explain?

Don't worry about it, I've been working on adding string math across the library, should hopefully be done soonish

[gshpychka]

@samgermain can you explain?

Don't worry about it, I've been working on adding string math across the library, should hopefully be done soonish

The link leads to a 404 for me.

Can you address my question above - assigning Decimal to exchange.number already enables fixed-point math, but the error I posted above is still broken.

[gshpychka]

@samgermain can you please address the question?

[samgermain]

@damian123 you can use exchange.number = str or exchange.number = decimal.Decimal it is almost finished

I've tried it, but I'm getting an error when trying to create an order with Decimal price and size:

File "/usr/local/lib/python3.9/site-packages/ccxt/okcoin.py", line 2071, in create_order
    request['price'] = self.price_to_precision(symbol, price)
  File "/usr/local/lib/python3.9/site-packages/ccxt/base/exchange.py", line 2952, in price_to_precision
    return self.decimal_to_precision(price, ROUND, market['precision']['price'], self.precisionMode, self.paddingMode)
  File "/usr/local/lib/python3.9/site-packages/ccxt/base/decimal_to_precision.py", line 39, in decimal_to_precision
    assert(isinstance(precision, float) or isinstance(precision, numbers.Integral))
AssertionError

@frosty00 passing this function without the assertions to exchange seems to work. Maybe a good idea to remove them, or account for Decimal?

I created a pull request that should fix your specific issue. If you run into any more issues regarding this topic please let me know and I can try to solve them.

Using decimal.Decimal will not work on every exchange yet. There are a number of PR's that need to be merged before that will be possible.

• bit2c bit2c.createOrder string math #14893
• bitfinex pro Bitfinex pro parse trade string math #18742
• bitfinex pro Bitfinex pro string math #17336
• bitfinex2 Bitfinex2 string math #18415
• bitflyer bitflyer.parseTransaction string math #18728
• bitmex bitmex fetchLedger string math #14719
• bitmex bitmex.parseLedgerEntry string math #17092
• bittrex bittrex.createOrder string math #14901
• cex cex Math -> Precise #14858
• exmo exmo.fetchCurrencies string math #16813
• huobijp Huobijp parseTrade string math #18744
• idex Idex string math #18894
• itbit itbit.parseTrade string math, safeTrade #16727
• mercado mercado.createOrder string math #17098
• novadax novadax.createOrder string math #17331
• okcoin okcoin.createOrder string math #17096
• probit probit.createOrder string math #18730
• tokocrypto Tokocrypto string math #18413
• upbit Upbit decimal #14873
• wavesexchange wavesexchange string math #18245
• zb Zb decimal #17337
• binance binance.createOrder market buy string math #14892
• bitfinex2 bitfinex2 parseTransaction string math #15000
• bitmart bitmart.createOrder string math #14895
• bitstamp Bitstamp fetchOrder string math #15019
• bitvavo bitvavo.createOrder string math #14903
• bl3p Bl3p precise #14904
• btcalpha btcalpha.createOrder string math #15015
• bybit bybit createSpotOrder & parseLedgerEntry string math #14915
• coinbase coinbase.parseLedgerEntry string math #15020
• currencycom currencycom.fetchMarkets string math #15017
• exchange.calculate_fee exchange.calculateFee string math #15028
• exmo Exmo string math #15003
• huobi huobi.createOrder market buy string math #14930
• huobijp Huobijp string math #14875
• independentreserve independentreserve.parseOrder string math #15016
• lbank2 lbank2.createOrder market buy string math #14980
• liquid Liquid precise #15007
• luno luno.parseLedgerEntry Math -> Precise #14860
• mexc mexc.createSpotOrder string math #14931
• ndax Ndax parseLedgerEntry string math #15001
• okx okx.createOrder market buy string math #14966
• phemex phemex.createOrder market buy string math #14976
• therock therock.parseOrder string math #15002

After all these PRs are merged, and after finding any remaining bugs, using decimal precision will be an option across the library

[samgermain]

@samgermain can you please address the question?

The fix to your problem was merged, so you should be able to use the Decimal type with okcoin. Using the Decimal type will not be completely possible for all exchanges however until the PRs listed above get merged.

[samgermain]

PRs related to this issue are linked in https://github.com/orgs/ccxt/projects/52

Once all PRs in that project are merged, bugs related to this issue will be intermittant, and solved as they are discovered, but string math should be universally available