New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[10 mBTC Bounty] General metadata improvements #1544

Open
npomfret opened this Issue Jun 21, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@npomfret
Contributor

npomfret commented Jun 21, 2017

I've just tried executing a trade on the BTC/LTC market on hitbtc which failed with the message 'Rounding necessary'. Now it seems that LTC and BTC are represented on the 8 exchange with 8 decimal places, as is the price on the BTC/LTC exchange. However, the quantity that is tradable on that market is rounded to just 1 decimal place. Is there a common place in this API (like the .json metadata files) to specify that value?

I was thinking something like:

{
  "currency_pairs": {
    "LTC/BTC": {
      "price_scale": 8,
      "quantity_scale": 2
    }
  }

  ...

  "BTC": {
    "scale": 8
  },
  "LTC": {
    "scale": 8
  },
@timmolter

This comment has been minimized.

Show comment
Hide comment
@timmolter

timmolter Jun 21, 2017

Member

The metadata may or may not have specifically that, I don't know for sure as I've not yet used that specific feature. The json could also very well be outdated. You'll have to investigate it yourself unless someone else chimes in here.

Member

timmolter commented Jun 21, 2017

The metadata may or may not have specifically that, I don't know for sure as I've not yet used that specific feature. The json could also very well be outdated. You'll have to investigate it yourself unless someone else chimes in here.

@npomfret

This comment has been minimized.

Show comment
Hide comment
@npomfret

npomfret Jun 22, 2017

Contributor

Thanks for the reply - I'll give it some thought.

Contributor

npomfret commented Jun 22, 2017

Thanks for the reply - I'll give it some thought.

@npomfret npomfret closed this Jun 22, 2017

@molexx

This comment has been minimized.

Show comment
Hide comment
@molexx

molexx Jun 24, 2017

I'd be interested to know any progress/thoughts on this, thanks.

molexx commented Jun 24, 2017

I'd be interested to know any progress/thoughts on this, thanks.

@npomfret

This comment has been minimized.

Show comment
Hide comment
@npomfret

npomfret Jun 25, 2017

Contributor

I'm quite new to this project, so my ideas might not be quite thought out enough yet...

It feels like in order to trade on any exchange, each needs some common metadata. Specifically, so far I've come across:

currency scale - for use when getting your wallet data
price scale - for use when making a limit order
quantity scale - for use when setting the quantity/size of any order (which doesn't seem to be represented anywhere)

And I think these should be present in the exchange metadata for all exchanges.

fees - these vary from exchange to exchange. Some have taker v's maker fees, some have none at all (coin floor for example). They are usually expressed as a percentage of trade size. But some exchange have more complex models, with discounts for example. Some have funding fees, as well as trade fees. And sometimes fees vary based on what's being traded (as well as the trade size).

I think maybe the fees are a bit trickier to model and perhaps should be removed from the static data al together?

I would be very useful if all trades returned by getTradeHistory would give amounts without fees applied and so the user can apply the fees separately. So far I've sound some implementations that do as I've described, and some that give amounts net (after fees have been removed from the trade value). This inconsistency makes it difficult to do accounting.

In addition, we also may have:

rate limits - to prevent a client spamming the exchange with requests. I think these are typically easy to model and probably belong in the static data.

I've discovered that each sub-module has a exchange-name.json at the root of the class path that contains the static data. It would have been helpful (for a beginner like me) if this was documented in the readme.md file. I've had to create my own copies of almost every one I've used because they're out of date. And also, I've discovered that some exchanges provide some of this metadata in the api, and those cases the static data can be overwritten by the dynamic data.

I wonder if the static data, which seems to change quite frequently might be best kept in a separate repository? That way they could be updated frequently by this community and users would have to wait for a release.

Just some thoughts... feel free to ignore me :)

Contributor

npomfret commented Jun 25, 2017

I'm quite new to this project, so my ideas might not be quite thought out enough yet...

It feels like in order to trade on any exchange, each needs some common metadata. Specifically, so far I've come across:

currency scale - for use when getting your wallet data
price scale - for use when making a limit order
quantity scale - for use when setting the quantity/size of any order (which doesn't seem to be represented anywhere)

And I think these should be present in the exchange metadata for all exchanges.

fees - these vary from exchange to exchange. Some have taker v's maker fees, some have none at all (coin floor for example). They are usually expressed as a percentage of trade size. But some exchange have more complex models, with discounts for example. Some have funding fees, as well as trade fees. And sometimes fees vary based on what's being traded (as well as the trade size).

I think maybe the fees are a bit trickier to model and perhaps should be removed from the static data al together?

I would be very useful if all trades returned by getTradeHistory would give amounts without fees applied and so the user can apply the fees separately. So far I've sound some implementations that do as I've described, and some that give amounts net (after fees have been removed from the trade value). This inconsistency makes it difficult to do accounting.

In addition, we also may have:

rate limits - to prevent a client spamming the exchange with requests. I think these are typically easy to model and probably belong in the static data.

I've discovered that each sub-module has a exchange-name.json at the root of the class path that contains the static data. It would have been helpful (for a beginner like me) if this was documented in the readme.md file. I've had to create my own copies of almost every one I've used because they're out of date. And also, I've discovered that some exchanges provide some of this metadata in the api, and those cases the static data can be overwritten by the dynamic data.

I wonder if the static data, which seems to change quite frequently might be best kept in a separate repository? That way they could be updated frequently by this community and users would have to wait for a release.

Just some thoughts... feel free to ignore me :)

@timmolter

This comment has been minimized.

Show comment
Hide comment
@timmolter

timmolter Jun 26, 2017

Member

Thanks, @npomfret . I think I may put some of what you wrote in the Wiki. I'll comment more on it soon...

Member

timmolter commented Jun 26, 2017

Thanks, @npomfret . I think I may put some of what you wrote in the Wiki. I'll comment more on it soon...

@molexx

This comment has been minimized.

Show comment
Hide comment
@molexx

molexx Jun 27, 2017

Yes a separate repo with the json files in sounds like a good idea. As they are loaded from the classpath it shouldn't require any changes other than moving the files around in git and adding a mvn module.
I like the current method (documented in the wiki) of merging files with data pulled from online APIs. It would be good for data to be removed from the .json if an API source is implemented.

Maker/taker fees could be done by splitting CurrencyPairMetaData.tradingFee into two e.g. makerTradingFee and takerTradingFee. The change could be made non-breaking by having tradingFee pick one to return but I'd probably be in favour of a breaking change in order to have accurate data.
Tiered fee levels sound complex to model but yes it does seem to be common.

Deposit/withdraw fees are on the currency rather than the currencypair and are usually a fixed amount (e.g. 0.0001 BTC). Maybe for some currencies there's a percentage-based fee too but I can't remember seeing one other than for fiat... which gets quite complex with different payment providers (OKPay etc.) and so might be easier to leave out at least initially.
If fixed amounts are all we need for now it should be straightforward to add something like depositFee and withdrawlFee to CurrencyMetaData.class.
Most (all?) currencies have a 'minimum network fee' for sending funds, some exchanges just quote that as the withdrawl fee (so don't actually charge any extra for themselves) and some add a bit on top. Maybe we'd like to model those two separately?

Public and private rate limits are I think already modelled fully in ExchangeMetaData.class but of course it's a matter of filling the .json data for every exchange.

I don't know what any of the 'scaling' values mean or are/should be used for, please could I have a quick description? :-)

molexx commented Jun 27, 2017

Yes a separate repo with the json files in sounds like a good idea. As they are loaded from the classpath it shouldn't require any changes other than moving the files around in git and adding a mvn module.
I like the current method (documented in the wiki) of merging files with data pulled from online APIs. It would be good for data to be removed from the .json if an API source is implemented.

Maker/taker fees could be done by splitting CurrencyPairMetaData.tradingFee into two e.g. makerTradingFee and takerTradingFee. The change could be made non-breaking by having tradingFee pick one to return but I'd probably be in favour of a breaking change in order to have accurate data.
Tiered fee levels sound complex to model but yes it does seem to be common.

Deposit/withdraw fees are on the currency rather than the currencypair and are usually a fixed amount (e.g. 0.0001 BTC). Maybe for some currencies there's a percentage-based fee too but I can't remember seeing one other than for fiat... which gets quite complex with different payment providers (OKPay etc.) and so might be easier to leave out at least initially.
If fixed amounts are all we need for now it should be straightforward to add something like depositFee and withdrawlFee to CurrencyMetaData.class.
Most (all?) currencies have a 'minimum network fee' for sending funds, some exchanges just quote that as the withdrawl fee (so don't actually charge any extra for themselves) and some add a bit on top. Maybe we'd like to model those two separately?

Public and private rate limits are I think already modelled fully in ExchangeMetaData.class but of course it's a matter of filling the .json data for every exchange.

I don't know what any of the 'scaling' values mean or are/should be used for, please could I have a quick description? :-)

@npomfret

This comment has been minimized.

Show comment
Hide comment
@npomfret

npomfret Jun 28, 2017

Contributor

WRT the fees modelling...

I agree that funding (deposit and withdrawal fees) are probably best ignored by this project, at least for now.

And I think to model the fees better then separating taker from maker would be a good step forward. In addition I think we need per currency-pair fees as some exchanges have different fees in different markets.

And... the fees are sometimes issued in the base currency, and sometimes the counter currency. Sometimes the user can choose which (and it's a port of your account settings).

@molexx the 'scaling' parameters are really important. If you don't scale your values properly the exchanges will reject your api calls. The error messages will say something like "needs rounding" or "price too accurate". Basically if you send a value at the wrong precision the exchange will either round it or reject it - I think its common practice to reject it.

So we have

  • currency-scale - for use when depositing and withdrawing currency. Possibly you'd also need use this to calculate your fees accurately
  • price-scale - for use when trading
  • amount-scale - doesn't exist in this project yet. Also for use when trading to set the quantity on your order. Maybe tick-size would be a better name. In some markets the trade size / quantity needs to be divisible by 100 or even 1000.
Contributor

npomfret commented Jun 28, 2017

WRT the fees modelling...

I agree that funding (deposit and withdrawal fees) are probably best ignored by this project, at least for now.

And I think to model the fees better then separating taker from maker would be a good step forward. In addition I think we need per currency-pair fees as some exchanges have different fees in different markets.

And... the fees are sometimes issued in the base currency, and sometimes the counter currency. Sometimes the user can choose which (and it's a port of your account settings).

@molexx the 'scaling' parameters are really important. If you don't scale your values properly the exchanges will reject your api calls. The error messages will say something like "needs rounding" or "price too accurate". Basically if you send a value at the wrong precision the exchange will either round it or reject it - I think its common practice to reject it.

So we have

  • currency-scale - for use when depositing and withdrawing currency. Possibly you'd also need use this to calculate your fees accurately
  • price-scale - for use when trading
  • amount-scale - doesn't exist in this project yet. Also for use when trading to set the quantity on your order. Maybe tick-size would be a better name. In some markets the trade size / quantity needs to be divisible by 100 or even 1000.
@timmolter

This comment has been minimized.

Show comment
Hide comment
@timmolter

timmolter Jul 2, 2017

Member

@npomfret @molexx Thanks for your thoughts on the matter.

It would have been helpful (for a beginner like me) if this was documented in the readme.md file.

https://github.com/timmolter/XChange/wiki/Design-Notes

See section on metadata. I should probably add an easy-to-see link in the README.

I wonder if the static data, which seems to change quite frequently might be best kept in a separate repository?

Well, the develop branch is more or less that. Plus the code is also changing so much chasing changes on exchanges' API. Every time I do a release there are hotfixes and additions literally the next day. :) It's tricky because there is no real way to run integration tests on all the code because we're dealing with live exchanges and API keys to private accounts.

I've had to create my own copies of almost every one I've used because they're out of date.

Please contribute your changes, as that's the only way we can hope to keep things updated.

And also, I've discovered that some exchanges provide some of this metadata in the api, and those cases the static data can be overwritten by the dynamic data.
I like the current method (documented in the wiki) of merging files with data pulled from online APIs. It would be good for data to be removed from the .json if an API source is implemented.

In all cases I've seen the dynamic data is used to supplement the static data. Where the dynamic data can be accessed it's considered to me more accurate than the static data and would override it, yes.

fees
but I'd probably be in favour of a breaking change in order to have accurate data.

Maker, taker and tiered fees would be a great addition to the metadata as it would help inform bots/algos/people before placing an order. Additionally, I think in many cases the fees charged are given with the user trades, which can be used for book keeping. A breaking change is fine.

rate limits

Those are there already.

we need per currency-pair fees as some exchanges have different fees in different markets.

Those are there already. mostly I suppose.

And... the fees are sometimes issued in the base currency, and sometimes the counter currency. Sometimes the user can choose which (and it's a port of your account settings).

yes, complicated. But could be done.

amount-scale - doesn't exist in this project yet.

Definitely could be added.

Member

timmolter commented Jul 2, 2017

@npomfret @molexx Thanks for your thoughts on the matter.

It would have been helpful (for a beginner like me) if this was documented in the readme.md file.

https://github.com/timmolter/XChange/wiki/Design-Notes

See section on metadata. I should probably add an easy-to-see link in the README.

I wonder if the static data, which seems to change quite frequently might be best kept in a separate repository?

Well, the develop branch is more or less that. Plus the code is also changing so much chasing changes on exchanges' API. Every time I do a release there are hotfixes and additions literally the next day. :) It's tricky because there is no real way to run integration tests on all the code because we're dealing with live exchanges and API keys to private accounts.

I've had to create my own copies of almost every one I've used because they're out of date.

Please contribute your changes, as that's the only way we can hope to keep things updated.

And also, I've discovered that some exchanges provide some of this metadata in the api, and those cases the static data can be overwritten by the dynamic data.
I like the current method (documented in the wiki) of merging files with data pulled from online APIs. It would be good for data to be removed from the .json if an API source is implemented.

In all cases I've seen the dynamic data is used to supplement the static data. Where the dynamic data can be accessed it's considered to me more accurate than the static data and would override it, yes.

fees
but I'd probably be in favour of a breaking change in order to have accurate data.

Maker, taker and tiered fees would be a great addition to the metadata as it would help inform bots/algos/people before placing an order. Additionally, I think in many cases the fees charged are given with the user trades, which can be used for book keeping. A breaking change is fine.

rate limits

Those are there already.

we need per currency-pair fees as some exchanges have different fees in different markets.

Those are there already. mostly I suppose.

And... the fees are sometimes issued in the base currency, and sometimes the counter currency. Sometimes the user can choose which (and it's a port of your account settings).

yes, complicated. But could be done.

amount-scale - doesn't exist in this project yet.

Definitely could be added.

@timmolter

This comment has been minimized.

Show comment
Hide comment
@timmolter

timmolter Aug 13, 2017

Member

Related Issues

Member

timmolter commented Aug 13, 2017

Related Issues

@timmolter timmolter changed the title from more metadata needed? to [50 mBTC Bounty] General metadata improvements Aug 13, 2017

@timmolter timmolter added the Bounty label Aug 13, 2017

@timmolter timmolter added this to the XChange 4 milestone Aug 13, 2017

@timmolter timmolter changed the title from [50 mBTC Bounty] General metadata improvements to [20 mBTC Bounty] General metadata improvements Oct 17, 2017

@timmolter timmolter changed the title from [20 mBTC Bounty] General metadata improvements to [10 mBTC Bounty] General metadata improvements Nov 17, 2017

@jnorthrup

This comment has been minimized.

Show comment
Hide comment
@jnorthrup

jnorthrup Apr 17, 2018

Contributor

I consider #2507 Currency as Interface to fall into the category of "General Metadata Improvements"

Contributor

jnorthrup commented Apr 17, 2018

I consider #2507 Currency as Interface to fall into the category of "General Metadata Improvements"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment