Add book depth chart

[Reckless-Satoshi]

Create a book depth chart as a react component that fetches data from /api/book/?currency=0&type=2.`
It could use plotly/chart.js/canvas.js .

A reference of what a usual book depth plot looks like:

It could be subsettable for currencies (only show liquidity for a single currency) or aggregate all currency offers (e.g. convert every fiat to USD)or a better overview of the whole market. On the the X axis it could show price denominated in the fiat currency, or premium (%) over external market prices.

This task is currently rewarded with ⚡ 300,000 Sats . Reply to this comment to be assigned.

[PeterMcBTC]

apparently the code is described in the following article https://towardsdatascience.com/learn-what-a-depth-chart-is-and-how-to-create-it-in-python-323d065e6f86
I will have a look at it

[Reckless-Satoshi]

@PeterMcBTC Awesome! The tutorial seems to be python based. It would certainly be more useful if the plot can be generated in the frontend in javascript with data fetched from http://unsafe.robosats.com/api/book/?currency=0&type=2.

[KoalaSat]

Is there any preference on the charts library?

I have good experiences with https://nivo.rocks, it adds a really good abstraction layer to d3 yet it still allows you to go as custom as you want.

Specifically I think a Line will do a good fit with this case:

• https://nivo.rocks/line/
• https://nivo.rocks/storybook/?path=/story/line--stacked-lines
• https://nivo.rocks/storybook/?path=/story/line--real-time-chart

[KoalaSat]

Also, I believe this is going to kill the back-end if we don't implement WebSockets, I did a quick research and looks like it's possible to run it over TorNet https://gist.github.com/TooTallNate/6253581

[Reckless-Satoshi]

Is there any preference on the charts library?

Something pretty and easy :) This is a very creative task, imposing a library wouldn't make sense.

Also, I believe this is going to kill the back-end if we don't implement WebSockets

Are you sure? I was not thinking of a realtime graph. I had in mind maybe use the very same orders fetched when you open the order book to add a pop up with the Depth Chart on the same page.... I did not think much into it to be honest, so we are open to anything in this regard, I am not even sure where exactly this book depth chart might fit best :)

However, if WebSockets are really needed, no problem. WebSockets are currently in use in some parts of the app that require real time (e.g. the chat). WebSockets work both over TOR and I2P. https://github.com/Reckless-Satoshi/robosats/blob/f2f6309ed19f8e35310e6c2b485af17a99a5a5a7/frontend/src/components/EncryptedChat.js#L42

I will assign you the task since you are already researching it. As always, no pressure, feel free to drop it any time if it's not what you thought.

[KoalaSat]

I was not thinking of a realtime graph.

Ah! On that case for sure websockets is not needed, I just assumed the idea was to show real time data. Then we'll just need a simple request. Maybe we can work now just on the chart and include real time data in the future.

I'm curious now, I assumed there was no websockets implemented because while working on WebLN I saw the the front-end constantly checking the order info endpoint, is there any good reason to do that or was just implemented before websockets?

Let me see then if I can play with Nivo a little bit and see if it fits our needs.

[KoalaSat]

I had a quick look into the API, can you explain what type is? It looks like on the book page only 2 is being used.

Also, if we want to create a global depth book we'll need a exchange rate, I see on the backend you are using blockchain.info and an alternative, does it makes sense to make this request on frontend for this purpose?

I have other thought about exchange rates I would like to share and as far I understand is not done on back-end. When dealing with only fiat, you use one of the currencies as base and apply the exchange rate over any other currencies. So on the average bank application you would see, for USD:

• USD -> USD
• EUR -> USD
• GBP -> USD

But in our case, our base will be USD but our exchange rate BTC, which I believe should be then like:

• USD -> BTC -> USD
• EUR -> BTC -> USD
• GBP -> BTC -> USD

Am I right? Is there something I missed?

[Reckless-Satoshi]

Maybe we can work now just on the chart and include real time data in the future.

Sounds great!

I'm curious now, I assumed there was no websockets implemented because while working on WebLN I saw the the front-end constantly checking the order info endpoint, is there any good reason to do that or was just implemented before websockets?

There is no good reason, but there are some reasons :D A synchronous REST API base trade pipeline was easier to implement as a demonstration for a newbie like me. I had too much on my plate to jump straight into mind-breaking async python. Then the REST endpoints simply proved to work well and reliably (unlike the chat that is websocket based and quite unstable), so we gave little priority to the task #34 . This task has been there since first week of development and it is a must to scale up.

Also, if we want to create a global depth book we'll need a exchange rate, I see on the backend you are using blockchain.info and an alternative, does it makes sense to make this request on frontend for this purpose?

The backend fetches prices since clients can't be trusted to price correctly. However, for simple data plotting it is not a bad idea. Still, I am a bit skeptical since ideally we want 100% of the client requests to only go towards RoboSats backend: fetching prices from third parties might hinder privacy.

But in our case, our base will be USD but our exchange rate BTC, which I believe should be then like:

USD -> BTC -> USD
EUR -> BTC -> USD
GBP -> BTC -> USD

I am not sure I follow. What is the use case here? Is this a strategy to synthesize every active order (many fiats) into a single chart (everything priced in USD) ?

[Reckless-Satoshi]

I had a quick look into the API, can you explain what type is? It looks like on the book page only 2 is being used.

https://github.com/Reckless-Satoshi/robosats/blob/f2f6309ed19f8e35310e6c2b485af17a99a5a5a7/api/models.py#L267-L269

Type 0 to fetch only "BUY" orders, 1 to fetch only "SELL" orders, 2 fetches everything.

[KoalaSat]

I am a bit skeptical since ideally we want 100% of the client requests to only go towards RoboSats backend: fetching prices from third parties might hinder privacy.

What about exposing this function...
https://github.com/Reckless-Satoshi/robosats/blob/5281176e3c233871da9a3f6ea6552489ff12aa76/api/utils.py#L68
...on an endpoint and use Robosats as proxy? I see the response is already cached for 3 seconds, so even better.

I am not sure I follow. What is the use case here? Is this a strategy to synthesize every active order (many fiats) into a single chart (everything priced in USD) ?

You nailed it, sorry if it wasn't clear. I assumed the amount of every order is on his original currency, if this is true, we need a way, as you said, to have the same base price and actually display on the chart the real amount value for a given currency.

Type 0 to fetch only "BUY" orders, 1 to fetch only "SELL" orders, 2 fetches everything.

This is good, I would do 2 request (1, 2) in parallel so we can be faster, I was even thinking on reusing the ones for the book table.

[Reckless-Satoshi]

What about exposing this function...

https://github.com/Reckless-Satoshi/robosats/blob/5281176e3c233871da9a3f6ea6552489ff12aa76/api/utils.py#L68

...on an endpoint and use Robosats as proxy? I see the response is already cached for 3 seconds, so even better.

This would work just great. However, we could just serve a new endpoint similar to /api/limits/ but only with the price key (or use /api/limits/ as is...?). This one sends the RoboSats backend currency exchange rates from the database (currently fetched from providers every 60 seconds). That way the client does not need to wait a super long round trip to get the prices.

You nailed it, sorry if it wasn't clear. I assumed the amount of every order is on his original currency, if this is true, we need a way, as you said, to have the same base price and actually display on the chart the real amount value for a given currency.

Exactly, that would be the best way to do it.

This is good, I would do 2 request (1, 2) in parallel so we can be faster, I was even thinking on reusing the ones for the book table.

Requesting with type=2 gets you everything. The Order Book on the frontend is requested with type=2 then it is filtered by the BookPage. Initially it used to make a different request every time the filter changed, but it felt sluggish. These are stored always in https://github.com/Reckless-Satoshi/robosats/blob/f2f6309ed19f8e35310e6c2b485af17a99a5a5a7/frontend/src/components/HomePage.js#L23
So we could simply use that array instead of making a new request only for the chart.. Every time the BookPage is mounted or the refresh book button is clicked, that array is updated.

[KoalaSat]

This would work just great. However, we could just serve a new endpoint similar to /api/limits/ but only with the price key (or use /api/limits/ as is...?).

This is exactly what I need, thanks!

[KoalaSat]

I have been playing with the data, I can't believe I had so much fun with this 😄

I wanted first to work on the data formatting so the screenshot bellow just uses nivo's default plot line chart. Anyways, after a mind blowing back and forward, this is the data I managed to generate:

I lost almost 1 hour triple checking the data because it's totally different to what I'm use to see on a depth chart, but I'm really confident that after applying properly the exchange rates, this is the real Depth Chart for Robosats. I think the overlapping is caused by the price ranges (I'm using the max), so it looks like there is a lot of buyers looking for small quantities, and a lot of sellers only offering big amounts, but they don't really match between each other. We can't forget also even if the desired price for a user is available, different payment methods can make the offer not viable.

These are the functions I use for the axis:

X Axis

const calculateBtc = (order: any): number => {
    const amount = parseInt(order.amount || order.max_amount)
    return amount / order.price
  }

Y Axis

 // We need to transform all currencies to the same base (ex. USD), we don't have the exchange rate
  // for EUR -> USD, but we know the rate of both to BTC, so we get advantage of it and apply a
  // simple rule of three
  const calculateFiatExchangeRate = (order: any): number => {
    return (order.price * limits[currency].price) / limits[order.currency].price
  }

I hope all this formulas and data makes sense for you, during the upcoming days I'll try to find some time to work on the chart view.

[Reckless-Satoshi]

Woah, that's looking fantastic!

The chart shape was to expect, there is overlap between between bid/ask prices :)

Between a point in the chart and the next one, there is a slope, yet usually Book depth charts have "steps". I painted in dark red what I mean:

I guess this is an easy fix by introducing some "virtual" points.

Have you thought about simply using the premium in the X axis instead of the price in dollars? It does not require of currency conversion. A toggle switch so the users can change the X axis between premium / price would be top!

[KoalaSat]

I guess this is an easy fix by introducing some "virtual" points.

Yes, that's my plan, but first I wanted to make sure I generate the right data

Doesn't order.price already includes the premium? 🤔

[Reckless-Satoshi]

Doesn't order.price already includes the premium? thinking

Yes. Both charts might look identical (maybe not). But users might find the X axis expressed as % very informative (I would like to see it :D)