[WIP] work on returnTicker #18

JonathonDunford · 2018-02-09T17:00:07Z

We will need another database table for ticker information.

Proposed information to store:

ticker = {
        'tokenAddr': ,
        'quoteVolume': ,
        'baseVolume': ,
        'last': ,
        'bid': ,
        'ask': ,   
    }

bid and ask can come from orders
last can come from trades

Would volume have to be determined by adding all of the orders in our orderbook?

freeatnet · 2018-02-09T17:08:40Z

How and when do we update bid and ask? Off the top of my head, they should at least change when new orders are created (and the price is higher / lower than previous one) and when there is no more volume available at the price reflected. How do we fetch that?

JonathonDunford · 2018-02-09T17:10:44Z

We haven't finished the code for #8 but that could be where we change the bid/ask based on volume changes.

OR

We could have something that scans our own database on a regular frequency, compares it to ticker data, then changes ticker data where it sees fit.

We can invalidate orders separately and the change would be reflected there.

…eplacement into returnTicker

freeatnet · 2018-02-10T20:37:56Z

app/services/websocket_server.py

@@ -231,6 +231,36 @@ def format_order(record):

    await sio.emit('market', response, room=sid)

+#return ticker information, if no token specified show all
+@sio.on('returnTicker')


There isn't a separate returnTicker event: instead, returnTicker is a key on a market event, which is emitted to a specific user in response to a getMarket event by that user.

…in the database now.

freeatnet · 2018-02-10T21:40:07Z

app/services/websocket_server.py

    if token:
+
+        #get ticker for passed in token
+        tickers = await get_tickers(token)


We should return all tickers regardless.

If you pass in a specific token, it shouldn't transmit unnecessary data?

This data fills out the values in the bottom-left ticker though, so it needs to be for all tokens.

Referenced here: #21

Fixed now.

freeatnet · 2018-02-10T21:41:29Z

app/services/websocket_server.py

+#get returnTicker, grabs data from tickers table
+async def get_tickers(token_hexstr):
+
+    #init


Nitpick: according to PEP8, the standard Python code style guide, "They should start with a # and a single space."

freeatnet · 2018-02-10T21:41:47Z

app/services/websocket_server.py

+async def get_tickers(token_hexstr):
+
+    #init
+    where = ''


placeholder_args should also be defined here.

freeatnet · 2018-02-10T21:45:48Z

app/services/websocket_server.py

+        "quoteVolume": str(contract.denormalize_value(ticker["quote_volume"])),
+        "baseVolume": str(contract.denormalize_value(ticker["base_volume"])),
+        "last": str(contract.denormalize_value(ticker["last"])),
+        "percentChange": str(contract.denormalize_value(ticker["percent_change"])),


Not a 100% sure what the values are going to look like in raw format, we should double-check these conversions once we have them.

freeatnet · 2018-02-10T21:46:01Z

app/services/websocket_server.py

+        "percentChange": str(contract.denormalize_value(ticker["percent_change"])),
+        "bid": str(contract.denormalize_value(ticker["bid"])),
+        "ask": str(contract.denormalize_value(ticker["ask"])),
+        "modified": datetime.now().isoformat()


What is the modified value supposed to reflect?

When the last time that ticker was updated

Sounds like it should come from the DB then. Is this a placeholder?

You're right. Whoops.

freeatnet · 2018-02-10T22:31:06Z

app/services/websocket_server.py

+        "bid": str(contract.denormalize_value(ticker["bid"])),
+        "ask": str(contract.denormalize_value(ticker["ask"])),
+        "modified": ticker["modified"]
+        # "modified": datetime.now().isoformat()


Dead lines should be removed.

Placeholder for when I write the code that updates the data. Will remove at that time.

freeatnet · 2018-02-10T22:31:37Z

app/services/websocket_server.py

+
+        # get ticker for passed in token
+        #tickers = await get_tickers(token)
+        tickers = await get_tickers()


We could probably extract that to above if we no longer differentiate by token.

freeatnet · 2018-02-10T22:37:30Z

app/services/websocket_server.py

+            """.format(where),
+            *placeholder_args)
+
+# format ticker information to camelcase from underscore, sanitize, add modified date


Oh, also, suggestion! Python docstrings might be a better way to do this

freeatnet · 2018-02-10T22:39:18Z

app/services/websocket_server.py

    if token:
+
+        # get ticker for passed in token
+        #tickers = await get_tickers(token)


Also dead code

freeatnet · 2018-02-10T22:59:27Z

app/services/websocket_server.py

    if token:
+


Unnecessary new lines

JonathonDunford · 2018-02-10T23:44:38Z

Now, we need something that actually populates this data.

I think the best method may be using an observer to scan our orders table and comparing actual data to the old data stored in tickers.

Maybe there is a better way to only update it on certain events, but this would be a catch all and we could cancel/update orders separately without fear of breaking this system.

JonathonDunford · 2018-02-11T16:04:27Z

Convo copied here for posterity.

BigBrother - Today at 10:56 AM
Well, instead of relying on ED data to update our tickers, we would scan our own orders + trades table to get bid/ask/volume from orders and last/percent from trades(edited)

freeatnet - Today at 10:57 AM
With you on that!

BigBrother - Today at 10:58 AM
And since the rest of the backend handles updating and making sure that data is accurate
regardless of ED connectability
That should be the most reliable source of data?
That's my thinking at least.

freeatnet - Today at 11:02 AM
Absolutely agreed. To clarify: In my mind, there are two ways of doing this:

Relying on data we store, we periodically update ticker data from scratch by looking up the right orders and trades.

Relying on data we process, we track certain events and update ticker incrementally, based on the context of the event

First way is much, much simpler. We may have revisit it for purposes of real-time updating in the future, but it should work for now.

Clean websocket_server a little.

freeatnet

Some minor things pointed out.

I'm also not sure about iterating over all orders to build a ticker. Perhaps, the ticker updater should know a list of token addresses which it should update; then, for each token, fetch fields with specific SQL queries.

For example, a ticker bid value can be obtained by running something like

SELECT MAX(amount_give / amount_get)
FROM orders
WHERE token_get = :zero_addr
    AND token_give = :token_addr
    AND (state = 'OPEN'::orderstate AND expires > :current_block AND available_volume > 0)

This approach is a little harder on the DB, but it is faster than iterating over all orders and is somewhat simpler to implement than partial updates.

freeatnet · 2018-02-12T15:29:13Z

app/services/ticker_observer.py

+    return
+
+# Main function. Grabs orders and trades, then scans through orders and trades to update `tickers`
+async def update_tickers():


Main function should be called main() and be called if the module is the main module of the python invocation, like so.

freeatnet · 2018-02-12T15:31:23Z

app/services/ticker_observer.py

+# Main function. Grabs orders and trades, then scans through orders and trades to update `tickers`
+async def update_tickers():
+
+    orders = await get_orders();


No semicolons needed if you have just one expression / statement on a line.

freeatnet · 2018-02-12T15:33:19Z

app/services/ticker_observer.py

+        return await conn.fetch(
+            """
+            SELECT token_give, amount_give, token_get, amount_get, expires
+            FROM orders


Pretty good SELECT statement for pulling all orders!

freeatnet · 2018-02-12T15:36:28Z

app/services/ticker_observer.py

+app = web.Application()
+sio.attach(app)
+
+ZERO_ADDR = "0x0000000000000000000000000000000000000000"


We should extract these into reusable constants. app/constants.py, perhaps?

freeatnet · 2018-02-12T16:24:39Z

app/services/ticker_observer.py

+
+            else:
+                # If there isn't a current recorded bid OR current recorded bid is lower than the bid in this order.
+                if not has_attr(ticker_updates[order["token_get"]], 'bid') || ticker_updates[order["token_get"]]["bid"] < this_orders_bid:


The logical or operator is or.

JonathonDunford · 2018-02-17T21:22:33Z

Moved to #40

start work on return Ticker call from websocket_server

e427fe9

JonathonDunford force-pushed the returnTicker branch from 5b46334 to e427fe9 Compare February 10, 2018 19:28

freeatnet and others added 4 commits February 10, 2018 22:52

Hotfix: Remove leaked imports of huey

95fdf45

start work on return Ticker call from websocket_server

c37ffc6

Merge branch 'returnTicker' of https://github.com/forkdelta/backend-r…

3bb08a0

…eplacement into returnTicker

Create Tickers table in alembic. Small updates to returnTicker

4f6687f

freeatnet reviewed Feb 10, 2018

View reviewed changes

Finish backend work on returnTicker, need to actually store the data …

8de146b

…in the database now.

JonathonDunford requested a review from freeatnet February 10, 2018 21:36

freeatnet reviewed Feb 10, 2018

View reviewed changes

Jonathon Dunford added 2 commits February 10, 2018 17:06

fix comments, fix modified pull

13c3047

return all tickers all the time in returnTicker

ddb03cf

freeatnet reviewed Feb 10, 2018

View reviewed changes

app/services/websocket_server.py Outdated

if token:

Copy link

Contributor

freeatnet Feb 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary new lines

Remove unnecessary code.

aca2fb7

Jonathon Dunford added 2 commits February 11, 2018 17:07

Start work on the "ticker observer"

c9d8cdb

Clean websocket_server a little.

Fix typo

72813dc

freeatnet reviewed Feb 12, 2018

View reviewed changes

This was referenced Feb 15, 2018

Initial production rollout checklist #24

Open

[#3] Add returnTicker API #40

Merged

JonathonDunford closed this Feb 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] work on returnTicker #18

[WIP] work on returnTicker #18

JonathonDunford commented Feb 9, 2018 •

edited

Loading

freeatnet commented Feb 9, 2018

JonathonDunford commented Feb 9, 2018

freeatnet Feb 10, 2018 •

edited

Loading

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018 •

edited

Loading

freeatnet Feb 10, 2018 •

edited

Loading

JonathonDunford Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford Feb 10, 2018

freeatnet Feb 10, 2018

JonathonDunford commented Feb 10, 2018

JonathonDunford commented Feb 11, 2018 •

edited by freeatnet

Loading

freeatnet left a comment •

edited

Loading

freeatnet Feb 12, 2018

freeatnet Feb 12, 2018

freeatnet Feb 12, 2018

freeatnet Feb 12, 2018

freeatnet Feb 12, 2018

JonathonDunford commented Feb 17, 2018 •

edited

Loading

[WIP] work on returnTicker #18

[WIP] work on returnTicker #18

Conversation

JonathonDunford commented Feb 9, 2018 • edited Loading

freeatnet commented Feb 9, 2018

JonathonDunford commented Feb 9, 2018

freeatnet Feb 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonathonDunford Feb 10, 2018 • edited Loading

Choose a reason for hiding this comment

freeatnet Feb 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonathonDunford commented Feb 10, 2018

JonathonDunford commented Feb 11, 2018 • edited by freeatnet Loading

freeatnet left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonathonDunford commented Feb 17, 2018 • edited Loading

JonathonDunford commented Feb 9, 2018 •

edited

Loading

freeatnet Feb 10, 2018 •

edited

Loading

JonathonDunford Feb 10, 2018 •

edited

Loading

freeatnet Feb 10, 2018 •

edited

Loading

JonathonDunford commented Feb 11, 2018 •

edited by freeatnet

Loading

freeatnet left a comment •

edited

Loading

JonathonDunford commented Feb 17, 2018 •

edited

Loading