Missing orders during OrderSync

[pgebheim]

Context

Please provide any relevant information about your setup

• Mesh in browser (10.0.0-unstable3)
• Mac an Linux and Windows
• Firefox 78.0.2
• Brave version 1.10.60 Chromium: 81.0.4044.138 (Official Build) unknown (64-bit)

We are testing this by using a standalone node which is the recipient of orders which then relays them through the p2p network. During order sync we see behavior where orders are partially synced, and then refreshing the page will progressively sync more orders until we have all of the orders that we knowingly posted.

One issue is I don't have good logs for this because it is running in a webworker and at the moment I'm not actually getting those logs spit out to the FF or Chrome consoles even through they should be.

Here's some screenshots detailing what has happened after there are N-thousand orders posted on an augur market.

---

Step 1

Window on the left: I wiped the databases entirely so we're starting from a fresh state. After OrderSync is done it looks like I received a total of 16 orders (this is the only market on mainnet, all orders are of size 10 -- 160 shares = 16 orders).

The window on the right was a browser open while orders were created. Those orders are still valid.

So it looks like I got 16 orders during OrderSync...

Step 2

Now, refresh the left window and orders are streaming in from OrderSync -- So the explanation here could be that we just happened to randomly select 5 peers that didn't have orders on the previous load.

Now waiting I see a bunch of orders come in, and then it stops and I receive no more with the following state which is missing > 300 orders.

Step 3

Refresh again, I get more orders and now I have all the ones that the other browser knows about.

Oddly, for 0.56 I actually find the 270 orders that I never received in the initial browser window (on the right). Also notable, I never get those orders on that side, I've refreshed a bunch etc.

So something must be wrong during ordersync because whatever nodes its talking to has these orders -- but the local syncing just stops getting them...

Step 4

Clear the state from the window on the RIGHT which previously had all the orders. On the first load it loads 16 orders, just like during step 1!

Step 5

Refresh the window on the right, OrdersStream in in big blocks (200-300) during the ordersync period. And now both tabs have all known published orders on this market.

[nuliknol]

@pgebheim you are not alone, I have the same problem, with version 9.4.1. This means, PR 874 doesn't fix this.

now, I checked how long ago this problematic order was inserted, and it was 1 hour before it was shown as missing. So, it stayed for 1 hour in the Node, then suddenly it was reported as missing, then 1 minute later, on the next sync, it was reported as existent again.

niko@desk :~/ae_logs$ grep 0x66e4c671bbde47e04f037fec0143fa8505d4f954643086f36faa307c6f8cde1c mesh_info.log 
INFO: 2020/07/30 18:52:48 Inserted open order 0x66e4c671bbde47e04f037fec0143fa8505d4f954643086f36faa307c6f8cde1c
INFO: 2020/07/30 19:50:07 Order 0x66e4c671bbde47e04f037fec0143fa8505d4f954643086f36faa307c6f8cde1c doesn't exist in Mesh Node, but does exist in the DB. Deleting. (DB_DIRTY_OORDERS)
INFO: 2020/07/30 19:51:11 Inserted open order 0x66e4c671bbde47e04f037fec0143fa8505d4f954643086f36faa307c6f8cde1c
niko@desk :~/ae_logs$ 

I am using 9.4.1 from master (commit c163356 to be specific) as server, and client sources are from 9.4.0

[nuliknol]

@jalextowle what about providing another RPC function like getOrderByHash(hash) , this function would access to meshdb.MeshDB object and check if the order exist there, and then return sucess/failure status?
This seems like a missing function from the point of view of design, so it could be a good moment to add it, what do you think?

In my case, this function would help to double check if order exists, before I delete it locally, so this way I would prevent deletion even if rpcclient.GetOrders() misses few orders. But if the order gets deleted in the database (MeshDB) also, then this would not work. But I can't figure why would an order get deleted from MeshDB if it was created 1 hour ago.... seems weird.

[pgebheim]

@nuliknol -- btw Augur runs with 10.0.0-unstable-5 browser-mesh at the moment. It has a bit of a custom WASM build that is being readied for a mainline 10.0.0 0x release.

The main bugs here were fixed in that release.