[Bug]: Unable to retrieve stop orders from datanode

[MuthuVega]

Problem encountered

Unable to retrieve the list of stop orders from the datanode.

Observed behaviour

Tried gRPC end point:

request = api.trd_v2.ListStopOrdersRequest()
response = api.grpc_stub_trd_data_v2.ListStopOrders(request)

Tried graphQL:

query AllStopOrders{
  stopOrders  {
    edges {
      node {
        id
      }
    }
  }
}

Response:

{
  "data": {
    "stopOrders": {
      "edges": null
    }
  }
}

Tried REST:

http://localhost:3028/api/v2/stoporders

Response:

{"orders":{"edges":[],"pageInfo":{"hasNextPage":false,"hasPreviousPage":false,"startCursor":"","endCursor":""}}}

Expected behaviour

All 3 methods should return the list of stop orders

Steps to reproduce

Run the above graphql query or try the REST endpoint.

Else , run the below system test.

System Test Name - test_stop_order_submit_datanode_bug
System Test Branch - feat_stop_order_bugs

Software version

vega - develop

Failing test

test_stop_order_submit_datanode_bug

Jenkins run

https://jenkins.ops.vega.xyz/job/common/job/system-tests-wrapper/80056/

Configuration used

No response

Relevant log output

2023-07-03 18:48:51.745:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 117]  Stop Order events - ['{"id":"197-5","block":"F922705AAB4F55CF7E6362E1A48300954A7FA72972938503CD96D772558DBF18","type":"BUS_EVENT_TYPE_STOP_ORDER","stopOrder":{"submission":{"marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","price":"130000000","size":"1000000","side":"SIDE_SELL","timeInForce":"TIME_IN_FORCE_FOK","type":"TYPE_LIMIT","reference":"test_stop_order_submit_datanode_bug[order_param0]-b1423e38","reduceOnly":true},"stopOrder":{"id":"68ccbc96264e87f4ef1ecde812e24d6e9198f4b2b5c1e0b33612b80a31d66bf6","ocoLinkId":"eb42ad5cb994dd2f61e74fdeaddbb80eb4db4404c3ae4d20629d4473c0935a03","triggerDirection":"TRIGGER_DIRECTION_FALLS_BELOW","status":"STATUS_PENDING","createdAt":"1688410121","updatedAt":"1688410121","partyId":"415fa062d2781f61abec23c04586d97e9dbf876ddb1a8bcbf31cf8c9315fb975","marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","trailingPercentOffset":"0.99"}},"version":1,"chainId":"testnet-001","txHash":"CA05497316D46DA0CA6016ADF09C7041339A7A335478A6040F81BE26A043279F"}\n', '{"id":"197-6","block":"F922705AAB4F55CF7E6362E1A48300954A7FA72972938503CD96D772558DBF18","type":"BUS_EVENT_TYPE_STOP_ORDER","stopOrder":{"submission":{"marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","price":"130000000","size":"1000000","side":"SIDE_SELL","timeInForce":"TIME_IN_FORCE_FOK","type":"TYPE_LIMIT","reference":"test_stop_order_submit_datanode_bug[order_param0]-b1423e38","reduceOnly":true},"stopOrder":{"id":"eb42ad5cb994dd2f61e74fdeaddbb80eb4db4404c3ae4d20629d4473c0935a03","ocoLinkId":"68ccbc96264e87f4ef1ecde812e24d6e9198f4b2b5c1e0b33612b80a31d66bf6","triggerDirection":"TRIGGER_DIRECTION_RISES_ABOVE","status":"STATUS_PENDING","createdAt":"1688410121","updatedAt":"1688410121","partyId":"415fa062d2781f61abec23c04586d97e9dbf876ddb1a8bcbf31cf8c9315fb975","marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","trailingPercentOffset":"0.99"}},"version":1,"chainId":"testnet-001","txHash":"CA05497316D46DA0CA6016ADF09C7041339A7A335478A6040F81BE26A043279F"}\n', '{"id":"197-7","block":"F922705AAB4F55CF7E6362E1A48300954A7FA72972938503CD96D772558DBF18","type":"BUS_EVENT_TYPE_STOP_ORDER","stopOrder":{"submission":{"marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","price":"130000000","size":"1000000","side":"SIDE_SELL","timeInForce":"TIME_IN_FORCE_FOK","type":"TYPE_LIMIT","reference":"test_stop_order_submit_datanode_bug[order_param0]-b1423e38","reduceOnly":true},"stopOrder":{"id":"68ccbc96264e87f4ef1ecde812e24d6e9198f4b2b5c1e0b33612b80a31d66bf6","ocoLinkId":"eb42ad5cb994dd2f61e74fdeaddbb80eb4db4404c3ae4d20629d4473c0935a03","triggerDirection":"TRIGGER_DIRECTION_FALLS_BELOW","status":"STATUS_PENDING","createdAt":"1688410121","updatedAt":"1688410121","partyId":"415fa062d2781f61abec23c04586d97e9dbf876ddb1a8bcbf31cf8c9315fb975","marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","trailingPercentOffset":"0.99"}},"version":1,"chainId":"testnet-001","txHash":"CA05497316D46DA0CA6016ADF09C7041339A7A335478A6040F81BE26A043279F"}\n', '{"id":"197-8","block":"F922705AAB4F55CF7E6362E1A48300954A7FA72972938503CD96D772558DBF18","type":"BUS_EVENT_TYPE_STOP_ORDER","stopOrder":{"submission":{"marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","price":"130000000","size":"1000000","side":"SIDE_SELL","timeInForce":"TIME_IN_FORCE_FOK","type":"TYPE_LIMIT","reference":"test_stop_order_submit_datanode_bug[order_param0]-b1423e38","reduceOnly":true},"stopOrder":{"id":"eb42ad5cb994dd2f61e74fdeaddbb80eb4db4404c3ae4d20629d4473c0935a03","ocoLinkId":"68ccbc96264e87f4ef1ecde812e24d6e9198f4b2b5c1e0b33612b80a31d66bf6","triggerDirection":"TRIGGER_DIRECTION_RISES_ABOVE","status":"STATUS_PENDING","createdAt":"1688410121","updatedAt":"1688410121","partyId":"415fa062d2781f61abec23c04586d97e9dbf876ddb1a8bcbf31cf8c9315fb975","marketId":"58edb8a800b96e76bb7dcb93be06a3a3e075df78bb08d76758e91def69ee35fd","trailingPercentOffset":"0.99"}},"version":1,"chainId":"testnet-001","txHash":"CA05497316D46DA0CA6016ADF09C7041339A7A335478A6040F81BE26A043279F"}\n']
2023-07-03 18:49:21.746:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 121] Stop orders retrieved via gRPC is {}
2023-07-03 18:49:21.748:[INFO] [MainProcess] [MainThread] [    aiohttp.py]:[     execute]:[ 278] >>> {"query": "query AllStopOrders {\n  stopOrders {\n    edges {\n      node {\n        id\n      }\n    }\n  }\n}"}
2023-07-03 18:49:21.752:[INFO] [MainProcess] [MainThread] [    aiohttp.py]:[     execute]:[ 320] <<< {"data":{"stopOrders":{"edges":null}}}
2023-07-03 18:49:21.753:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 126] Stop orders retrieved via graphql is {'stopOrders': {'edges': None}}
2023-07-03 18:49:21.753:[INFO] [MainProcess] [MainThread] [     apiMod.py]:[    get_rest]:[ 129] REST API request url='http://127.0.0.1:3028/api/v2/stoporders' params={} payload={}
2023-07-03 18:49:21.757:[INFO] [MainProcess] [MainThread] [     apiMod.py]:[    get_rest]:[ 131] REST API response status 200
2023-07-03 18:49:21.758:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 130] Stop orders retrieved via REST is <Response [200]>

[pscott31]

hmm, I tried to replicate this by manually inserting a stop order in the database and looking at the grpc web ui and it worked. Are you using any filters or anything? Perhaps they are not getting added to the database for some reason..

If it's still available would you mind taking a peek in your DB and seeing if there are any rows in the stop orders table? select * from stop_orders

[MuthuVega]

@pscott31 - Don't think the stop orders are making it to the database table.

Here is the updated Jenkins Job with the test:

https://jenkins.ops.vega.xyz/blue/organizations/jenkins/common%2Fsystem-tests-wrapper/detail/system-tests-wrapper/80402/pipeline/

Relevant Logs:

2023-07-04 17:05:55.778:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 135] Assets retrieved from db table assets are 

2023-07-04 17:05:55.779:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 136] [(<memory at 0x7f65f4f3cdc0>, 'Vega', 'VEGA', 18, Decimal('0'), '', '0x67175Da1D5e966e40D11c4B2519392B2058373de', Decimal('0'), Decimal('0'), 'STATUS_ENABLED', <memory at 0x7f65f4f3c580>, datetime.datetime(2023, 7, 4, 17, 2, 35, 403500, tzinfo=datetime.timezone.utc)), (<memory at 0x7f65f4f3ca00>, 'VOTE', 'VOTE', 5, Decimal('1'), '10000', '', Decimal('0'), Decimal('0'), 'STATUS_ENABLED', <memory at 0x7f65f4f3c4c0>, datetime.datetime(2023, 7, 4, 17, 2, 35, 403500, tzinfo=datetime.timezone.utc)), (<memory at 0x7f65f584af40>, 'AssetID', 'TAS', 18, Decimal('1000000000000000000'), '100000000000000000000000000', '', Decimal('0'), Decimal('0'), 'STATUS_PROPOSED', <memory at 0x7f65f584a100>, datetime.datetime(2023, 7, 4, 17, 4, 29, 444189, tzinfo=datetime.timezone.utc)), (<memory at 0x7f65f584ad00>, 'AssetID', 'TAS', 18, Decimal('1000000000000000000'), '100000000000000000000000000', '', Decimal('0'), Decimal('0'), 'STATUS_ENABLED', <memory at 0x7f65f584aa00>, datetime.datetime(2023, 7, 4, 17, 4, 35, 35055, tzinfo=datetime.timezone.utc)), (<memory at 0x7f65f584a1c0>, 'AssetID', 'TAS', 18, Decimal('1000000000000000000'), '100000000000000000000000000', '', Decimal('0'), Decimal('0'), 'STATUS_PROPOSED', <memory at 0x7f65f5c67580>, datetime.datetime(2023, 7, 4, 17, 4, 41, 114164, tzinfo=datetime.timezone.utc)), (<memory at 0x7f65f5c67940>, 'AssetID', 'TAS', 18, Decimal('1000000000000000000'), '100000000000000000000000000', '', Decimal('0'), Decimal('0'), 'STATUS_ENABLED', <memory at 0x7f65f5c67280>, datetime.datetime(2023, 7, 4, 17, 4, 47, 223554, tzinfo=datetime.timezone.utc))]

2023-07-04 17:05:55.794:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 140] Stop orders retrieved from db table are 

2023-07-04 17:05:55.795:[INFO] [MainProcess] [MainThread] [stop_orders_test.py]:[test_stop_order_submit_datanode_bug]:[ 141] []