get_fee error: invalid response

[bh2smith]

We have been getting a fairly steady flow of the following in-descriptive errors in our logs

ERROR orderbook::api::get_fee_info: get_fee error err=Got invalid response: Error("data did not match any variant of untagged enum Response", line: 0, column: 0)

which appears to be a long outstanding (non urgent) issue with serde-rs

This is the place in the code is here

http://github.com/gnosis/gp-v2-services/blob/f8739a5178ae75de49bd90575c155fc6c8586d73/orderbook/src/api/get_fee_info.rs#L56-L57

and in the same way again here:

http://github.com/gnosis/gp-v2-services/blob/f8739a5178ae75de49bd90575c155fc6c8586d73/orderbook/src/api/get_fee_info.rs#L123-L124

I have not managed to recover this response in a way that would allow us to adapt the code and catch this error. I am beginning to suspect that the response is actually "no response" so perhaps we could try and work with that.

[vkgnosis]

We are probably missing some .context(...) somewhere which is why we don't see where the error comes from. I'll go through the get_fee code now to find the culprit.

[nlordell]

The issue is a trace_callMany call is not returning a valid JSON RPC repsonse.

Specifically:

2021-05-28T06:19:40.855Z DEBUG shared::transport: [id:1216000] sending request: '{"jsonrpc":"2.0","method":"trace_callMany","params":[[[{"data":"0x70a082310000000000000000000000003328f5f2cecaf00a2443082b657cedeaf70bfaef","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]],[{"data":"0xa9059cbb0000000000000000000000003328f5f2cecaf00a2443082b657cedeaf70bfaef0000000000000000000000000000000000000000000000000de0b6b3a7640000","from":"0xe632ded5195e945a31f56d674aab0c0c9e7e812c","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]],[{"data":"0x70a082310000000000000000000000003328f5f2cecaf00a2443082b657cedeaf70bfaef","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]],[{"data":"0x70a08231000000000000000000000000cd7328a5d376d5530f054eaf0b9d235a4fd36059","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]],[{"data":"0xa9059cbb000000000000000000000000cd7328a5d376d5530f054eaf0b9d235a4fd360590000000000000000000000000000000000000000000000000de0b6b3a7640000","from":"0x3328f5f2cecaf00a2443082b657cedeaf70bfaef","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]],[{"data":"0x70a082310000000000000000000000003328f5f2cecaf00a2443082b657cedeaf70bfaef","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]],[{"data":"0x70a08231000000000000000000000000cd7328a5d376d5530f054eaf0b9d235a4fd36059","to":"0xdd1ad9a21ce722c151a836373babe42c868ce9a4"},["trace"]]],"latest"],"id":1216000}'
...
2021-05-28T06:19:59.020Z DEBUG shared::transport: [id:1216000] returned an error: 'Got invalid response: Error("data did not match any variant of untagged enum Response", line: 0, column: 0)'

[nlordell]

While the context would help, it would be really nice to see the actual response to know what the node is sending back.

[vkgnosis]

We log the response in the transports logs already though, right? I could add it to the error directly but not sure how useful that is because it could make it very long.

[vkgnosis]

Ah we probably don't log anything if it can't decode the outer jsonrpc json.

[bh2smith]

Yes, these solutions are definitely an improvement, but it would be really nice to know what exactly we are failing to enumerate and catch this in a way that we can adapt to. I have asked devops for the response data and hope to at least be able to catch this response in a way that we can deal with in the future.

[vkgnosis]

Using our own http Transport implementation we would be able to log those responses too.

[bh2smith]

Using our own http Transport implementation we would be able to log those responses too.

Yes, I think we will have to do this (at least temporarily) because devops has not been able to recover the response data for this RPC request.

[vkgnosis]

Would be nice to have in general. I'll take stab at it.

[bh2smith]

BTW, @vkgnosis we got the response data from devops. It was an empty response.

[fleupold]

Would be nice to have in general. I'll take stab at it.

@vkgnosis not sure if you remember, but we had this in v1 (https://github.com/gnosis/dex-services/blob/2b4e608dd9555e4a9ecfe35d0d946a2f1fbb0140/services-core/src/transport.rs#L52). In this case we can probably get rid of the LoggingTransport Layer (we shouldn't log these verbose infos twice).

[bh2smith]

I believe this was closed by #682 - with our custom HTTP transport.

[nlordell]

I think this turned out to be a node issue.