Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sig 11 in portfolio::prices_loop #366

Closed
cipig opened this issue Mar 30, 2019 · 15 comments

Comments

@cipig
Copy link

commented Mar 30, 2019

My mm2 node crashed over night with this message:

Signal caught! sig 11
  crash_reports.rs:58] mm2::mm2::crash_reports::signal_handler::h146c77eccc77aa72
  prices.rs:79] portfolio::prices::PricePingRequest::new::h3982a2b8ad3051f9
  prices.rs:156] portfolio::prices::broadcast_my_prices::h8d395e195e86e463
  portfolio.rs:1178] portfolio::prices_loop::h284171406e50becd
  lib.rs:102] __rust_maybe_catch_panic

I will start it with gdb and try to get a backtrace.

@ArtemGr

This comment has been minimized.

Copy link
Collaborator

commented Mar 30, 2019

@artemii235

This comment has been minimized.

Copy link
Collaborator

commented Mar 30, 2019

The crash is caused by invoking unsafe request signing code, I will check this. Possibly just null-check is required.

@artemii235 artemii235 self-assigned this Mar 30, 2019

@cipig

This comment has been minimized.

Copy link
Author

commented Apr 14, 2019

Signal caught! sig 11
  crash_reports.rs:58] mm2::mm2::crash_reports::signal_handler::h84175a480c77aada
  prices.rs:79] portfolio::prices::PricePingRequest::new::h2565cfadd926512c
  prices.rs:159] portfolio::prices::broadcast_my_prices::hc88c36ca79b1a2f5
  portfolio.rs:1178] portfolio::prices_loop::ha2831cbebd4d2a34
  lib.rs:102] __rust_maybe_catch_panic

same crash, just with newer mm2 version... looks like the same, just some line numbers are different, but maybe it helps in finding the issue

@artemii235

This comment has been minimized.

Copy link
Collaborator

commented Apr 15, 2019

@cipig Thanks for 1 more report, yes, it's not fixed yet, will fix on this week.

artemii235 added a commit that referenced this issue Apr 15, 2019
@artemii235

This comment has been minimized.

Copy link
Collaborator

commented Apr 15, 2019

@cipig Please try to update your node(s) with the most recent commit and check that it doesn't crash anymore. You might see message Price request signature is null sometimes instead of crash, it's fine for now, MM2 will attempt to recreate the request in some time.

@cipig

This comment has been minimized.

Copy link
Author

commented Apr 15, 2019

mm2 updated. Could take a while to catch this though, it was happenning very rarely, one event in couple of days.

@cipig

This comment has been minimized.

Copy link
Author

commented Apr 18, 2019

It happenned. mm2 does not crash any more. But problem is that it does not recover from this. It started to show this messages

· 2019-04-18 13:03:02 +0000 [broadcast_my_prices XSN KMD] ping request creation failed prices:80] Price request signature is null
· 2019-04-18 13:03:02 +0000 [broadcast_my_prices XZC KMD] ping request creation failed prices:80] Price request signature is null
· 2019-04-18 13:03:02 +0000 [broadcast_my_prices KMD CHIPS] ping request creation failed prices:80] Price request signature is null

and continues like that since then.

@artemii235

This comment has been minimized.

Copy link
Collaborator

commented Apr 18, 2019

@cipig Hi, interesting, I will look into it again.

@cipig

This comment has been minimized.

Copy link
Author

commented May 11, 2019

a mm2 node with only BTC/KMD pair active is running for a week without getting this error
another node with 50 active pairs usually dies after 24h
seems related to the number of active pairs and thus the number of setprice calls done

@artemii235

This comment has been minimized.

Copy link
Collaborator

commented May 13, 2019

@cipig The issue happens in "old" message signing code that will be ported to Rust soon so it'll be fixed, probably on this week.

@artemii235 artemii235 added this to the Beta release milestone May 27, 2019

artemii235 added a commit that referenced this issue May 29, 2019
artemii235 added a commit that referenced this issue May 29, 2019
Merge pull request #416 from artemii235/mm2-price-ping-signature
Use Rust code to sing "postprice" requests #366
@artemii235

This comment has been minimized.

Copy link
Collaborator

commented May 29, 2019

@cipig it should be fixed now, could you test please? NB: code is not backwards compatible, please make sure that you update all your nodes.

@cipig

This comment has been minimized.

Copy link
Author

commented May 29, 2019

I will update all nodes, but to see if this gone, i need #414 too, because my setprice calls are without volume.

@cipig

This comment has been minimized.

Copy link
Author

commented May 29, 2019

All nodes updated.
Seeing this on all of them, every 10 seconds

29 19:21:37, common:678] RPC error response: lp_ordermatch:693] Invalid Signature

dont't know if it is related, but it does not seem to affect anything, orderbook-entries set with

curl --url "http://127.0.0.1:7783" --data "{\"userpass\":\"$userpass\",\"method\":\"setprice\",\"base\":\"RICK\",\"rel\":\"MORTY\",\"price\":1,\"volume\":1}"
curl --url "http://127.0.0.1:7783" --data "{\"userpass\":\"$userpass\",\"method\":\"setprice\",\"base\":\"MORTY\",\"rel\":\"RICK\",\"price\":1,\"volume\":1}"

are showing up fine on other nodes

@artemii235

This comment has been minimized.

Copy link
Collaborator

commented May 30, 2019

@cipig 29 19:21:37, common:678] RPC error response: lp_ordermatch:693] Invalid Signature is fine for now, you might receive a postprice message from outdated node which is not compatible with new version.

@cipig

This comment has been minimized.

Copy link
Author

commented Jun 3, 2019

mm2 node is running for several days and the problem didn't came back, so i say it is fixed.
Thanks.

@cipig cipig closed this Jun 3, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.