Crash in network_broadcast_api::on_applied_block

[theoreticalbts]

@roadscape reports a crash only reproducible on his workstation, the only obvious potential crash bug is fixed here but he reports the crash continues to occur even with this patch.

[roadscape]

Crash 1 - I also happened to lose internet connection at this exact time. What's strange is that it does not appear my tx was included in a block, yet it crashed anyway.

378609ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":24,"method":"call","params":[1,"get_api_by_name",["network_broadcast_api"]]}
378609ms th_a       api.cpp:107                   get_api_by_name      ] found api: network_broadcast_api
378616ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":25,"method":"call","params":[4,"broadcast_transaction_with_callback",[0,{"ref_block_num":44399,"ref_block_prefix":1698810405,"expiration":"2016-09-09T20:06:03","operations":[["vote",{"voter":"test-safari","author":"peskov","permlink":"wings-featuring-vasily-peskov-as-author","weight":10000}]],"extensions":[],"signatures":["20442697dc02670584e8cc22f5800c4179334484549309bd730305bc8ed5d5ad9d3454d4207ba9d4745b8d06968a8711e66a819824fe855d7ccd0de56dffba109c"]}]]}
400176ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":26,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
430181ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":27,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
465126ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":28,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
500452ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":29,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
541159ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":30,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
572074ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":31,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
587781ms ntp        ntp.cpp:98                    request_now          ] 0 exception: unspecified
process exited with: Host not found (authoritative)
    {"message":"Host not found (authoritative)"}
    asio  asio.cpp:88 resolve_handler

    {}
    ntp  asio.cpp:168 resolve
616272ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":32,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
650519ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":33,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
695747ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":34,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
738593ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":35,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
772772ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":36,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
779682ms th_a       api.cpp:148                   on_applied_block     ] in network_broadcast_api::on_applied_block() (block={b})
779684ms th_a       api.cpp:148                   on_applied_block     ] in network_broadcast_api::on_applied_block() (block={b})
779686ms th_a       api.cpp:148                   on_applied_block     ] in network_broadcast_api::on_applied_block() (block={b})
779687ms th_a       api.cpp:148                   on_applied_block     ] in network_broadcast_api::on_applied_block() (block={b})
779688ms th_a       api.cpp:148                   on_applied_block     ] in network_broadcast_api::on_applied_block() (block={b})
779688ms th_a       api.cpp:185                   on_applied_block     ] processing expiration for tx 524c894f0cc42b8ae2ad06e2847b851bc707c25b in block 4828532
779688ms th_a       api.cpp:193                   on_applied_block     ] tx was not confirmed
779688ms th_a       api.cpp:197                   on_applied_block     ] adding fc::async task
Segmentation fault: 11

Crash 2 - normal connectivity

1108845ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":14,"method":"call","params":[1,"get_api_by_name",["network_broadcast_api"]]}
1108845ms th_a       api.cpp:107                   get_api_by_name      ] found api: network_broadcast_api
1108852ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":15,"method":"call","params":[4,"broadcast_transaction_with_callback",[0,{"ref_block_num":44634,"ref_block_prefix":3393032021,"expiration":"2016-09-09T20:18:42","operations":[["vote",{"voter":"test-safari","author":"xntryk1","permlink":"swapmeet-finds-135","weight":10000}]],"extensions":[],"signatures":["205b7f2c1fa690b9520237f16be1f37cf696b1238ccabe4f219ec7299df8dd98da45f0709191c859e2ab0db5a1cd3d8ff83188d0467ad9e01b8fc05ce3b2a3d2be"]}]]}
1108883ms th_a       websocket_api.cpp:88          on_message           ] message: {"id":14,"method":"call","params":[0,"get_dynamic_global_properties",[]]}
1110371ms th_a       api.cpp:148                   on_applied_block     ] in network_broadcast_api::on_applied_block() (block={b})
1110371ms th_a       api.cpp:158                   on_applied_block     ] processing callback for tx 4828780:2=4f7940780572a91fea4e0ba3f9d6ca93d668688e
1110371ms th_a       api.cpp:160                   on_applied_block     ] adding fc::async task
Segmentation fault: 11

[theoreticalbts]

So basically according to @bytemaster the solution here is to re-architect the code so the API method only queues responses, and a separate task actually writes out those responses, so we're not calling fc::async() in the API method.

[abitmore]

Pull request for fixing crash 1: steemit/fc#5

//Update:
It seems all NTP exceptions have been caught somewhere, so should not be the reason of crash.
Similar issue was reported in #77 long ago.

[theoreticalbts]

Likely duplicate of #36. The network_broadcast API will be deprecated and replaced by a stateless network_broadcast2 API, as discussed in #490. Closing this ticket as wontfix.