Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added graceful shutdown hook #4047

Merged
merged 9 commits into from
Mar 31, 2020
Merged

Conversation

freimair
Copy link
Member

@freimair freimair commented Mar 11, 2020

Fixes #3884, fixes #2465

Graceful shutdown has only been done in case of an error or when
using the GUI. A regular eg. seednode shutdown is not covered
though.

Now, a kill -TERM ... or a kill -INT or [Ctrl+C] triggers a graceful shutdown procedure.

Tested on a Bisq app and seednode service launch by firing up and then

  • kill -TERM javapid
  • kill -INT javapid
  • run from terminal followed by hitting [Ctrl+C]

@wiz
Copy link
Member

wiz commented Mar 11, 2020

@ripcurlx @sqrrm is it too late to get this into v1.2.8 ?

@wiz
Copy link
Member

wiz commented Mar 17, 2020

I was expecting to see the "Bisq is shutting down" modal popup, same as cmd + Q to quit, but it did not appear. Is it really doing the graceful shutdown same as cmd + Q ? Also, can you handle SIGINT as well as SIGTERM ?

Copy link
Member

@wiz wiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expected Result

Last line of log output should be

Mar-18 02:28:35.012 [JavaFX Application Thread] INFO  bisq.core.app.BisqExecutable: Graceful shutdown completed. Exiting now. 

Actual Result

Last line of log output is random

Copy link
Contributor

@ripcurlx ripcurlx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NACK - #4047 (review)

Could you please have a look at the comment by @wiz? Thanks!

@freimair
Copy link
Member Author

freimair commented Mar 24, 2020

ah yes, I had the hopes that after achieving the same for the monitor back in the days, it would also work here.

2 issues:

  • first, I found and fixed the shutdown chaos Tor error message at shutdown #2465. I rearranged the shutdown procedure, now at least via the GUI, graceful shutdown works as expected (to be refined and committed)
  • second, the OS does forward the TERM to all child processes and thus, tor gets shut down before we are ready for it. I had solved that somehow for the monitor, but right now, I am having a hard time finding out why it does work there but not here.

stand by...

@wiz: even if it works someday this will never display something in the GUI. The gui does not have the necessary infrastructure.

Graceful shutdown has only be done in case of an error or when
using the GUI. A regular eg. seednode shutdown is not covered
though.

Now, SIGTERM and SIGINT triggers a graceful shutdown procedure.
The close connection process did fire up worker threads to actually
close the connections. Yet, once all threads have been spawned,
the code proceeds assuming that there are no connections left open
without checking.
This lead to situations where tor has been shutdown already but
open connections. These connections tried to gracefully close but
without tor, that only caused a wall of exceptions.
@@ -225,7 +225,7 @@ public void onSuccess(Connection connection) {
}

public void onFailure(@NotNull Throwable throwable) {
log.info("onFailure at sendMessage: peersNodeAddress={}\n\tmessage={}\n\tthrowable={}", peersNodeAddress, networkEnvelope.getClass().getSimpleName(), throwable.toString());
log.debug("onFailure at sendMessage: peersNodeAddress={}\n\tmessage={}\n\tthrowable={}", peersNodeAddress, networkEnvelope.getClass().getSimpleName(), throwable.toString());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific reason why we don't want to have this in our logs anymore?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this logline is only triggered in a very limited and rare set of situations. One of them being after tor has been shutdown already (but not if tor is gone without a trigger from Bisq). Plus, this log line does not help support. Iff the info is necessary, --logLevel=DEBUG is there.

Generally, log outputs are scattered throughout the p2p package. I am currently working on a battle plan to refactor the message handling in order to get more reliability and control over what happens (we do have a lost message every now and then, and I found hints on why that is).

Here, the tor object is a member variable and there are cases where
this member variable is not set yet.
Situation arose where a sigterm/sigint shutdown is requested and due
to the member variable not set tor was left running.
The actual System::exit commands have been scattered around various
places in the code. Sometimes, actual system exit depended on the
calling code to reach its end of execution.
Before, the graceful shutdown procedures have been executed in the
user thread. However, the sync mechanics for connections/offers
caused a lockup, as some little parts of the code do execute on the
user thread as well.
@ripcurlx ripcurlx merged commit 9891e58 into bisq-network:master Mar 31, 2020
@ripcurlx ripcurlx added this to the v1.3.0 milestone Apr 1, 2020
@freimair freimair deleted the graceful_shutdown branch April 2, 2020 11:57
wiz added a commit to wiz/bisq that referenced this pull request Aug 1, 2020
wiz added a commit to wiz/bisq that referenced this pull request Aug 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bisq does not handle SIGTERM properly Tor error message at shutdown
3 participants