Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Give CValidationInterface Support for calling notifications on the CScheduler Thread #10179

Merged
merged 7 commits into from Jul 11, 2017

Conversation

Projects
None yet
7 participants
@TheBlueMatt
Copy link
Contributor

commented Apr 10, 2017

Apologies for the plumbing-only no-changes PR, but the "move wallet updates out of cs_main into a background thread" changeset is a bit big for all one go, so instead I'm doing this first. It simply gives CValidationInterface the neccessary plumbing to handle callbacks on the CScheduler thread. See TheBlueMatt@4e82e40 for a commit which switches a callback into the background thread.

The CScheduler thread was super lonely, so I decided to use that instead of adding new threads.

This conflicts trivially with #10178, but not enough to avoid doing parallel reviews. After that and this are merged, "move wallet callbacks into background thread" is just one more (somewhat large, sadly) PR away. See https://github.com/TheBlueMatt/bitcoin/commits/2017-01-wallet-cache-inmempool for the full branch.

@fanquake fanquake added the Validation label Apr 10, 2017

@laanwj laanwj added this to Blockers in High-priority for review Apr 13, 2017

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch Apr 17, 2017

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2017

Rebased after ##10178 merge.

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch Apr 17, 2017

src/validationinterface.cpp Outdated
// our own queue here :(
CCriticalSection cs_callbacksPending;
std::list<std::function<void (void)>> callbacksPending;
std::atomic<bool> fCallbacksRunning = ATOMIC_VAR_INIT(false);

This comment has been minimized.

Copy link
@morcos

morcos Apr 24, 2017

Member

nit: i'm the last person who should comment on coding style, but isn't this easier to read:
std::atomic<bool> fCallbacksRunning(false)

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 24, 2017

Author Contributor

My C++-fu is not good enough to get that to compile. Not actually sure why, though, frankly.

This comment has been minimized.

Copy link
@sipa

sipa Apr 24, 2017

Member

Could you try

std::atomic<bool> fCallbacksRunning = std::atomic<bool>(false);

?

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 24, 2017

Author Contributor

Yes, that works, but I think thats worse than using the init wrapper, which is what that wrapper exists for.

This comment has been minimized.

Copy link
@theuni

theuni Apr 26, 2017

Member

Use braced initialization in a class/struct definition:

std::atomic<bool> fCallbacksRunning{false};

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 26, 2017

Author Contributor

Rewrote to just use a regular bool inside the lock, no reason to really have an atomic here.

@morcos

This comment has been minimized.

Copy link
Member

commented Apr 24, 2017

fast review utACK

@laanwj

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

utACK 010f3ae

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2017

src/validationinterface.cpp Outdated
callback = callbacksPending.front();
callbacksPending.pop_front();
}
callback();

This comment has been minimized.

Copy link
@laanwj

laanwj Apr 25, 2017

Member

This doesn't seem exception-safe. If an exception is raised inside here, fCallbacksRunning will stay set forever.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 25, 2017

Author Contributor

Indeed. Put a generic try {} catch(...) {Log} around it, not sure what else to do there.

This comment has been minimized.

Copy link
@laanwj

laanwj Apr 26, 2017

Member

You could catch the exception, clear the flag, and throw again. Though RAII is usually the best way in C++ to cover all exit paths.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 26, 2017

Author Contributor

OK, made a local class that will RAII it :)

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch to 1bcb07f Apr 25, 2017

@sipa
Copy link
Member

left a comment

I'll give you the same comment as you gave on #10148... you're adding unused and untested functionality.

The first commit seems a useful refactoring that could be its own PR, but the rest seem to be preparations for something that doesn't exist yet.

}

CMainSignals::~CMainSignals() {
delete internals;

This comment has been minimized.

Copy link
@sipa

sipa Apr 26, 2017

Member

Use a std::unique_ptr instead?

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 26, 2017

Author Contributor

Cant, sadly, as the whole point was to not make CMainSignalsInterface sit in the .h to avoid having a bost/signals include in half our codebase.

This comment has been minimized.

Copy link
@sipa

sipa Apr 26, 2017

Member

std::unique_ptr seems to work fine with forward-declared types: sipa@a4ecaad

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 26, 2017

Author Contributor

Well, I'll be! Took your patch and squashed (hope you dont mind).

This comment has been minimized.

Copy link
@sipa

sipa Apr 26, 2017

Member

Sorry, but I'm going to insist that you maintain this commit intact (nevermind that it's likely the exact same patch you'd come up with if I hadn't shown you) forever in your PR history. Note: sarcasm.

}

void AddToProcessQueue(const std::function<void (void)>& func) {
if (!scheduler)

This comment has been minimized.

Copy link
@sipa

sipa Apr 26, 2017

Member

That's scary, it would lose a callback. Shouldn't it assert in this case?

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 26, 2017

Author Contributor

assert()ed

return;
}

while (true) {

This comment has been minimized.

Copy link
@sipa

sipa Apr 26, 2017

Member

Instead of looping until there are no more scheduled callbacks, isn't easier/better to return, but reschedule itself? That way a large set of queued callbacks won't prevent the scheduler from running other (non-callback) jobs in the mean time.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Apr 26, 2017

Author Contributor

Done.

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch 9 times, most recently Apr 26, 2017

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented Apr 27, 2017

@sipa re: unused and untested code, see #10286, which I think it would be reasonable to require at least concept acks on prior to merging this.

@ryanofsky
Copy link
Contributor

left a comment

utACK 4fceec59372b778f9966485f2a13f97e21b1db4c

src/validationinterface.h Outdated
struct CMainSignalsInstance;
class CMainSignals {
private:
std::unique_ptr<CMainSignalsInstance> internals;

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky May 2, 2017

Contributor

In commit "Make ValidationInterface signals-type-agnostic"

I don't think you need to have this internals pointer in order to do the signal routing / CScheduler stuff that's ostensibly the motivation for this change. (The signals could just be regular class members.). If you like this better because it helps hide the boost dependency, though, that seems fine. This is just the pImpl pattern, http://stackoverflow.com/questions/8972588/is-the-pimpl-idiom-really-used-in-practice.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt May 2, 2017

Author Contributor

Yea, the real motivation there was to not have to #include boost signals garbage everywhere, nothing more.

src/validationinterface.cpp Outdated
// We are not allowed to assume the scheduler only runs in one thread,
// but must ensure all callbacks happen in-order, so we end up creating
// our own queue here :(
CCriticalSection cs_callbacksPending;

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky May 2, 2017

Contributor

In commit "Handle more than one CScheduler thread in CValidationInterface"

This seems like it would a lot simpler it just spawned a single thread to run the callbacks in sequence instead of relying on CScheduler and then doing a bunch of synchronization on top of it to keep things scheduled and prevent more than one callback from executing at the same time.

Also, I think it would be nice if this functionality were implemented in a standalone class that could be unit tested.

Anyway, implementation seems fine, I couldn't find any bugs.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt May 2, 2017

Author Contributor

Moved to scheduler.h, testing left as an excersize for the reader :p.

src/validationinterface.cpp Outdated
callback();
}

void AddToProcessQueue(const std::function<void (void)>& func) {

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky May 2, 2017

Contributor

In commit "Handle more than one CScheduler thread in CValidationInterface":

Should pass func by value or rvalue reference so you can move it into the callbacksPending list without a copy.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt May 2, 2017

Author Contributor

Done.

src/validationinterface.cpp Outdated
if (callbacksPending.empty()) return;
fCallbacksRunning = true;

callback = callbacksPending.front();

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky May 2, 2017

Contributor

In commit "Handle more than one CScheduler thread in CValidationInterface"

Could add std::move here.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt May 2, 2017

Author Contributor

Done.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented May 3, 2017

utACK 643a988101f12581725d08d71cfcb5d932a062ac. I like the new ProcessQueue location and std::move additions.

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch May 4, 2017

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2017

Squashed and tweaked commit wording.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented May 5, 2017

utACK 830c26b55a2b4bc93509d3fa7f17a0b383a95835. Only change aside from the history squashing is switching to the schedule() default arg.

src/validationinterface.cpp Outdated
}

void CMainSignals::UnregisterBackgroundSignalScheduler() {
m_internals->m_scheduler = NULL;
m_internals.reset(nullptr);

This comment has been minimized.

Copy link
@theuni

theuni Jun 27, 2017

Member

This potentially leaves events in the SingleThreadedSchedulerClient process queue with a dangling pointer to "this", no? I believe SingleThreadedSchedulerClient needs interrupt/stop functionality.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Jun 27, 2017

Author Contributor

Interrupt() will stop the processing of events long before we ever call UnregisterBackgroundSignalScheduler in init/bitcoind.cpp/qt/bitcoin.cpp. There isnt really a great way to solve this, I think, without just making the CScheduler own the thread(/group), but I went ahead and pushed a commit which will just call any remaining callbacks when Unregister is called.

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Jul 10, 2017

Contributor

In commit "Support more than one CScheduler thread for serial clients"

It'd be good to just reset the scheduler pointer here instead of going overboard and destroying all the boost signals at this point as well. It just seems like a random and unexpected thing to be doing in a function called UnregisterBackgroundSignalScheduler especially given new flush behavior which allows signals be forwarded after the scheduler is stopped.

@laanwj

This comment has been minimized.

Copy link
Member

commented Jun 28, 2017

@sipa Just that I had a concern at the time, and was too quick to utACK. I should re-review.

src/validationinterface.h Outdated
/** Notifies listeners that a key for mining is required (coinbase) */
boost::signals2::signal<void (std::shared_ptr<CReserveScript>&)> ScriptForMining;
virtual void GetScriptForMining(std::shared_ptr<CReserveScript>&) {};

This comment has been minimized.

Copy link
@laanwj

laanwj Jun 28, 2017

Member

#10683 gets rid of GetScriptForMining in validationinterface, which I think it's good because it mutates its argument. Doing that in a decoupled thread would be disastrous.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Jun 28, 2017

Author Contributor

All of the move-to-background-thread stuff is going to be a slow per-function process. See, eg, #10652 which moves a few functions to the background for net_processing and, of course, #10286 which moves a few to the background for wallet. Agreed that GetScriptForMining should go away and then it wont be a concern :).

@laanwj laanwj self-assigned this Jun 28, 2017

@laanwj laanwj removed this from Blockers in High-priority for review Jun 29, 2017

@theuni

This comment has been minimized.

Copy link
Member

commented Jul 3, 2017

utACK (after rebase) now that #10683 is in.

TheBlueMatt added some commits Apr 27, 2017

Make ValidationInterface signals-type-agnostic
(by hiding boost::signals stuff in the .cpp)

This allows us to give it a bit more intelligence as we move
forward, including routing some signals through CScheduler. While
the introduction of a "internals" pointer in the class is pretty
ugly, the fact that we no longer need to include boost/signals
directly from validationinterface.h is very much worth the loss.
@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented Jul 4, 2017

Rebased.

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch 2 times, most recently Jul 4, 2017

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented Jul 7, 2017

Redid the shutdown callback-flushing at @morcos's request - now flushes earlier in the Shutdown() process in a much better location - after all our peers have stopped processing (and thus cant generate callbacks) and before wallet flushing.

TheBlueMatt added some commits Jan 19, 2017

Give CMainSignals a reference to the global scheduler
...so that it can run some signals in the background later
Support more than one CScheduler thread for serial clients
This will be used by CValidationInterface soon.

This requires a bit of work as we need to ensure that most of our
callbacks happen in-order (to avoid synchronization issues in
wallet) - we keep our own internal queue and push things onto it,
scheduling a queue-draining function immediately upon new
callbacks.

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch 2 times, most recently Jul 7, 2017

Flush CValidationInterface callbacks prior to destruction
Note that the CScheduler thread cant be running at this point,
it has already been stopped with the rest of the init threadgroup.
Thus, just calling any remaining loose callbacks during Shutdown()
is sane.

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2017-01-wallet-cache-inmempool-3 branch to 3192975 Jul 7, 2017

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

commented Jul 7, 2017

Added a comment on flush/drop behavior -
"Any future callbacks will be dropped. This should absolutely be safe - if missing a callback results in an unrecoverable situation, unclean shutdown would too. The only reason to do the above flushes is to let the wallet catch up with our current chain to avoid any strange pruning edge cases and make next startup faster by avoiding rescan."

@morcos

This comment has been minimized.

Copy link
Member

commented Jul 7, 2017

utACK 3192975

@theuni

This comment has been minimized.

Copy link
Member

commented Jul 7, 2017

re-utACK 3192975, though very hesitantly after thinking through the tear-down some more. The shutdown process is going to get really hard to trace when callbacks start coming in on different threads.

I really hope some ownership model will begin to emerge.

@ryanofsky
Copy link
Contributor

left a comment

utACK 3192975. Changes since last review were rebase after multiwallet & ScriptForMining removal, and adding the flush commit.

I left a few suggestions pertaining to the flush commit. I think this code would be good to clean up at some point, though it don't think it should hold up the PR.

@@ -44,6 +44,10 @@ void CMainSignals::UnregisterBackgroundSignalScheduler() {
m_internals.reset(nullptr);
}

void CMainSignals::FlushBackgroundCallbacks() {
m_internals->m_schedulerClient.EmptyQueue();

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Jul 10, 2017

Contributor

In commit "Flush CValidationInterface callbacks prior to destruction"

Shutdown order is convoluted enough that I think it would be good to add asserts, like asserting m_internals is non-null here and that scheduler is shut down or shutting down at this point (nThreadsServicingQueue == 0 || shouldStop). Latter might be more appropriate inside the scheduling code.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Jul 11, 2017

Author Contributor

Added.

@@ -251,6 +251,7 @@ void Shutdown()
}
#endif
UnregisterAllValidationInterfaces();
GetMainSignals().UnregisterBackgroundSignalScheduler();

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Jul 10, 2017

Contributor

In commit "Give CMainSignals a reference to the global scheduler"

It seems like the right place for this unregister call would be earlier in Shutdown(), after scheduler thread is cancelled and the last signal is sent, for consistency with the register call, which is made when the scheduler thread is started.

This would let you flush the background queue when the signal scheduler is destroyed and not need separate FlushBackgroundCallbacks and EmptyQueue calls made in advance.

Also this would bring shutdown code closer to an raii-style ordering, which should make cleanup easier in the future and would make the various object lifetimes a little easier to understand now.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Jul 11, 2017

Author Contributor

We cant unregister the background scheduler from the validation interface until we're sure nothing is gonna generate anymore callbacks (so, really, after the pcoinsTip/etc deletions). If we want to mirror the initialization order, we'd have to move it even further down, not up, as de-init in RAII order would be after wallet deltion.

@@ -101,6 +101,9 @@ class SingleThreadedSchedulerClient {
public:
SingleThreadedSchedulerClient(CScheduler *pschedulerIn) : m_pscheduler(pschedulerIn) {}
void AddToProcessQueue(std::function<void (void)> func);

// Processes all remaining queue members on the calling thread, blocking until queue is empty

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Jul 10, 2017

Contributor

In commit "Flush CValidationInterface callbacks prior to destruction"

Comment should point out that this should only be called when scheduler is not active. Otherwise this could burn cpu racing with scheduler thread to run the next callback.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Jul 11, 2017

Author Contributor

Added the assert instead

src/validationinterface.cpp Outdated
}

void CMainSignals::UnregisterBackgroundSignalScheduler() {
m_internals->m_scheduler = NULL;
m_internals.reset(nullptr);

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Jul 10, 2017

Contributor

In commit "Support more than one CScheduler thread for serial clients"

It'd be good to just reset the scheduler pointer here instead of going overboard and destroying all the boost signals at this point as well. It just seems like a random and unexpected thing to be doing in a function called UnregisterBackgroundSignalScheduler especially given new flush behavior which allows signals be forwarded after the scheduler is stopped.

@laanwj laanwj merged commit 1f668b6 into bitcoin:master Jul 11, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Jul 11, 2017

Merge #10179: Give CValidationInterface Support for calling notificat…
…ions on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5

laanwj added a commit that referenced this pull request Nov 15, 2017

Merge #10286: Call wallet notify callbacks in scheduler thread (witho…
…ut cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on #10179, this effectively reverts #9583, regaining most of the original speedups of #7946.

  This concludes the work of #9725, #10178, and #10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9

@codablock codablock referenced this pull request Mar 15, 2019

Merged

Backports 0.15 pr8 #2763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.