Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Give CValidationInterface Support for calling notifications on the CScheduler Thread #10179

Merged
merged 7 commits into from Jul 11, 2017

Conversation

@TheBlueMatt
Copy link
Contributor

@TheBlueMatt TheBlueMatt commented Apr 10, 2017

Apologies for the plumbing-only no-changes PR, but the "move wallet updates out of cs_main into a background thread" changeset is a bit big for all one go, so instead I'm doing this first. It simply gives CValidationInterface the neccessary plumbing to handle callbacks on the CScheduler thread. See TheBlueMatt@4e82e40 for a commit which switches a callback into the background thread.

The CScheduler thread was super lonely, so I decided to use that instead of adding new threads.

This conflicts trivially with #10178, but not enough to avoid doing parallel reviews. After that and this are merged, "move wallet callbacks into background thread" is just one more (somewhat large, sadly) PR away. See https://github.com/TheBlueMatt/bitcoin/commits/2017-01-wallet-cache-inmempool for the full branch.

@laanwj laanwj added this to Blockers in High-priority for review Apr 13, 2017
@TheBlueMatt TheBlueMatt force-pushed the 2017-01-wallet-cache-inmempool-3 branch from 36ec5ee to 6d4e7db Apr 17, 2017
@TheBlueMatt
Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Apr 17, 2017

Rebased after ##10178 merge.

Loading

@TheBlueMatt TheBlueMatt force-pushed the 2017-01-wallet-cache-inmempool-3 branch from 6d4e7db to e5d0c66 Apr 17, 2017
// our own queue here :(
CCriticalSection cs_callbacksPending;
std::list<std::function<void (void)>> callbacksPending;
std::atomic<bool> fCallbacksRunning = ATOMIC_VAR_INIT(false);
Copy link
Member

@morcos morcos Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i'm the last person who should comment on coding style, but isn't this easier to read:
std::atomic<bool> fCallbacksRunning(false)

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My C++-fu is not good enough to get that to compile. Not actually sure why, though, frankly.

Loading

Copy link
Member

@sipa sipa Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you try

std::atomic<bool> fCallbacksRunning = std::atomic<bool>(false);

?

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that works, but I think thats worse than using the init wrapper, which is what that wrapper exists for.

Loading

Copy link
Member

@theuni theuni Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use braced initialization in a class/struct definition:

std::atomic<bool> fCallbacksRunning{false};

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrote to just use a regular bool inside the lock, no reason to really have an atomic here.

Loading

@morcos
Copy link
Member

@morcos morcos commented Apr 24, 2017

fast review utACK

Loading

@laanwj
Copy link
Member

@laanwj laanwj commented Apr 25, 2017

utACK 010f3ae

Loading

@TheBlueMatt
Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Apr 25, 2017

Loading

callback = callbacksPending.front();
callbacksPending.pop_front();
}
callback();
Copy link
Member

@laanwj laanwj Apr 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem exception-safe. If an exception is raised inside here, fCallbacksRunning will stay set forever.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. Put a generic try {} catch(...) {Log} around it, not sure what else to do there.

Loading

Copy link
Member

@laanwj laanwj Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could catch the exception, clear the flag, and throw again. Though RAII is usually the best way in C++ to cover all exit paths.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, made a local class that will RAII it :)

Loading

@TheBlueMatt TheBlueMatt force-pushed the 2017-01-wallet-cache-inmempool-3 branch from e5d0c66 to 1bcb07f Apr 25, 2017
Copy link
Member

@sipa sipa left a comment

I'll give you the same comment as you gave on #10148... you're adding unused and untested functionality.

The first commit seems a useful refactoring that could be its own PR, but the rest seem to be preparations for something that doesn't exist yet.

Loading

}

CMainSignals::~CMainSignals() {
delete internals;
Copy link
Member

@sipa sipa Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a std::unique_ptr instead?

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cant, sadly, as the whole point was to not make CMainSignalsInterface sit in the .h to avoid having a bost/signals include in half our codebase.

Loading

Copy link
Member

@sipa sipa Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

std::unique_ptr seems to work fine with forward-declared types: sipa@a4ecaad

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I'll be! Took your patch and squashed (hope you dont mind).

Loading

Copy link
Member

@sipa sipa Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but I'm going to insist that you maintain this commit intact (nevermind that it's likely the exact same patch you'd come up with if I hadn't shown you) forever in your PR history. Note: sarcasm.

Loading

}

void AddToProcessQueue(const std::function<void (void)>& func) {
if (!scheduler)
Copy link
Member

@sipa sipa Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's scary, it would lose a callback. Shouldn't it assert in this case?

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert()ed

Loading

return;
}

while (true) {
Copy link
Member

@sipa sipa Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of looping until there are no more scheduled callbacks, isn't easier/better to return, but reschedule itself? That way a large set of queued callbacks won't prevent the scheduler from running other (non-callback) jobs in the mean time.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Loading

@TheBlueMatt TheBlueMatt force-pushed the 2017-01-wallet-cache-inmempool-3 branch 9 times, most recently from 4e96a11 to 4fceec5 Apr 27, 2017
@TheBlueMatt
Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Apr 27, 2017

@sipa re: unused and untested code, see #10286, which I think it would be reasonable to require at least concept acks on prior to merging this.

Loading

Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK 4fceec59372b778f9966485f2a13f97e21b1db4c

Loading

struct CMainSignalsInstance;
class CMainSignals {
private:
std::unique_ptr<CMainSignalsInstance> internals;
Copy link
Contributor

@ryanofsky ryanofsky May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Make ValidationInterface signals-type-agnostic"

I don't think you need to have this internals pointer in order to do the signal routing / CScheduler stuff that's ostensibly the motivation for this change. (The signals could just be regular class members.). If you like this better because it helps hide the boost dependency, though, that seems fine. This is just the pImpl pattern, http://stackoverflow.com/questions/8972588/is-the-pimpl-idiom-really-used-in-practice.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, the real motivation there was to not have to #include boost signals garbage everywhere, nothing more.

Loading

// We are not allowed to assume the scheduler only runs in one thread,
// but must ensure all callbacks happen in-order, so we end up creating
// our own queue here :(
CCriticalSection cs_callbacksPending;
Copy link
Contributor

@ryanofsky ryanofsky May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Handle more than one CScheduler thread in CValidationInterface"

This seems like it would a lot simpler it just spawned a single thread to run the callbacks in sequence instead of relying on CScheduler and then doing a bunch of synchronization on top of it to keep things scheduled and prevent more than one callback from executing at the same time.

Also, I think it would be nice if this functionality were implemented in a standalone class that could be unit tested.

Anyway, implementation seems fine, I couldn't find any bugs.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to scheduler.h, testing left as an excersize for the reader :p.

Loading

callback();
}

void AddToProcessQueue(const std::function<void (void)>& func) {
Copy link
Contributor

@ryanofsky ryanofsky May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Handle more than one CScheduler thread in CValidationInterface":

Should pass func by value or rvalue reference so you can move it into the callbacksPending list without a copy.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Loading

if (callbacksPending.empty()) return;
fCallbacksRunning = true;

callback = callbacksPending.front();
Copy link
Contributor

@ryanofsky ryanofsky May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Handle more than one CScheduler thread in CValidationInterface"

Could add std::move here.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Loading

@ryanofsky
Copy link
Contributor

@ryanofsky ryanofsky commented May 3, 2017

utACK 643a988101f12581725d08d71cfcb5d932a062ac. I like the new ProcessQueue location and std::move additions.

Loading

@TheBlueMatt TheBlueMatt force-pushed the 2017-01-wallet-cache-inmempool-3 branch from d3f0ad3 to 830c26b May 4, 2017
@TheBlueMatt
Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented May 4, 2017

Squashed and tweaked commit wording.

Loading

@ryanofsky
Copy link
Contributor

@ryanofsky ryanofsky commented May 5, 2017

utACK 830c26b55a2b4bc93509d3fa7f17a0b383a95835. Only change aside from the history squashing is switching to the schedule() default arg.

Loading

Note that the CScheduler thread cant be running at this point,
it has already been stopped with the rest of the init threadgroup.
Thus, just calling any remaining loose callbacks during Shutdown()
is sane.
@TheBlueMatt TheBlueMatt force-pushed the 2017-01-wallet-cache-inmempool-3 branch from c45d59c to 3192975 Jul 7, 2017
@TheBlueMatt
Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Jul 7, 2017

Added a comment on flush/drop behavior -
"Any future callbacks will be dropped. This should absolutely be safe - if missing a callback results in an unrecoverable situation, unclean shutdown would too. The only reason to do the above flushes is to let the wallet catch up with our current chain to avoid any strange pruning edge cases and make next startup faster by avoiding rescan."

Loading

@morcos
Copy link
Member

@morcos morcos commented Jul 7, 2017

utACK 3192975

Loading

@theuni
Copy link
Member

@theuni theuni commented Jul 7, 2017

re-utACK 3192975, though very hesitantly after thinking through the tear-down some more. The shutdown process is going to get really hard to trace when callbacks start coming in on different threads.

I really hope some ownership model will begin to emerge.

Loading

Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK 3192975. Changes since last review were rebase after multiwallet & ScriptForMining removal, and adding the flush commit.

I left a few suggestions pertaining to the flush commit. I think this code would be good to clean up at some point, though it don't think it should hold up the PR.

Loading

@@ -44,6 +44,10 @@ void CMainSignals::UnregisterBackgroundSignalScheduler() {
m_internals.reset(nullptr);
}

void CMainSignals::FlushBackgroundCallbacks() {
m_internals->m_schedulerClient.EmptyQueue();
Copy link
Contributor

@ryanofsky ryanofsky Jul 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Flush CValidationInterface callbacks prior to destruction"

Shutdown order is convoluted enough that I think it would be good to add asserts, like asserting m_internals is non-null here and that scheduler is shut down or shutting down at this point (nThreadsServicingQueue == 0 || shouldStop). Latter might be more appropriate inside the scheduling code.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Jul 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Loading

@@ -251,6 +251,7 @@ void Shutdown()
}
#endif
UnregisterAllValidationInterfaces();
GetMainSignals().UnregisterBackgroundSignalScheduler();
Copy link
Contributor

@ryanofsky ryanofsky Jul 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Give CMainSignals a reference to the global scheduler"

It seems like the right place for this unregister call would be earlier in Shutdown(), after scheduler thread is cancelled and the last signal is sent, for consistency with the register call, which is made when the scheduler thread is started.

This would let you flush the background queue when the signal scheduler is destroyed and not need separate FlushBackgroundCallbacks and EmptyQueue calls made in advance.

Also this would bring shutdown code closer to an raii-style ordering, which should make cleanup easier in the future and would make the various object lifetimes a little easier to understand now.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Jul 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cant unregister the background scheduler from the validation interface until we're sure nothing is gonna generate anymore callbacks (so, really, after the pcoinsTip/etc deletions). If we want to mirror the initialization order, we'd have to move it even further down, not up, as de-init in RAII order would be after wallet deltion.

Loading

src/scheduler.h Outdated
@@ -101,6 +101,9 @@ class SingleThreadedSchedulerClient {
public:
SingleThreadedSchedulerClient(CScheduler *pschedulerIn) : m_pscheduler(pschedulerIn) {}
void AddToProcessQueue(std::function<void (void)> func);

// Processes all remaining queue members on the calling thread, blocking until queue is empty
Copy link
Contributor

@ryanofsky ryanofsky Jul 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Flush CValidationInterface callbacks prior to destruction"

Comment should point out that this should only be called when scheduler is not active. Otherwise this could burn cpu racing with scheduler thread to run the next callback.

Loading

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt Jul 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the assert instead

Loading

}

void CMainSignals::UnregisterBackgroundSignalScheduler() {
m_internals->m_scheduler = NULL;
m_internals.reset(nullptr);
Copy link
Contributor

@ryanofsky ryanofsky Jul 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Support more than one CScheduler thread for serial clients"

It'd be good to just reset the scheduler pointer here instead of going overboard and destroying all the boost signals at this point as well. It just seems like a random and unexpected thing to be doing in a function called UnregisterBackgroundSignalScheduler especially given new flush behavior which allows signals be forwarded after the scheduler is stopped.

Loading

@laanwj laanwj merged commit 1f668b6 into bitcoin:master Jul 11, 2017
1 check passed
Loading
laanwj added a commit that referenced this issue Jul 11, 2017
…ions on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
laanwj added a commit that referenced this issue Nov 15, 2017
…ut cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on #10179, this effectively reverts #9583, regaining most of the original speedups of #7946.

  This concludes the work of #9725, #10178, and #10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Aug 5, 2019
…tifications on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Aug 17, 2019
…tifications on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Aug 17, 2019
…tifications on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Sep 4, 2019
…tifications on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Sep 7, 2019
…tifications on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
barrystyle added a commit to PACGlobalOfficial/PAC that referenced this issue Jan 22, 2020
…tifications on the CScheduler Thread

1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)

Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Feb 13, 2020
… (without cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on bitcoin#10179, this effectively reverts bitcoin#9583, regaining most of the original speedups of bitcoin#7946.

  This concludes the work of bitcoin#9725, bitcoin#10178, and bitcoin#10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Feb 13, 2020
… (without cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on bitcoin#10179, this effectively reverts bitcoin#9583, regaining most of the original speedups of bitcoin#7946.

  This concludes the work of bitcoin#9725, bitcoin#10178, and bitcoin#10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Feb 29, 2020
… (without cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on bitcoin#10179, this effectively reverts bitcoin#9583, regaining most of the original speedups of bitcoin#7946.

  This concludes the work of bitcoin#9725, bitcoin#10178, and bitcoin#10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Feb 29, 2020
… (without cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on bitcoin#10179, this effectively reverts bitcoin#9583, regaining most of the original speedups of bitcoin#7946.

  This concludes the work of bitcoin#9725, bitcoin#10178, and bitcoin#10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Feb 18, 2021
…hread

6cecb7b AddToWalletIfInvolvingMe should test pIndex, not posInBlock (furszy)
73499ff Add missing lock in CScheduler::AreThreadsServicingQueue() (Matt Corallo)
6d1a60e ValidationInterface compiler warning fix (furszy)
9e523a7 Fix shutdown in case of errors during initialization (Wladimir J. van der Laan)
207c17f Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
64c525b Otherwise threads will try to continue modifying data structures while the dump process is being executed. (Matt Corallo)
8022463 Support more than one CScheduler thread for serial clients (Matt Corallo)
6532fca Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
d583d12 Give CMainSignals a reference to the global scheduler so that it can run some signals in the background later (Matt Corallo)
c26d18c Remove unused NotifyTransactionLock signal. (furszy)
2c9de45 Remove unused UpdatedTransaction signal and connected slot. (furszy)
8f38657 make ConnectBlock static in validation.cpp (furszy)

Pull request description:

  Another step forward in the wallet signals processing in the background thread + node validation <--> wallet separation long working paths.

   Adapting:
   * bitcoin#10178
   * bitcoin#10179
   * bitcoin#10221
   * bitcoin#10914
   * bitcoin#11783
   * Plus added an extra cleanup to the validation interface removing unused `NotifyTransactionLock` signal and its respective slots.

ACKs for top commit:
  Fuzzbawls:
    ACK 6cecb7b
  random-zebra:
    ACK 6cecb7b and merging...

Tree-SHA512: ac1c128cb3e4ec47c2803da68ca4d4b16164687fc790f3cc1d0761e73bd61b4edc681e2ce045e52564dd13a8279495e9058a6aae316196bd7ad9c97a07dd0b20
ckti added a commit to ckti-ioncore-current/ion that referenced this issue Mar 28, 2021
… (without cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on bitcoin#10179, this effectively reverts bitcoin#9583, regaining most of the original speedups of bitcoin#7946.

  This concludes the work of bitcoin#9725, bitcoin#10178, and bitcoin#10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
gades added a commit to cosanta/cosanta-core that referenced this issue Jun 30, 2021
… (without cs_main)

89f0312 Remove redundant pwallet nullptr check (Matt Corallo)
c4784b5 Add a dev notes document describing the new wallet RPC blocking (Matt Corallo)
3ea8b75 Give ZMQ consistent order with UpdatedBlockTip on scheduler thread (Matt Corallo)
cb06edf Fix wallet RPC race by waiting for callbacks in sendrawtransaction (Matt Corallo)
e545ded Also call other wallet notify callbacks in scheduler thread (Matt Corallo)
17220d6 Use callbacks to cache whether wallet transactions are in mempool (Matt Corallo)
5d67a78 Add calls to CWallet::BlockUntilSyncedToCurrentChain() in RPCs (Matt Corallo)
5ee3172 Add CWallet::BlockUntilSyncedToCurrentChain() (Matt Corallo)
0b2f42d Add CallFunctionInQueue to wait on validation interface queue drain (Matt Corallo)
2b4b345 Add ability to assert a lock is not held in DEBUG_LOCKORDER (Matt Corallo)
0343676 Call TransactionRemovedFromMempool in the CScheduler thread (Matt Corallo)
a7d3936 Add a CValidationInterface::TransactionRemovedFromMempool (Matt Corallo)

Pull request description:

  Based on bitcoin#10179, this effectively reverts bitcoin#9583, regaining most of the original speedups of bitcoin#7946.

  This concludes the work of bitcoin#9725, bitcoin#10178, and bitcoin#10179.

  See individual commit messages for more information.

Tree-SHA512: eead4809b0a75d1fb33b0765174ff52c972e45040635e38cf3686cef310859c1e6b3c00e7186cbd17374c6ae547bfbd6c1718fe36f26c76ba8a8b052d6ed7bc9
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants