-
Notifications
You must be signed in to change notification settings - Fork 35.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace Connman and BanMan globals with NodeContext local #16839
Conversation
I'm looking for concept ACKS before detailed review at the moment. Since this a naming and refactoring change, it'd be great if people could take few minutes and skim the diff, and say whether the new code looks ugly or ok, and offer any suggestions. I'm happy to take suggestions and also split this up more. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK.
src/interfaces/chain.cpp
Outdated
{ | ||
ChainClients& clients = m_node.chain_clients; | ||
ChainClients::iterator it = clients.emplace(clients.end(), client); | ||
return MakeHandler([&clients, it] { clients.erase(it); }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it better to capture member chain_clients
than this
?
Also, it would be nice to explain that a std::list
is used otherwise this .erase
would fail.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it better to capture member
chain_clients
thanthis
?
I think a good starting point is to just capture what's needed without unused data or object references. But in this case, capturing this
instead of &clients
is worse because in addition to capturing unneeded data, it requires the Handler
object to be destroyed before the Chain
object, and there's no reason to create that kind of error-prone dependency.
Also, it would be nice to explain that a
std::list
is used otherwise this.erase
would fail.
Sure, added comment.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Concept ACK |
Fine with me, but this seems only like a first step, since other globals such as |
124bbca
to
7225ebe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for feedback!
Rebased 124bbca -> 7225ebe (pr/noglob.3 -> pr/noglob.4) due to conflict with #16787. Also added requested comment and no-wallet build fix.
re: #16839 (comment)
Fine with me, but this seems only like a first step, since other globals such as ::mempool are still global. and not members of the Node class.
Yes, it's not all or nothing. It's perfectly fine for ::mempool to remain a global, but if we want to write multiple-instance tests for code currently using ::mempool, we would have the option of moving it to the Node struct and still accessing it conveniently.
src/interfaces/chain.cpp
Outdated
{ | ||
ChainClients& clients = m_node.chain_clients; | ||
ChainClients::iterator it = clients.emplace(clients.end(), client); | ||
return MakeHandler([&clients, it] { clients.erase(it); }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it better to capture member
chain_clients
thanthis
?
I think a good starting point is to just capture what's needed without unused data or object references. But in this case, capturing this
instead of &clients
is worse because in addition to capturing unneeded data, it requires the Handler
object to be destroyed before the Chain
object, and there's no reason to create that kind of error-prone dependency.
Also, it would be nice to explain that a
std::list
is used otherwise this.erase
would fail.
Sure, added comment.
7225ebe
to
a3dc533
Compare
Concept ACK. This looks like a step in the right direction in terms of cutting down on globals and moving towards being able to test multi-node behavior without having to rely on the functional tests. Is your intent to split this into separate commits at some point for detailed review? |
Could go either way. These are basically boring text replacements that should be about as much effort to review split up or joined, but I'm happy to split it up. I think I could split it up into 4 or 5 commits roughly matching the bullet points in the description. Overall diff would be a little bigger because some lines ( |
I'm fine reviewing either, though I guess if it were easy to do some scripted-diffs that would be nice. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great concept ACK! See comments on minor points, also I think maybe the Handler/Cleanup handler logic could be simplified in future work, didn't spend time on the how yet..
src/interfaces/node.h
Outdated
@@ -254,6 +255,10 @@ class Node | |||
using NotifyHeaderTipFn = | |||
std::function<void(bool initial_download, int height, int64_t block_time, double verification_progress)>; | |||
virtual std::unique_ptr<Handler> handleNotifyHeaderTip(NotifyHeaderTipFn fn) = 0; | |||
|
|||
//! Return pointer to internal chain interface, useful for testing. | |||
//! Only works if node interface is being called in same process as implementation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find comment fuzzy, do you mean only works if node interface is set by the process of the implementation ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #16839 (comment)
I find comment fuzzy, do you mean only works if node interface is set by the process of the implementation ?
Sorry, this was pretty confusing. Better not to refer to processes in comments before #10102, so I removed this. But to answer the question: this always returns non-null currently and after #10102 will return null if the node is a remote process.
src/node/node.h
Outdated
//! Node struct containing chain state and connection state. | ||
//! | ||
//! More state could be moved into this struct to eliminate global variables, | ||
//! and allow creating multiple instances of validation and networking objects |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So you would create multiple Node in one test process, connect them and let them talk maybe with random introduction of interferences in-between ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #16839 (comment)
So you would create multiple Node in one test process, connect them and let them talk maybe with random introduction of interferences in-between ?
With caveats, yes, since code that isn't referring to global variables can be instantiated multiple times for testing in a single process.
@@ -46,4 +47,9 @@ UniValue blockheaderToJSON(const CBlockIndex* tip, const CBlockIndex* blockindex | |||
/** Used by getblockstats to get feerates at different percentiles by weight */ | |||
void CalculatePercentilesByWeight(CAmount result[NUM_GETBLOCKSTATS_PERCENTILES], std::vector<std::pair<CAmount, int64_t>>& scores, int64_t total_weight); | |||
|
|||
//! Pointer to node state that needs to be declared as a global to be accessible | |||
//! RPC methods. Due to limitations of the RPC framework, there's currently no |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How to override RPC framework limitations ? Encapsulating RPC functions in its own structure like it has been done for the P2P stack with CConnman ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #16839 (comment)
How to override RPC framework limitations ? Encapsulating RPC functions in its own structure like it has been done for the P2P stack with CConnman ?
Yes, that's one way. Another option would be to add a std::any
member to the JSONRPCRequest
struct, so arbitrary context could be passed into RPC methods in a type-safe way.
-BEGIN VERIFY SCRIPT- s() { git grep -l "$1" src | xargs sed -i "s/$1/$2/g"; } s 'struct InitInterfaces' 'struct NodeContext' s 'InitInterfaces interfaces' 'NodeContext node' s 'InitInterfaces& interfaces' 'NodeContext\& node' s 'InitInterfaces m_interfaces' 'NodeContext m_context' s 'InitInterfaces\* g_rpc_interfaces' 'NodeContext* g_rpc_node' s 'g_rpc_interfaces = &interfaces' 'g_rpc_node = \&node' s 'g_rpc_interfaces' 'g_rpc_node' s 'm_interfaces' 'm_context' s 'interfaces\.chain' 'node.chain' s '\(AppInitMain\|Shutdown\|Construct\)(interfaces)' '\1(node)' s 'init interfaces' 'chain clients' -END VERIFY SCRIPT-
So g_connman and g_banman globals can be removed next commit.
-BEGIN VERIFY SCRIPT- sed -i 's:#include <interfaces/chain.h>:#include <banman.h>\n#include <interfaces/chain.h>\n#include <net.h>\n#include <net_processing.h>:' src/node/context.cpp sed -i 's/namespace interfaces {/class BanMan;\nclass CConnman;\nclass PeerLogicValidation;\n&/' src/node/context.h sed -i 's/std::unique_ptr<interfaces::Chain> chain/std::unique_ptr<CConnman> connman;\n std::unique_ptr<PeerLogicValidation> peer_logic;\n std::unique_ptr<BanMan> banman;\n &/' src/node/context.h sed -i '/std::unique_ptr<[^>]\+> \(g_connman\|g_banman\|peerLogic\);/d' src/banman.h src/net.h src/init.cpp sed -i 's/g_connman/m_context.connman/g' src/interfaces/node.cpp sed -i 's/g_banman/m_context.banman/g' src/interfaces/node.cpp sed -i 's/g_connman/m_node.connman/g' src/interfaces/chain.cpp src/test/setup_common.cpp sed -i 's/g_banman/m_node.banman/g' src/test/setup_common.cpp sed -i 's/g_connman/node.connman/g' src/init.cpp src/node/transaction.cpp sed -i 's/g_banman/node.banman/g' src/init.cpp sed -i 's/peerLogic/node.peer_logic/g' src/init.cpp sed -i 's/g_connman/g_rpc_node->connman/g' src/rpc/mining.cpp src/rpc/net.cpp src/rpc/rawtransaction.cpp sed -i 's/g_banman/g_rpc_node->banman/g' src/rpc/net.cpp sed -i 's/std::shared_ptr<CWallet> wallet =/node.context()->connman = std::move(test.m_node.connman);\n &/' src/qt/test/wallettests.cpp -END VERIFY SCRIPT-
Wallet code should use interfaces::Chain and not directly access to node state. Add a g_rpc_chain replacement global for wallet code to use, and move g_rpc_node definition to a libbitcoin_server source file so there are link errors if wallet code tries to access it.
Looks ready for merge after rebase |
Rebased 34a7a50 -> 362ded4 (pr/noglob.15 -> pr/noglob.16) due to conflict with #16202 Also renamed |
ACK 362ded4 |
362ded4 Avoid using g_rpc_node global in wallet code (Russell Yanofsky) 8922d7f scripted-diff: Remove g_connman, g_banman globals (Russell Yanofsky) e6f4f89 Pass NodeContext, ConnMan, BanMan references more places (Russell Yanofsky) 4d5448c MOVEONLY: Move NodeContext struct to node/context.h (Russell Yanofsky) 301bd41 scripted-diff: Rename InitInterfaces to NodeContext (Russell Yanofsky) Pull request description: This change is mainly a naming / organization change intended to simplify #10102. It: - Renames struct InitInterfaces to struct NodeContext and moves it from src/init.h to src/node/context.h. This is a cosmetic change intended to make the point of the struct more obvious. - Gets rid of BanMan and ConnMan globals making them NodeContext members instead. Getting rid of these globals has been talked about in past as a way to implement testing and simulations. Making them NodeContext members is a way of keeping them accessible without the globals. - Splits g_rpc_interfaces global into g_rpc_node and g_rpc_chain globals. This better separates node and wallet rpc methods. Node RPC methods should have access NodeContext, while wallet RPC methods should only have indirect access to node functionality via interfaces::Chain. - Adds NodeContext& references to interfaces::Chain class and the interfaces::MakeChain() function. This is needed to access ConnMan and BanMan instances without the globals. - Gets rid of redundant Node and Chain instances in Qt tests. This is needed due to the previous MakeChain change, and also makes test setup a little more straightforward. More cleanup could be done in the future, but it will require deduplication of bitcoind, bitcoin-qt, and TestingSetup init code. ACKs for top commit: laanwj: ACK 362ded4 Tree-SHA512: 9ae6ff1e33423291d1e52056bac95e0874538390892a6e83c4c115b3c73155a8827c0191b46eb3d14e3b3f6c23ccb08095490880fbc3188026319c71739f7db2
Post-humous ACK, diff from 50e0bb2 is renaming
Agree may have avoid us confusion there. |
Summary: -BEGIN VERIFY SCRIPT- s() { git grep -l "$1" src | xargs sed -i "s/$1/$2/g"; } s 'struct InitInterfaces' 'struct NodeContext' s 'InitInterfaces interfaces' 'NodeContext node' s 'InitInterfaces& interfaces' 'NodeContext\& node' s 'InitInterfaces m_interfaces' 'NodeContext m_context' s 'InitInterfaces\* g_rpc_interfaces' 'NodeContext* g_rpc_node' s 'g_rpc_interfaces = &interfaces' 'g_rpc_node = \&node' s 'g_rpc_interfaces' 'g_rpc_node' s 'm_interfaces' 'm_context' s 'interfaces\.chain' 'node.chain' s '\(AppInitMain\|Shutdown\|Construct\)(interfaces)' '\1(node)' s 'init interfaces' 'chain clients' -END VERIFY SCRIPT- This is a parial backport of Core [[bitcoin/bitcoin#16839 | PR16839]] : bitcoin/bitcoin@301bd41 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6108
Summary: This is a partial backport of Core [[bitcoin/bitcoin#16839 | PR16839]] : bitcoin/bitcoin@4d5448c Depends on D6108 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6109
…aces Summary: So g_connman and g_banman globals can be removed next commit. bitcoin/bitcoin@e6f4f89 --- Partial backport of Core [[bitcoin/bitcoin#16839 | PR16839]] Test Plan: ninja check-all ninja Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6409
Summary: ```-BEGIN VERIFY SCRIPT- sed -i 's:#include <interfaces/chain.h>:#include <banman.h>\n#include<interfaces/chain.h>\n#include <net.h>\n#include <net_processing.h>:' src/node/context.cpp sed -i 's/namespace interfaces {/class BanMan;\nclass CConnman;\nclass PeerLogicValidation;\n&/' src/node/context.h sed -i 's/std::unique_ptr<interfaces::Chain>chain/std::unique_ptr<CConnman> connman;\n std::unique_ptr<PeerLogicValidation> peer_logic;\nstd::unique_ptr<BanMan> banman;\n &/' src/node/context.h sed -i '/std::unique_ptr<[^>]\+> \(g_connman\|g_banman\|peerLogic\);/d'src/banman.h src/net.h src/init.cpp sed -i 's/g_connman/m_context.connman/g' src/interfaces/node.cpp sed -i 's/g_banman/m_context.banman/g' src/interfaces/node.cpp sed -i 's/g_connman/m_node.connman/g' src/interfaces/chain.cpp src/test/util/setup_common.cpp sed -i 's/g_banman/m_node.banman/g' src/test/util/setup_common.cpp sed -i 's/g_connman/node.connman/g' src/init.cpp src/node/transaction.cpp sed -i 's/g_banman/node.banman/g' src/init.cpp sed -i 's/peerLogic/node.peer_logic/g' src/init.cpp sed -i 's/g_connman/g_rpc_node->connman/g' src/rpc/mining.cpp src/rpc/net.cpp src/rpc/rawtransaction.cpp sed -i 's/g_banman/g_rpc_node->banman/g' src/rpc/net.cpp sed -i 's/std::shared_ptr<CWallet> wallet =/node.context()->connman = std::move(test.m_node.connman);\n &/' src/qt/test/wallettests.cpp -END VERIFY SCRIPT-``` ALSO the changes to src/test/util/setup_common.cpp and src/rpc/blockchain.h from the following diff have been anticipated into this one because boost test failures. bitcoin/bitcoin@8922d7f --- Partial backport of Core [[bitcoin/bitcoin#16839 | PR16839]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6418
Summary: Wallet code should use interfaces::Chain and not directly access to node state. Add a g_rpc_chain replacement global for wallet code to use, and move g_rpc_node definition to a libbitcoin_server source file so there are link errors if wallet code tries to access it. bitcoin/bitcoin@362ded4 --- Depends on D6418 Concludes backport of Core [[bitcoin/bitcoin#16839 | PR16839]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Subscribers: deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6419
Summary: -BEGIN VERIFY SCRIPT- s() { git grep -l "$1" src | xargs sed -i "s/$1/$2/g"; } s 'struct InitInterfaces' 'struct NodeContext' s 'InitInterfaces interfaces' 'NodeContext node' s 'InitInterfaces& interfaces' 'NodeContext\& node' s 'InitInterfaces m_interfaces' 'NodeContext m_context' s 'InitInterfaces\* g_rpc_interfaces' 'NodeContext* g_rpc_node' s 'g_rpc_interfaces = &interfaces' 'g_rpc_node = \&node' s 'g_rpc_interfaces' 'g_rpc_node' s 'm_interfaces' 'm_context' s 'interfaces\.chain' 'node.chain' s '\(AppInitMain\|Shutdown\|Construct\)(interfaces)' '\1(node)' s 'init interfaces' 'chain clients' -END VERIFY SCRIPT- This is a parial backport of Core [[bitcoin/bitcoin#16839 | PR16839]] : bitcoin/bitcoin@301bd41 Test Plan: ninja all check-all Differential Revision: https://reviews.bitcoinabc.org/D6108
Summary: This is a partial backport of Core [[bitcoin/bitcoin#16839 | PR16839]] : bitcoin/bitcoin@4d5448c Depends on D6108 Test Plan: - `ninja all check-all` - do an equivalent autoconf build with tests (`make check ; test/functional/test_runner.py`) Differential Revision: https://reviews.bitcoinabc.org/D6109
…ext local -BEGIN VERIFY SCRIPT- s() { git grep -l "$1" src | xargs sed -i "s/$1/$2/g"; } s 'struct InitInterfaces' 'struct NodeContext' s 'InitInterfaces interfaces' 'NodeContext node' s 'InitInterfaces& interfaces' 'NodeContext\& node' s 'InitInterfaces m_interfaces' 'NodeContext m_context' s 'InitInterfaces\* g_rpc_interfaces' 'NodeContext* g_rpc_node' s 'g_rpc_interfaces = &interfaces' 'g_rpc_node = \&node' s 'g_rpc_interfaces' 'g_rpc_node' s 'm_interfaces' 'm_context' s 'interfaces\.chain' 'node.chain' s '\(AppInitMain\|Shutdown\|Construct\)(interfaces)' '\1(node)' s 'init interfaces' 'chain clients' -END VERIFY SCRIPT-
This change is mainly a naming / organization change intended to simplify #10102. It:
Renames struct InitInterfaces to struct NodeContext and moves it from
src/init.h to src/node/context.h. This is a cosmetic change intended to make
the point of the struct more obvious.
Gets rid of BanMan and ConnMan globals making them NodeContext members
instead. Getting rid of these globals has been talked about in past as a way
to implement testing and simulations. Making them NodeContext members is a
way of keeping them accessible without the globals.
Splits g_rpc_interfaces global into g_rpc_node and g_rpc_chain globals. This
better separates node and wallet rpc methods. Node RPC methods should have
access NodeContext, while wallet RPC methods should only have indirect access
to node functionality via interfaces::Chain.
Adds NodeContext& references to interfaces::Chain class and the
interfaces::MakeChain() function. This is needed to access ConnMan and BanMan
instances without the globals.
Gets rid of redundant Node and Chain instances in Qt tests. This is
needed due to the previous MakeChain change, and also makes test setup a
little more straightforward. More cleanup could be done in the future, but it
will require deduplication of bitcoind, bitcoin-qt, and TestingSetup init
code.