New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
p2p: No delay in adding fixed seeds if -dnsseed=0 and peers.dat is empty #19884
Conversation
efdbbff
to
bcbd637
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@practicalswift I'm sorry this took so long. Ready for review. |
@n-thumann Thank you, for the prompt review. |
bcbd637
to
ce0686c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK. Maybe add a regression test in feature_config_args.py
to cover these cases.
@dhruv I saw you removed the |
ce0686c
to
34c215c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jonatack Added a regression test in feature_config_args.py to cover the cases. It's a slow test though because we have to wait 60 seconds. Is there anything I can do to reduce that interval for tests? I thought about making it configurable via cli flags but I will look to your guidance on whether that is standard practice.
@practicalswift Thank you for catching that. I overlooked the implication of the It seems that the |
34c215c
to
e7de5b5
Compare
Appveyor has passed on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK e7de5b5
Thanks for adding the tests. No strong opinion whether it's better to drop the slow first test or keep it. There is a setmocktime
RPC often used in the functional tests, and a -mocktime=<n>
config option that could be passed an extra_arg
to start_node
, but AFAICT neither one will have any effect and I'm not sure it's worth changing the code to be more testable unless you have an elegant way to do it or it could be generalized and useful elsewhere.
A couple of nit comments follow; you can ignore them unless you need to retouch for other reasons.
Is this a use case worth optimising for? The linked issue doesn't really provide a reason why this use case is worthwhile, as far as I can see, just that the filer thought it was surprising. The only case I can see where you'd want to set |
Fair question, @ajtowns. Let's page @practicalswift and see if he can shed light on his original use case. |
There are multiple important use cases for
I hope you're not advocating the removal of
I'm not sure I would call removal of a 60 second sleep an optimization. To me it is clearly a bug fix :) Also, please note that the behaviour is unchanged if The only effect of this change is that users of Given the above clarification: do your skepticism to this bug fix remain? :) |
FWIW, if you use |
Yes I know, but then you'll have to trust the exit node operator not to tamper with the DNS responses :)
|
Of course, and I'm not commenting on the validity of the use case in general. Just that "You don't want to use clearnet at all" on its own is not a reason to use -dnsseed=0. |
Fair point. I should have written "You don't want to use clearnet at all and you don't trust exit node operators" :) |
By "DNS seeding" you're in this context referring to the "establish AFAICT we've never supported sending actual DNS queries to the DNS seed nodes via proxy. From the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested ACK e7de5b5
This fixes the bug which makes -dnsseed=0
users have to wait out a 60 second sleep intended only for -dnsseed=1
users :)
@practicalswift Yes, when -proxy is enabled, no actual DNS seeding is done, but ADDR_FETCH connections are established to the seeder's names instead. That was the reason for introducing ADDR_FETCH connections (formerly called oneshot connections) initially. |
Ah, and are thus only connecting to the ~89 fixed seeds with onion addresses, and not going via exit nodes at all? That makes more sense. I think that currently, if you do Perhaps rather than a time check, it would be better to check for something like "if addrman is empty, there are no ADDR_FETCH nodes left to try, no open connections and no added nodes to keep trying" ? OTOH, if it is a change for the better, it might make sense to move the add fixed nodes code to its own function, and call it from ThreadDNSAddressSeed directly (after trying all the dns seeds, with addrman still empty, and only after the opencon thread as processed any ADDR_FETCH nodes that got queued up), and from CConnman::Start when (edit: ADDR_FETCH not oneshot) |
c68eef3
to
aaf9cae
Compare
aaf9cae
to
eb1eef4
Compare
Rebased |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As suggested in the description, I tested:
rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0
without the patch (note the timestamps):
2021-01-02T00:00:49Z msghand thread start
2021-01-02T00:01:50Z Adding fixed seed nodes as DNS doesn't seem to be available.
With the patch:
2021-01-02T00:05:38Z msghand thread start
2021-01-02T00:05:38Z Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted
Ran clang-format-diff.py
(it found nothing). Nice PR! Suggestions are minor and optional.
ACK eb1eef4
(but take with a grain of salt; I don't know this area of the code well.)
@@ -145,11 +146,61 @@ def test_networkactive(self): | |||
self.start_node(0, extra_args=['-nonetworkactive=1']) | |||
self.stop_node(0) | |||
|
|||
def test_seed_peers(self): | |||
self.log.info('Test seed peers') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.log.info('Test seed peers') | |
self.log.info('Test seed peers, this will take about 2 minutes') |
with self.nodes[0].assert_debug_log(expected_msgs=["Loaded 0 addresses from peers.dat", | ||
"DNS seeding disabled", | ||
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can also check for the absence of debug.log messages (I'm not sure if it's worth it in this case):
with self.nodes[0].assert_debug_log(expected_msgs=["Loaded 0 addresses from peers.dat", | |
"DNS seeding disabled", | |
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]): | |
with self.nodes[0].assert_debug_log(expected_msgs=[ | |
"Loaded 0 addresses from peers.dat", | |
"DNS seeding disabled", | |
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n" | |
], unexpected_msgs=["60 seconds"]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed the indentation. I did not add the unexpected_msgs=["60 seconds"]
as it is implicit in the 2 second timeout for assert_debug_log
.
eb1eef4
to
500315d
Compare
Thanks for the review @LarryRuane, and my apologies for the long wait. Comments addressed. Rebased. Ready for further review. |
500315d
to
3fdea1f
Compare
Fixed failing linter. |
re-reviewed, re-tested |
Code review ACK 3fdea1f |
"DNS seeding disabled", | ||
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]): | ||
self.start_node(0, extra_args=['-dnsseed=0']) | ||
assert time.time() - start < 5 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's slightly risky to have a test that depends on real time instead of mocked times. CI environments can be extremely slow, so this bound might introduce random failures.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right - that could make for a flaky test. I picked 5 seconds as an arbitrary bound. What I really wanted to check was that the node did not wait 60 seconds to use fixed seeds.
assert time.time() - start < 5 | |
assert time.time() - start < 60 |
accomplished that. And if CI is so slow that 60 seconds pass, the test will fail anyway because the logs will say "Adding fixed seeds as 60 seconds have passed and addrman is empty" instead of "Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, good idea!
In the longer term I think we should change this to use mock time but for this PR that would be too much of a hassle.
…mpty. Add -fixedseeds arg.
3fdea1f
to
fe3e993
Compare
re-ACK fe3e993 |
Updated tests for CI stability. Ready for further review. |
re-ACK fe3e993 |
…0 and peers.dat is empty fe3e993 [p2p] No delay in adding fixed seeds if -dnsseed=0 and peers.dat is empty. Add -fixedseeds arg. (Dhruv Mehta) Pull request description: Closes bitcoin#19795 Before PR: If `peers.dat` is empty and `-dnsseed=0`, bitcoind will fallback on to fixed seeds but only after a 60 seconds delay. After PR: There's no 60 second delay. To reproduce: `rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0` without and with patch code Other changes in the PR: - `-fixedseeds` command line argument added: `-dnsseed=0 -fixedseeds=0 -addnode=X` provides a trusted peer only setup. `-dnsseed=0 -fixedseeds=0` allows for a `addnode` RPC to add a trusted peer without falling back to hardcoded seeds. ACKs for top commit: LarryRuane: re-ACK fe3e993 laanwj: re-ACK fe3e993 Tree-SHA512: 79449bf4e83a315be6dbac9bdd226de89d2a3f7f76d9c5640a2cb3572866e6b0e8ed67e65674c9824054cf13119dc01c7e1a33848daac6b6c34dbc158b6dba8f
with self.nodes[0].assert_debug_log(expected_msgs=[ | ||
"Loaded 0 addresses from peers.dat", | ||
"DNS seeding disabled", | ||
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and and
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see #21165
"0 addresses found from DNS seeds", | ||
"Adding fixed seeds as 60 seconds have passed and addrman is empty"], timeout=80): | ||
self.start_node(0, extra_args=['-dnsseed=1']) | ||
assert time.time() - start >= 60 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wouldn't it work to use setmocktime
inside the context manager above and check here: node.uptime() >= 60
to speed up the test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I looked at this last September (#19884 (review)) and it didn't look feasible, interesting if it is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$ time test/functional/feature_config_args.py
2021-02-12T17:02:39.647000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_kuphxowf
2021-02-12T17:02:45.294000Z TestFramework (INFO): Test config args logging
2021-02-12T17:02:45.605000Z TestFramework (INFO): Test seed peers, this will take about 2 minutes
2021-02-12T17:02:48.587000Z TestFramework (INFO): Test -networkactive option
2021-02-12T17:02:54.023000Z TestFramework (INFO): Stopping nodes
2021-02-12T17:02:54.127000Z TestFramework (INFO): Cleaning up /tmp/bitcoin_func_test_kuphxowf on exit
2021-02-12T17:02:54.127000Z TestFramework (INFO): Tests successful
real 0m14.682s
user 0m3.842s
sys 0m0.745s
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see #21165
fa730e9 test: Avoid connecting to real network when running tests (MarcoFalke) fa1b713 test: Assume node is running in subtests (MarcoFalke) Pull request description: Introduced in #19884 ACKs for top commit: Sjors: ACK fa730e9 Tree-SHA512: fe132a9ffe2fae1ab16857a3dec9839526fdf74d27a1ae794fbffca8356f639c4b916dc888b260281e9cc793916706c18d1687ebb5a076d4e1c481d218d308d3
…ing tests fa730e9 test: Avoid connecting to real network when running tests (MarcoFalke) fa1b713 test: Assume node is running in subtests (MarcoFalke) Pull request description: Introduced in bitcoin#19884 ACKs for top commit: Sjors: ACK fa730e9 Tree-SHA512: fe132a9ffe2fae1ab16857a3dec9839526fdf74d27a1ae794fbffca8356f639c4b916dc888b260281e9cc793916706c18d1687ebb5a076d4e1c481d218d308d3
…mpty. Add -fixedseeds arg. Github-Pull: bitcoin#19884 Rebased-From: fe3e993
…mpty. Add -fixedseeds arg. Summary: ``` Before PR: If peers.dat is empty and -dnsseed=0, bitcoind will fallback on to fixed seeds but only after a 60 seconds delay. After PR: There's no 60 second delay. To reproduce: rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0 without and with patch code Other changes in the PR: -fixedseeds command line argument added: -dnsseed=0 -fixedseeds=0 -addnode=X provides a trusted peer only setup. -dnsseed=0 -fixedseeds=0 allows for a addnode RPC to add a trusted peer without falling back to hardcoded seeds. ``` Note to reviewers 1: the p2p_dos_header_tree.py test change is cherry picked from bitcoin/bitcoin@fa730e9. Without this the test will connect to the real testnet and eventually sync the headers with other nodes, that would cause the test to fail. This will be undone when the PR is backported, which requires D10909 as a dependency. Note to reviewers 2: the test feature_config_args.py now takes forever to run, this will be fixed in a follow-up. Backport of [[bitcoin/bitcoin#19884 | core#19884]]. Depends on D10907. Ref T1696. Test Plan: ./test/functional/test_runner.py feature_config_args.py Reviewers: #bitcoin_abc, PiRK Reviewed By: #bitcoin_abc, PiRK Subscribers: PiRK Maniphest Tasks: T1696 Differential Revision: https://reviews.bitcoinabc.org/D10908
Closes #19795
Before PR: If
peers.dat
is empty and-dnsseed=0
, bitcoind will fallback on to fixed seeds but only after a 60 seconds delay.After PR: There's no 60 second delay.
To reproduce:
rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0
without and with patch codeOther changes in the PR:
-fixedseeds
command line argument added:-dnsseed=0 -fixedseeds=0 -addnode=X
provides a trusted peer only setup.-dnsseed=0 -fixedseeds=0
allows for aaddnode
RPC to add a trusted peer without falling back to hardcoded seeds.