Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

p2p: No delay in adding fixed seeds if -dnsseed=0 and peers.dat is empty #19884

Merged
merged 1 commit into from Feb 12, 2021

Conversation

dhruv
Copy link
Contributor

@dhruv dhruv commented Sep 5, 2020

Closes #19795

Before PR: If peers.dat is empty and -dnsseed=0, bitcoind will fallback on to fixed seeds but only after a 60 seconds delay.
After PR: There's no 60 second delay.

To reproduce:
rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0 without and with patch code

Other changes in the PR:

  • -fixedseeds command line argument added: -dnsseed=0 -fixedseeds=0 -addnode=X provides a trusted peer only setup. -dnsseed=0 -fixedseeds=0 allows for a addnode RPC to add a trusted peer without falling back to hardcoded seeds.

@dhruv dhruv changed the title No delay in adding fixed seeds if -dnsseed=0 and peers.dat is empty p2p: No delay in adding fixed seeds if -dnsseed=0 and peers.dat is empty Sep 5, 2020
Copy link
Contributor

@n-thumann n-thumann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK bcbd637 on master at 3ba25e3, was able to reproduce the issue and confirm that it´s fixed with this PR ✌️

@dhruv
Copy link
Contributor Author

dhruv commented Sep 5, 2020

@practicalswift I'm sorry this took so long. Ready for review.

@dhruv
Copy link
Contributor Author

dhruv commented Sep 5, 2020

@n-thumann Thank you, for the prompt review.

@DrahtBot DrahtBot added the P2P label Sep 5, 2020
src/net.cpp Outdated Show resolved Hide resolved
src/net.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. Maybe add a regression test in feature_config_args.py to cover these cases.

src/net.cpp Outdated Show resolved Hide resolved
src/net.cpp Outdated Show resolved Hide resolved
@practicalswift
Copy link
Contributor

practicalswift commented Sep 5, 2020

@dhruv I saw you removed the done logic: are you sure that !addrman.empty() is guaranteed after addrman.Add(convertSeed6(Params().FixedSeeds()), local);? AFAICT that condition must be hold for the new code to be equivalent to the old code.

Copy link
Contributor Author

@dhruv dhruv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonatack Added a regression test in feature_config_args.py to cover the cases. It's a slow test though because we have to wait 60 seconds. Is there anything I can do to reduce that interval for tests? I thought about making it configurable via cli flags but I will look to your guidance on whether that is standard practice.

src/net.cpp Outdated Show resolved Hide resolved
src/net.cpp Outdated Show resolved Hide resolved
@dhruv
Copy link
Contributor Author

dhruv commented Sep 6, 2020

@dhruv I saw you removed the done logic: are you sure that !addrman.empty() is guaranteed after addrman.Add(convertSeed6(Params().FixedSeeds()), local);? AFAICT that condition must be hold for the new code to be equivalent to the old code.

@practicalswift Thank you for catching that. I overlooked the implication of the static qualifier on done and mistakenly thought the code was redundant. It is now added back in.

It seems that the done logic is used to make sure that addrman.Add(convertSeed6(Params().FixedSeeds()), local); gets called no more than once regardless of the outcome. When I followed the code in CAddrMan, it seems that the invocation just adds the fixed seeds to the address tables so long as the seeds are well-formed. So invoking the deterministic function more than once is unnecessary. I'm new around here, so I'm hoping you can confirm my understanding.

@dhruv
Copy link
Contributor Author

dhruv commented Sep 7, 2020

Appveyor has passed on e7de5b5, Github is just not reflecting it.

Copy link
Contributor

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK e7de5b5

Thanks for adding the tests. No strong opinion whether it's better to drop the slow first test or keep it. There is a setmocktime RPC often used in the functional tests, and a -mocktime=<n> config option that could be passed an extra_arg to start_node, but AFAICT neither one will have any effect and I'm not sure it's worth changing the code to be more testable unless you have an elegant way to do it or it could be generalized and useful elsewhere.

A couple of nit comments follow; you can ignore them unless you need to retouch for other reasons.

src/net.cpp Outdated Show resolved Hide resolved
test/functional/feature_config_args.py Outdated Show resolved Hide resolved
@ajtowns
Copy link
Contributor

ajtowns commented Sep 10, 2020

If peers.dat is empty and -dnsseed=0

Is this a use case worth optimising for? The linked issue doesn't really provide a reason why this use case is worthwhile, as far as I can see, just that the filer thought it was surprising.

The only case I can see where you'd want to set -dnsseed=0 is if you somehow don't have DNS despite being connected to the real internet, or if you're really worried about privacy. But if you're really worried about privacy, connecting to fixed seeds is probably worse than doing a dns query, and if you were doing something clever, like triggering an outbound/inbound connection to your own node, losing the 60 second delay might also drop privacy.

@dhruv
Copy link
Contributor Author

dhruv commented Sep 11, 2020

Is this a use case worth optimising for? The linked issue doesn't really provide a reason why this use case is worthwhile, as far as I can see, just that the filer thought it was surprising.

Fair question, @ajtowns. Let's page @practicalswift and see if he can shed light on his original use case.

@practicalswift
Copy link
Contributor

practicalswift commented Sep 11, 2020

The only case I can see where you'd want to set -dnsseed=0 is if you somehow don't have DNS despite being connected to the real internet, or if you're really worried about privacy. But if you're really worried about privacy, connecting to fixed seeds is probably worse than doing a dns query, and if you were doing something clever, like triggering an outbound/inbound connection to your own node, losing the 60 second delay might also drop privacy.

There are multiple important use cases for -dnsseed=0:

  • You don't want to use clearnet at all (-dnsseed=0 -onlynet=onion -listen=0)
  • You don't want to broadcast to the DNS seed operators (or anyone listening on the path to those DNS seed operators) that you're starting a new node (-dnsseed=0 -listen=0)
  • You want to test how Bitcoin Core behaves if the DNS seeds are unavailable

I hope you're not advocating the removal of -dnsseed=0 :)

Is this a use case worth optimising for? The linked issue doesn't really provide a reason why this use case is worthwhile, as far as I can see, just that the filer thought it was surprising.

I'm not sure I would call removal of a 60 second sleep an optimization. To me it is clearly a bug fix :)

Also, please note that the behaviour is unchanged if -dnsseed=0 is not set.

The only effect of this change is that users of -dnsseed=0 won't have to wait an extra 60 seconds for their node to start.

Given the above clarification: do your skepticism to this bug fix remain? :)

@sipa
Copy link
Member

sipa commented Sep 11, 2020

You don't want to use clearnet at all (-dnsseed=0 -onlynet=onion -listen=0)

FWIW, if you use -proxy, DNS seeding happens over the proxy (which in the case of Tor means the exit node will do the resolving for you).

@practicalswift
Copy link
Contributor

practicalswift commented Sep 11, 2020

You don't want to use clearnet at all (-dnsseed=0 -onlynet=onion -listen=0)

FWIW, if you use -proxy, DNS seeding happens over the proxy (which in the case of Tor means the exit node will do the resolving for you).

Yes I know, but then you'll have to trust the exit node operator not to tamper with the DNS responses :)

-dnsseed=0 -onlynet=onion -listen=0 avoids that.

@sipa
Copy link
Member

sipa commented Sep 11, 2020

Of course, and I'm not commenting on the validity of the use case in general. Just that "You don't want to use clearnet at all" on its own is not a reason to use -dnsseed=0.

@practicalswift
Copy link
Contributor

Of course, and I'm not commenting on the validity of the use case in general. Just that "You don't want to use clearnet at all" on its own is not a reason to use -dnsseed=0.

Fair point. I should have written "You don't want to use clearnet at all and you don't trust exit node operators" :)

@practicalswift
Copy link
Contributor

practicalswift commented Sep 11, 2020

FWIW, if you use -proxy, DNS seeding happens over the proxy (which in the case of Tor means the exit node will do the resolving for you).

By "DNS seeding" you're in this context referring to the "establish ADDR_FETCH connections via proxy to seed nodes"-logic, right?

AFAICT we've never supported sending actual DNS queries to the DNS seed nodes via proxy.

From the SetNameProxy documentation: "SOCKS5's support for UDP-over-SOCKS5 has been considered, but no SOCK5 server in common use (most notably Tor) actually implements UDP support, and a DNS resolver is beyond the scope of this project."

Copy link
Contributor

@practicalswift practicalswift left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested ACK e7de5b5

This fixes the bug which makes -dnsseed=0 users have to wait out a 60 second sleep intended only for -dnsseed=1 users :)

@sipa
Copy link
Member

sipa commented Sep 11, 2020

@practicalswift Yes, when -proxy is enabled, no actual DNS seeding is done, but ADDR_FETCH connections are established to the seeder's names instead. That was the reason for introducing ADDR_FETCH connections (formerly called oneshot connections) initially.

@ajtowns
Copy link
Contributor

ajtowns commented Sep 12, 2020

Fair point. I should have written "You don't want to use clearnet at all and you don't trust exit node operators" :)

Ah, and are thus only connecting to the ~89 fixed seeds with onion addresses, and not going via exit nodes at all? That makes more sense.

I think that currently, if you do -onlynet=onion -listen=0 -dnsseed=0 -addnode=mytrustedpeer.onion, and you successfully connect to mytrustedpeer.onion within 60 seconds, you won't add the fixed seeds (your addrman will contain mypeer.onion rather than being empty), whereas with this change you will add the fixed seeds. Not sure that's a change for the better?

Perhaps rather than a time check, it would be better to check for something like "if addrman is empty, there are no ADDR_FETCH nodes left to try, no open connections and no added nodes to keep trying" ?

OTOH, if it is a change for the better, it might make sense to move the add fixed nodes code to its own function, and call it from ThreadDNSAddressSeed directly (after trying all the dns seeds, with addrman still empty, and only after the opencon thread as processed any ADDR_FETCH nodes that got queued up), and from CConnman::Start when -dnsseed=0 and addrman is empty. That way you avoid unnecessary waiting in all cases, I think.

(edit: ADDR_FETCH not oneshot)

@dhruv
Copy link
Contributor Author

dhruv commented Dec 11, 2020

Rebased

Copy link
Contributor

@LarryRuane LarryRuane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As suggested in the description, I tested:

rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0

without the patch (note the timestamps):

2021-01-02T00:00:49Z msghand thread start
2021-01-02T00:01:50Z Adding fixed seed nodes as DNS doesn't seem to be available.

With the patch:

2021-01-02T00:05:38Z msghand thread start
2021-01-02T00:05:38Z Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted

Ran clang-format-diff.py (it found nothing). Nice PR! Suggestions are minor and optional.
ACK eb1eef4
(but take with a grain of salt; I don't know this area of the code well.)

src/init.cpp Outdated Show resolved Hide resolved
src/init.cpp Outdated Show resolved Hide resolved
src/net.cpp Outdated Show resolved Hide resolved
test/functional/feature_config_args.py Outdated Show resolved Hide resolved
@@ -145,11 +146,61 @@ def test_networkactive(self):
self.start_node(0, extra_args=['-nonetworkactive=1'])
self.stop_node(0)

def test_seed_peers(self):
self.log.info('Test seed peers')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.log.info('Test seed peers')
self.log.info('Test seed peers, this will take about 2 minutes')

Comment on lines 169 to 175
with self.nodes[0].assert_debug_log(expected_msgs=["Loaded 0 addresses from peers.dat",
"DNS seeding disabled",
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also check for the absence of debug.log messages (I'm not sure if it's worth it in this case):

Suggested change
with self.nodes[0].assert_debug_log(expected_msgs=["Loaded 0 addresses from peers.dat",
"DNS seeding disabled",
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]):
with self.nodes[0].assert_debug_log(expected_msgs=[
"Loaded 0 addresses from peers.dat",
"DNS seeding disabled",
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"
], unexpected_msgs=["60 seconds"]):

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the indentation. I did not add the unexpected_msgs=["60 seconds"] as it is implicit in the 2 second timeout for assert_debug_log.

@dhruv
Copy link
Contributor Author

dhruv commented Feb 6, 2021

Thanks for the review @LarryRuane, and my apologies for the long wait.

Comments addressed. Rebased. Ready for further review.

@dhruv
Copy link
Contributor Author

dhruv commented Feb 6, 2021

Fixed failing linter.

@LarryRuane
Copy link
Contributor

re-reviewed, re-tested
ACK 3fdea1f

@laanwj
Copy link
Member

laanwj commented Feb 11, 2021

Code review ACK 3fdea1f

"DNS seeding disabled",
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]):
self.start_node(0, extra_args=['-dnsseed=0'])
assert time.time() - start < 5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's slightly risky to have a test that depends on real time instead of mocked times. CI environments can be extremely slow, so this bound might introduce random failures.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right - that could make for a flaky test. I picked 5 seconds as an arbitrary bound. What I really wanted to check was that the node did not wait 60 seconds to use fixed seeds.

Suggested change
assert time.time() - start < 5
assert time.time() - start < 60

accomplished that. And if CI is so slow that 60 seconds pass, the test will fail anyway because the logs will say "Adding fixed seeds as 60 seconds have passed and addrman is empty" instead of "Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted".

Copy link
Member

@laanwj laanwj Feb 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good idea!

In the longer term I think we should change this to use mock time but for this PR that would be too much of a hassle.

@LarryRuane
Copy link
Contributor

re-ACK fe3e993

@dhruv
Copy link
Contributor Author

dhruv commented Feb 12, 2021

Updated tests for CI stability. Ready for further review.

@laanwj
Copy link
Member

laanwj commented Feb 12, 2021

re-ACK fe3e993
checked that the only change since my last ACK is the < time bounds for the test

@laanwj laanwj merged commit e9c037b into bitcoin:master Feb 12, 2021
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Feb 12, 2021
…0 and peers.dat is empty

fe3e993 [p2p] No delay in adding fixed seeds if -dnsseed=0 and peers.dat is empty. Add -fixedseeds arg. (Dhruv Mehta)

Pull request description:

  Closes bitcoin#19795

  Before PR: If `peers.dat` is empty and `-dnsseed=0`, bitcoind will fallback on to fixed seeds but only after a 60 seconds delay.
  After PR: There's no 60 second delay.

  To reproduce:
  `rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0` without and with patch code

  Other changes in the PR:
  - `-fixedseeds` command line argument added: `-dnsseed=0 -fixedseeds=0 -addnode=X` provides a trusted peer only setup. `-dnsseed=0 -fixedseeds=0` allows for a `addnode` RPC to add a trusted peer without falling back to hardcoded seeds.

ACKs for top commit:
  LarryRuane:
    re-ACK fe3e993
  laanwj:
    re-ACK fe3e993

Tree-SHA512: 79449bf4e83a315be6dbac9bdd226de89d2a3f7f76d9c5640a2cb3572866e6b0e8ed67e65674c9824054cf13119dc01c7e1a33848daac6b6c34dbc158b6dba8f
with self.nodes[0].assert_debug_log(expected_msgs=[
"Loaded 0 addresses from peers.dat",
"DNS seeding disabled",
"Adding fixed seeds as -dnsseed=0, -addnode is not provided and and all -seednode(s) attempted\n"]):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and and

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see #21165

"0 addresses found from DNS seeds",
"Adding fixed seeds as 60 seconds have passed and addrman is empty"], timeout=80):
self.start_node(0, extra_args=['-dnsseed=1'])
assert time.time() - start >= 60
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't it work to use setmocktime inside the context manager above and check here: node.uptime() >= 60 to speed up the test?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at this last September (#19884 (review)) and it didn't look feasible, interesting if it is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ time test/functional/feature_config_args.py 
2021-02-12T17:02:39.647000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_kuphxowf
2021-02-12T17:02:45.294000Z TestFramework (INFO): Test config args logging
2021-02-12T17:02:45.605000Z TestFramework (INFO): Test seed peers, this will take about 2 minutes
2021-02-12T17:02:48.587000Z TestFramework (INFO): Test -networkactive option
2021-02-12T17:02:54.023000Z TestFramework (INFO): Stopping nodes
2021-02-12T17:02:54.127000Z TestFramework (INFO): Cleaning up /tmp/bitcoin_func_test_kuphxowf on exit
2021-02-12T17:02:54.127000Z TestFramework (INFO): Tests successful

real	0m14.682s
user	0m3.842s
sys	0m0.745s

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see #21165

maflcko pushed a commit that referenced this pull request Feb 25, 2021
fa730e9 test: Avoid connecting to real network when running tests (MarcoFalke)
fa1b713 test: Assume node is running in subtests (MarcoFalke)

Pull request description:

  Introduced in #19884

ACKs for top commit:
  Sjors:
    ACK fa730e9

Tree-SHA512: fe132a9ffe2fae1ab16857a3dec9839526fdf74d27a1ae794fbffca8356f639c4b916dc888b260281e9cc793916706c18d1687ebb5a076d4e1c481d218d308d3
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Feb 25, 2021
…ing tests

fa730e9 test: Avoid connecting to real network when running tests (MarcoFalke)
fa1b713 test: Assume node is running in subtests (MarcoFalke)

Pull request description:

  Introduced in bitcoin#19884

ACKs for top commit:
  Sjors:
    ACK fa730e9

Tree-SHA512: fe132a9ffe2fae1ab16857a3dec9839526fdf74d27a1ae794fbffca8356f639c4b916dc888b260281e9cc793916706c18d1687ebb5a076d4e1c481d218d308d3
luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Jun 27, 2021
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jan 27, 2022
…mpty. Add -fixedseeds arg.

Summary:
```
Before PR: If peers.dat is empty and -dnsseed=0, bitcoind will fallback on to fixed seeds but only after a 60 seconds delay.
After PR: There's no 60 second delay.

To reproduce:
rm ~/.bitcoin/peers.dat && src/bitcoind -dnsseed=0 without and with patch code

Other changes in the PR:

    -fixedseeds command line argument added: -dnsseed=0 -fixedseeds=0 -addnode=X provides a trusted peer only setup. -dnsseed=0 -fixedseeds=0 allows for a addnode RPC to add a trusted peer without falling back to hardcoded seeds.
```

Note to reviewers 1: the p2p_dos_header_tree.py test change is cherry picked from bitcoin/bitcoin@fa730e9. Without this the test will connect to the real testnet and eventually sync the headers with other nodes, that would cause the test to fail. This will be undone when the PR is backported, which requires D10909 as a dependency.
Note to reviewers 2: the test feature_config_args.py now takes forever to run, this will be fixed in a follow-up.

Backport of [[bitcoin/bitcoin#19884 | core#19884]].

Depends on D10907.

Ref T1696.

Test Plan:
  ./test/functional/test_runner.py feature_config_args.py

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Subscribers: PiRK

Maniphest Tasks: T1696

Differential Revision: https://reviews.bitcoinabc.org/D10908
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bitcoin Core does not immediately use the fixed seed nodes if DNS seeding is disabled (-dnsseed=0)