Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain/promote IPv6 usage in sample config #2866

Closed
wants to merge 2 commits into from

Conversation

@invalidator
Copy link
Contributor

commented Feb 24, 2019

  • Explain how to bind to both IPv4 and IPv6 interfaces
  • Provide a hint in the default [port_peer] section
  • Do not enable it by default

Note that use of '::' and IPv4-mapped IPv6 depends on a sysctl value setting 'net.ipv6.bindv6only = 0' which seems to be the default on most Linux distributions.

Explain/promote IPv6 usage in sample config
- Explain how to bind to both IPv4 and IPv6 interfaces
- Provide a hint in the default [port_peer] section
- Do not enable it by default 

Note that use of '::'  and IPv4-mapped IPv6 depends on a sysctl value setting 'net.ipv6.bindv6only = 0' which seems to be the default on most Linux distributions.
@ripplelabs-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Feb 24, 2019

Thank you for your submission. It will be reviewed soon and submitted for processing in CI.

@ripplelabs-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Feb 25, 2019

Thank you for your submission. It will be reviewed soon and submitted for processing in CI.

@ripplelabs-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Feb 25, 2019

Jenkins Build Summary

Built from this commit

Built at 20190503 - 23:07:05

Test Results

Build Type Log Result Status
Approval Check console Build aborted by [SYSTEM] FAIL 🔴
@nbougalis

This comment has been minimized.

Copy link
Member

commented Feb 27, 2019

Cool, I learned something new today! I appreciate you adding this hint in the config file comments!

@nbougalis
Copy link
Member

left a comment

Left a small comment for your consideration.

@@ -148,7 +148,8 @@
# ip = <IP-address>
#
# Required. Determines the IP address of the network interface to bind
# to. To bind to all available interfaces, uses 0.0.0.0
# to. To bind to all available interfaces, use 0.0.0.0
# For binding to both IPv4 and IPv6 interfaces, use ::

This comment has been minimized.

Copy link
@nbougalis

nbougalis Feb 28, 2019

Member

I think that this comment doesn't account for differences between platforms. Windows, according to https://serverfault.com/questions/21657/semantics-of-and-0-0-0-0-in-dual-stack-oses, will bind only to IPv6 addresses when :: is used.

(@ximinez, @miguelportilla: if you have IPv6 enabled & available can you check on Windows? @HowardHinnant, @scottschurr: if you have IPv6 enabled & available can you check on Mac?)

Would it be prudent to link to the above answer and reword this comment slightly:

The special notation 0.0.0.0 will bind on all available IPv4 interfaces. The special notation :: will bind to all available IPv6 interfaces and may, depending on your operating system and configuration, also bind on all IPv4 interfaces.
For more, please consult https://serverfault.com/questions/21657/semantics-of-and-0-0-0-0-in-dual-stack-oses and your O/S documentation.

Do we maybe want to add distinct keywords/directives that mean "all IPv4", "all IPv6" and "everything!"

This comment has been minimized.

Copy link
@ximinez

ximinez Feb 28, 2019

Contributor

Under Windows, my current config specifies ip = 127.0.0.1 and admin = 127.0.0.1.
Test 1. Control: I started in standalone with the current config.

$ rippled.exe --rpc_ip=127.0.0.1:5015 -q ping`
{
   "result" : {
      "role" : "admin",
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[::1]:5015 -q ping           {
   "error" : "internal",
   "error_code" : 71,
   "error_message" : "Internal error.",
   "error_what" : "no response from server"
}
# IPv6 address reported by ipconfig
$ ./build/cmake/msvc.ON/Debug/rippled.exe --rpc_ip=[2600:8807:c140:8ef:5117:598a:eeba:3633]:5015 -q ping
{
   "error" : "internal",
   "error_code" : 71,
   "error_message" : "Internal error.",
   "error_what" : "no response from server"
}
# Link-local IPv6 reported by ipconfig
$ ./build/cmake/msvc.ON/Debug/rippled.exe --rpc_ip=[fe80::5117:598a:eeba:3633]:5015 -q ping
{
   "error" : "internal",
   "error_code" : 71,
   "error_message" : "Internal error.",
   "error_what" : "no response from server"
}

Test 2. All IPv4 (ip = 0.0.0.0).

$ rippled.exe --rpc_ip=127.0.0.1:5015 -q ping
{
   "result" : {
      "role" : "admin",
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[::1]:5015 -q ping           {
   "error" : "internal",
   "error_code" : 71,
   "error_message" : "Internal error.",
   "error_what" : "no response from server"
}
# IPv6 address reported by ipconfig
$ ./build/cmake/msvc.ON/Debug/rippled.exe --rpc_ip=[2600:8807:c140:8ef:5117:598a:eeba:3633]:5015 -q ping
{
   "error" : "internal",
   "error_code" : 71,
   "error_message" : "Internal error.",
   "error_what" : "no response from server"
}
# Link-local IPv6 reported by ipconfig
$ ./build/cmake/msvc.ON/Debug/rippled.exe --rpc_ip=[fe80::5117:598a:eeba:3633]:5015 -q ping
{
   "error" : "internal",
   "error_code" : 71,
   "error_message" : "Internal error.",
   "error_what" : "no response from server"
}

Test 3: All IPv6 (ip = ::)

$ rippled.exe --rpc_ip=127.0.0.1:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}

$ /rippled.exe --rpc_ip=[::1]:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[2600:8807:c140:8ef:5117:598a:eeba:3633]:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[fe80::5117:598a:eeba:3633]:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}

Notice that none of those options report an admin connection. This is pretty disappointing.
Test 4: All IPv6 with IPv6 admin (ip = ::, admin = ::1)

$ rippled.exe --rpc_ip=127.0.0.1:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[::1]:5015 -q ping
{
   "result" : {
      "role" : "admin",
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[2600:8807:c140:8ef:5117:598a:eeba:3633]:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}
$ rippled.exe --rpc_ip=[fe80::5117:598a:eeba:3633]:5015 -q ping
{
   "result" : {
      "status" : "success"
   }
}

Conclusion:
It looks like :: under Windows causes rippled to listen on all IPv4 and IPv6 addresses, but does not allow an IPv4 address to be the admin connection. If you want IPv6, you'll have to configure an IPv6 admin address.

This comment has been minimized.

Copy link
@ximinez

ximinez Feb 28, 2019

Contributor

After reading the serverfault article, I checked what netstat reports with rippled running listening to ::. It shows as listening to both IPv4 and IPv6:

$ netstat -an | grep "5015"
  TCP    0.0.0.0:5015           0.0.0.0:0              LISTENING
  TCP    [::]:5015              [::]:0                 LISTENING

So either rippled has a special case or Windows has some automatic workaround.

This comment has been minimized.

Copy link
@mellery451

mellery451 Feb 28, 2019

Contributor

I think it might be asio on windows setting this option (https://github.com/boostorg/asio/blob/25dc6780c2c73dd6a4d74e65e854fc0f705cbb60/include/boost/asio/detail/impl/socket_ops.ipp#L1394-L1408), although I might be reading that wrong. I'm pretty sure we don't set/change that option in rippled.

This comment has been minimized.

Copy link
@scottschurr

scottschurr Mar 1, 2019

Contributor

FWIW, I poked at this a bit on macOS with @mellery451's help. On a stand alone instance, if I set the following in my config file:

[port_rpc]
ip = ::
port = 5005
protocol = http
admin = ::

then I get the following responses to the next two commands:

$ build/rippled --rpc_ip=127.0.0.1:5005 -q ping
{
   "result" : {
      "role" : "admin",
      "status" : "success"
   }
}
$ build/rippled --rpc_ip=[::]:5005 -q ping
{
   "result" : {
      "role" : "admin",
      "status" : "success"
   }
}
@invalidator

This comment has been minimized.

Copy link
Contributor Author

commented Feb 28, 2019

Thanks for the consideration and tests, I didn't even think of Windows and Mac (*BSD) etc. when filing this.

When setting up a dual stack (linux) validator I couldn't figure out how to talk the peer protocol on both v4 and v6 addresses, hence settled for :: which seemed to work. I then noticed that the only IPv6 51235/tcp entry in my nftables meter was a portscan (instead of a peer) so felt it was worthy to suggest this change.

Thinking about it, the ambiguity around IPv6 configuration and behavior across different systems (including firewall settings) is known for hindering adoption. Inadvertently exposing (admin) services comes to mind.

Perhaps it would be better to be specific about listening on multiple addresses and support something like:
ip = [ v4, v4, v6 ]
for rpc/ws/peer protocols ?

From what I gather this would take changes beyond the config file that I'm incapable of writing and in hindsight -- perhaps I should have filed a bug or feature request.

Thanks again for your efforts.

@ximinez

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2019

Perhaps it would be better to be specific about listening on multiple addresses and support something like:
ip = [ v4, v4, v6 ]
for rpc/ws/peer protocols ?

It's not the most elegant solution, but it is possible to define multiple server sections for the same protocol on the same port for different IP addresses.

[server]
port_rpc1
port_rpc2
port_rpc3
# ...plus all your other ports...

[port_rpc1]
port = 5015
ip = ::1
admin = ::1
protocol = https, http

[port_rpc2]
port = 5015
ip = 192.168.0.99
protocol = https, http

[port_rpc3]
port = 5015
ip = 127.0.0.1
admin = 127.0.0.1
protocol = https, http

As long as you don't wind up with sections that duplicate IPs listening to the same port, this works.

You'd also have to do this duplication for each "type" of listener you want.

@ripplelabs-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Apr 1, 2019

Thank you for your submission. It will be reviewed soon and submitted for processing in CI.

@ximinez

ximinez approved these changes Apr 4, 2019

Copy link
Contributor

left a comment

With the experiments that showed other OSs doing more-or-less what was expected, I'm ok to sign off on this. 👍

@ripplelabs-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Apr 30, 2019

Thank you for your submission. It will be reviewed soon and submitted for processing in CI.

@mellery451 mellery451 self-assigned this May 3, 2019

@mellery451 mellery451 self-requested a review May 3, 2019

@mellery451

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

I ran the tests outlined by @ximinez on macos and got similar results. I'd like to clarify test-case 3 which configured the unspecified listening v6 addr but localhost v4 as admin. One way to get that to work properly is to use ip = :: and admin = ::ffff:127.0.0.1,::1 which will cause admin to be accepted on both on localhost mapped v4 and localhost v6. That particular admin dual-localhost config might be worth explicitly calling out in the example config.

@ximinez

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

@mellery451 Your admin = ::ffff:127.0.0.1,::1 example works perfectly on Windows, and shows both the localhost IPs as having admin connections. It would definitely be worth calling out in the config, because it's a heck of a lot more elegant than my suggestion #2866 (comment)

@mellery451

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

@invalidator thanks for adding this information. I made a few more additions in this commit for your consideration (feel free to fold into yours if you are so inclined).

@nbougalis

This comment has been minimized.

Copy link
Member

commented May 22, 2019

@invalidator, I am going to be getting this merged into 1.3.0-b4 after folding @mellery451's commit into yours.

Are you OK with that?

@invalidator

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2019

@nbougalis sounds good, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.