Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/netip: add new IP address package, use in net #46518

Closed
bradfitz opened this issue Jun 2, 2021 · 23 comments
Closed

net/netip: add new IP address package, use in net #46518

bradfitz opened this issue Jun 2, 2021 · 23 comments

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jun 2, 2021

I propose we fix #18804 (net: reconsider representation of IP) by importing the inet.af/netaddr into the standard library, probably as net/netaddr, so the IP type is netaddr.IP (the net package already has an IP type). We wouldn't need to import all of it, at least not right away, but I think it'd be nice if we did... the package types work together very well.

This blog post explains the advantages of netaddr.IP over net.IP:

https://tailscale.com/blog/netaddr-new-ip-type-for-go/

Notably, it's small, comparable, and doesn't allocate.

Code is at https://github.com/inetaf/netaddr. Everybody has a Go CLA filed ( inetaf/netaddr#181) and the license is compatible.

Having an allocation-free IP address in the standard library means we'd be able to fix #45886 ("net: add API to receive multiple UDP packets (potentially in one system call)") which ends with #45886 (comment) ("We may want to wait on doing anything here until we figure out what to do with IP addresses, which still allocate."). More generally, this would improve Go's UDP performance, which has never gotten much attention. But with things like QUIC and WireGuard (both UDP-based), Go's alloc-heavy UDP handling is starting to get in the way.

Yes, we'd then have two IP types in the standard library.

As part of this, we'd then add:

  • conversion funcs in net to go between them (net would depend on netaddr, not vice-versa)
  • the net package internals would switch to netaddr.IP for performance/allocs, converting to net.IP or IPAddr on the edges only.
  • a couple new methods would be added (notably to UDPConn) to send/receive UDP packets using netaddr.IP alloc-free (we'd then have ~5 ways to send/recv UDP packets instead of ~4)

/cc @danderson, @josharian, @mdlayher, @crawshaw, @tklauser (co-authors), @neild (co-author for #45886)

@gopherbot gopherbot added this to the Proposal milestone Jun 2, 2021
@mvdan
Copy link
Member

@mvdan mvdan commented Jun 2, 2021

cc @marten-seemann for quic-go

@dotwaffle
Copy link

@dotwaffle dotwaffle commented Jun 2, 2021

If there's any way we could address the JSON marshalling of CIDR address formats at the same time, that would be very convenient -- rather than having to rely on making custom encoders etc.

I realise that the backwards compatibility guarantee means that's unlikely to be fixed in the standard library version of "net", but if it could be addressed in this proposed new package, it would certainly make the life of network engineers much simpler!

Example: https://play.golang.org/p/3as8Ux9AzJG

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Jun 2, 2021
@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Jun 2, 2021

If there's any way we could address the JSON marshalling of CIDR address formats at the same time, that would be very convenient -- rather than having to rely on making custom encoders etc.

https://play.golang.org/p/5QWiD-tArCc

fmt:	 192.0.2.0/24
json:	 "192.0.2.0/24"

@taktv6
Copy link

@taktv6 taktv6 commented Jun 2, 2021

As someone who had to invent their own IP datatype (https://pkg.go.dev/github.com/bio-routing/bio-rd/net#IP) for the very same reasons, I'm really glad to see progress on this topic. Looking forward to see a decent IP datatype in Golang.

Thanks!

@rsc rsc changed the title proposal: add new IP address type, netaddr package proposal: net/netaddr: add new IP address type, netaddr package Jun 2, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Jun 2, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals Jun 2, 2021
@kylelemons
Copy link
Contributor

@kylelemons kylelemons commented Jun 3, 2021

Would it be worth considering adding a runtime primitive of some sort to help avoid the "workarounds" in the intern package? I don't think it would need to be done at the same time, but having a user for it in the stdlib might make it more compelling.

@Merovius
Copy link

@Merovius Merovius commented Jun 3, 2021

@kylelemons There have been and are some proposals related to that, most notable probably #43615. In the interim, the workarounds are much less problematic when they are in the stdlib, because if anything is changed in the implementation that would prevent them from working, they can be changed alongside with it.

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Jun 4, 2021

@kylelemons, what @Merovius said. And to be clear (to anybody just researching the intern package and being horrified), we are definitely not proposing exposing the intern package or any of its API. It's very much an internal implementation detail.

@zephyr
Copy link

@zephyr zephyr commented Jun 7, 2021

I don’t want to start bikeshedding, but i think that the proposed type names look a little bit cluttered, since they all share the IP prefix:

net/netaddr.IP
net/netaddr.IPPort
net/netaddr.IPPrefix
net/netaddr.IPRange
net/netaddr.IPSet
net/netaddr.IPSetBuilder

Couldn’t the IP-part just move into the package name instead:

net/ip.Addr
net/ip.Port
net/ip.Prefix
net/ip.Range
net/ip.Set
net/ip.SetBuilder

Or maybe something like net/netip as package name, if net/ip is too short?

Otherwise: Thank you, @bradfitz, for this great proposal and your highly enjoyable blog post!

@mdlayher
Copy link
Member

@mdlayher mdlayher commented Jun 7, 2021

My only nitpick about the package name ip is that I'd guess that is the single most commonly used variable name for any variable of type net.IP, so there's a high likelihood of having to change significant sections of code to avoid shadowing the import net/ip in code being retrofit with the new package APIs.

@zephyr
Copy link

@zephyr zephyr commented Jun 7, 2021

@mdlayher While i still think that ip is the nicest option in theory, you have certainly a valid practical concern. Maybe something net/ipa (for ›Internet Protocol address«), to avoid shadowing and jet have something short and sensible?

@andig
Copy link
Contributor

@andig andig commented Jun 7, 2021

Might also make it ip2 as introduced in Go "2". Or use netip as package if we can forget that it belongs below net for a second.

@rsc
Copy link
Contributor

@rsc rsc commented Jul 21, 2021

Opened a discussion at #47323.

@golang golang locked and limited conversation to collaborators Jul 21, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 2, 2021

Change https://golang.org/cl/339309 mentions this issue: net/netaddr: add new IP address package

@rsc
Copy link
Contributor

@rsc rsc commented Sep 1, 2021

Over on #47323, people seem generally OK with package net/netip containing

type Addr
type AddrPort
type Prefix
type Range

Commenting here for more visibility. Please comment on #47323 if you have opinions. Thanks.

@rsc rsc changed the title proposal: net/netaddr: add new IP address type, netaddr package proposal: net/netip: add new IP address package, use in net Sep 8, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Sep 8, 2021

Given that there seems to be consensus on the four types, it seems like the full API for those would be:

package net

func (r *Resolver) LookupNetAddr(ctx context.Context, network, host string) ([]netip.Addr, error)

func TCPAddrPort(addr netip.AddrPort) *TCPAddr
func (a *TCPAddr) AddrPort() netip.AddrPort

func UDPAddrPort(addr netip.AddrPort) *UDPAddr
func (a *UDPAddr) AddrPort() netip.AddrPort
 
func (c *UDPConn) ReadMsgUDPAddrPort(b, oob []byte) (n, oobn, flags int, addr netip.AddrPort, err error)
func (c *UDPConn) WriteToUDPAddrPort(b []byte, addr netip.AddrPort) (int, error)
func (c *UDPConn) WriteMsgUDPAddrPort(b, oob []byte, addr netip.AddrPort) (n, oobn int, err error)


package netip

type Addr struct { ... unexported ... }

func From4(a,b,c,d byte) Addr
func From16(addr [16]byte) Addr  // "raw", no unmapping
func FromSlice(std []byte) (ip Addr, ok bool)  // "raw", no unmapping
func IPv6LinkLocalAllNodes() Addr
func IPv6Unspecified() Addr
func MustParseAddr(s string) Addr
func ParseAddr(s string) (Addr, error)

func (ip Addr) AppendTo(b []byte) []byte
func (ip Addr) As16() [16]byte
func (ip Addr) As4() [4]byte
func (ip Addr) BitLen() uint8
func (ip Addr) Compare(ip2 Addr) int
func (ip Addr) Is4() bool
func (ip Addr) Is4in6() bool
func (ip Addr) Is6() bool
func (ip Addr) IsGlobalUnicast() bool
func (ip Addr) IsInterfaceLocalMulticast() bool
func (ip Addr) IsLinkLocalMulticast() bool
func (ip Addr) IsLinkLocalUnicast() bool
func (ip Addr) IsLoopback() bool
func (ip Addr) IsMulticast() bool
func (ip Addr) IsPrivate() bool
func (ip Addr) IsUnspecified() bool
func (ip Addr) IsValid() bool
func (ip Addr) IsZero() bool
func (ip Addr) Less(ip2 Addr) bool
func (ip Addr) MarshalBinary() ([]byte, error)
func (ip Addr) MarshalText() ([]byte, error)
func (ip Addr) Next() Addr
func (ip Addr) Prefix(bits uint8) (Prefix, error)
func (ip Addr) Prev() Addr
func (ip Addr) String() string
func (ip Addr) StringExpanded() string
func (ip Addr) Unmap() Addr
func (ip *IP) UnmarshalBinary(b []byte) error
func (ip *IP) UnmarshalText(text []byte) error
func (ip Addr) WithZone(zone string) Addr
func (ip Addr) Zone() string

type AddrPort struct { ... unexported ... }

func AddrPortFrom(ip Addr, port uint16) AddrPort
func ParseAddrPort(s string) (AddrPort, error)
func MustParseAddrPort(s string) AddrPort

func (p AddrPort) AppendTo(b []byte) []byte
func (p AddrPort) Addr() Addr
func (p AddrPort) IsValid() bool
func (p AddrPort) IsZero() bool
func (p AddrPort) MarshalText() ([]byte, error)
func (p AddrPort) Port() uint16
func (p AddrPort) String() string
func (p *AddrPort) UnmarshalText(text []byte) error
func (p AddrPort) WithIP(ip Addr) AddrPort
func (p AddrPort) WithPort(port uint16) AddrPort

type Prefix struct { ... unexported ... }

func PrefixFrom(ip Addr, bits int) Prefix
func MustParsePrefix(s string) Prefix
func ParsePrefix(s string) (Prefix, error)

func (p Prefix) AppendTo(b []byte) []byte
func (p Prefix) Bits() int
func (p Prefix) Contains(ip Addr) bool
func (p Prefix) Addr() Addr
func (p Prefix) IsSingleIP() bool
func (p Prefix) IsValid() bool
func (p Prefix) IsZero() bool
func (p Prefix) MarshalText() ([]byte, error)
func (p Prefix) Overlaps(o Prefix) bool
func (p Prefix) String() string
func (p *Prefix) UnmarshalText(text []byte) error

I've also updated the discussion's top comment with this API. Please comment there if you have opinions. Thanks.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 15, 2021

Minor updates at #47323 (comment).

@rsc
Copy link
Contributor

@rsc rsc commented Sep 22, 2021

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Accept in Proposals Sep 22, 2021
@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Oct 6, 2021

@rsc, I assume your #46518 (comment) comment is just missing the Range API, as the comment before it includes it as one of the four types?

@rsc rsc moved this from Likely Accept to Accepted in Proposals Oct 6, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Oct 6, 2021

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: net/netip: add new IP address package, use in net net/netip: add new IP address package, use in net Oct 6, 2021
@rsc rsc removed this from the Proposal milestone Oct 6, 2021
@rsc rsc added this to the Backlog milestone Oct 6, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 2, 2021

Change https://golang.org/cl/360634 mentions this issue: doc/go1.18: add net/netip

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 7, 2021

Change https://golang.org/cl/370134 mentions this issue: doc/go1.18: clarify additions to net package API

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 21, 2021

Change https://golang.org/cl/373834 mentions this issue: doc/go1.18: list new net/netip and net functions and methods

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Proposals
Accepted
Development

No branches or pull requests