New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall: allow registration of new socket types for package net #15021

Open
mdlayher opened this Issue Mar 29, 2016 · 33 comments

Comments

Projects
None yet
10 participants
@mdlayher
Member

mdlayher commented Mar 29, 2016

Overview

At this time, there is no mechanism for socket types outside of the standard library to access the runtime network poller. This proposal, if accepted, would enable a resolution to issue #10565. This would enable packages outside of the standard library to take advantage of the runtime network poller, instead of implementing their own network polling mechanism.

Proposed Change

I propose adding a new API to package net which enables registration of arbitrary sockets for use with the runtime network poller. The design of this API is based upon a comment from @rsc found here: #11492 (comment).

It seems to me that the net package should just keep using (and providing) only FileConn but perhaps we can put a registration mechanism in package syscall to let clients register converters between sockaddrs and net.Addr for non-standard sockaddr types.

This is what I was able to come up with after a little bit of experimentation. Parameter list is to be determined, but this is what I was able to get working with my prototype on a Linux system. Efforts will be made to make this mechanism as generic and cross-platform friendly as possible, but it may not be implemented immediately on non-UNIX platforms. From what I can tell, syscall.Sockaddr does appear to be available on all platforms.

package net

// Registration mechanism, perhaps called in init() or main() when a
// socket is first initalized.
func RegisterSocket(
    family int,
    sockaddr syscall.Sockaddr,
    addr Addr,
    convertSockaddr func(syscall.Sockaddr) Addr,
    convertNetAddr func(Addr) syscall.Sockaddr,
)

// Generic net.Conn and net.PacketConn implementation which embeds the
// internal net.conn type.  Checks for registered socket hooks to determine
// validity of sent and received net.Addr implementations.
type SocketConn struct {
    conn
}

Example

Using a modified version of package net, I was able to gain access to the runtime network poller and
simplify my raw sockets package code to something like the following:

// Called in init() in package raw
net.RegisterSocket(
    syscall.AF_PACKET,
    &syscall.SockaddrLinklayer{},
    &Addr{},
    // internal conversion functions for syscall.SockaddrLinklayer <-> raw.Addr
    convertSockaddr,
    convertNetAddr,
)

sock, _ := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, proto)
_ = syscall.Bind(sock, &syscall.SockaddrLinklayer{
    Protocol: pbe,
    Ifindex:  ifi.Index,
})
f := os.NewFile(uintptr(sock), "linklayer")
// c is type net.SocketConn, backed by raw socket (uses raw.Addr for addressing)
c := net.FilePacketConn(f)

Summary

The runtime network poller is an excellent mechanism, and enabling access to it will allow the future development of packages for raw ethernet sockets, netlink sockets, and other platform-specific socket types.

If this proposal is accepted, I'd happily seek guidance from @mikioh regarding creating the best possible API for this feature. In addition, this would enable me to contribute code from my raw ethernet socket package as a resolution to #8432.

Questions and comments appreciated, and thanks for your time.

/cc @rsc @mikioh

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 29, 2016

Hi,

Thanks for the work you've put in to figuring out what's possible. Can you
say more about why these are in package net instead of package syscall?
It's a bit of a non-starter for package net's API to expose so much (really
anything) from package syscall. If this were in syscall it would be more
palatable, and at first glance I don't see why it can't be. But I haven't
tried, so probably I'm missing something.

Thanks.
Russ

On Tue, Mar 29, 2016 at 2:18 PM Matt Layher notifications@github.com
wrote:

Overview

At this time, there is no mechanism for socket types outside of the
standard library to access the runtime network poller. This proposal, if
accepted, would enable a resolution to issue #10565
#10565. This would enable packages
outside of the standard library to take advantage of the runtime network
poller, instead of implementing their own network polling mechanism.

Proposed Change

I propose adding a new API to package net which enables registration of
arbitrary sockets for use with the runtime network poller. The design of
this API is based upon a comment from @rsc https://github.com/rsc found
here: #11492 (comment)
#11492 (comment).

It seems to me that the net package should just keep using (and providing)
only FileConn but perhaps we can put a registration mechanism in package
syscall to let clients register converters between sockaddrs and net.Addr
for non-standard sockaddr types.

This is what I was able to come up with after a little bit of
experimentation. Parameter list is to be determined, but this is what I was
able to get working with my prototype on a Linux system. Efforts will be
made to make this mechanism as generic and cross-platform friendly as
possible, but it may not be implemented immediately on non-UNIX platforms.
From what I can tell, syscall.Sockaddr does appear to be available on all
platforms.

package net
// Registration mechanism, perhaps called in init() or main() when a// socket is first initalized.func RegisterSocket(
family int,
sockaddr syscall.Sockaddr,
addr Addr,
convertSockaddr func(syscall.Sockaddr) Addr,
convertNetAddr func(Addr) syscall.Sockaddr,
)
// Generic net.Conn and net.PacketConn implementation which embeds the// internal net.conn type. Checks for registered socket hooks to determine// validity of sent and received net.Addr implementations.type SocketConn struct {
conn
}

Example

Using a modified version of package net, I was able to gain access to the
runtime network poller and
simplify my raw sockets package https://github.com/mdlayher/raw code to
something like the following:

// Called in init() in package raw
net.RegisterSocket(
syscall.AF_PACKET,
&syscall.SockaddrLinklayer{},
&Addr{},
// internal conversion functions for syscall.SockaddrLinklayer <-> raw.Addr
convertSockaddr,
convertNetAddr,
)
sock, _ := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, proto)
_ = syscall.Bind(sock, &syscall.SockaddrLinklayer{
Protocol: pbe,
Ifindex: ifi.Index,
})f := os.NewFile(uintptr(sock), "linklayer")// c is type net.SocketConn, backed by raw socket (uses raw.Addr for addressing)c := net.FilePacketConn(f)

Summary

The runtime network poller is an excellent mechanism, and enabling access
to it will allow the future development of packages for raw ethernet
sockets, netlink sockets, and other platform-specific socket types.

If this proposal is accepted, I'd happily seek guidance from @mikioh
https://github.com/mikioh regarding creating the best possible API for
this feature.

Questions and comments appreciated, and thanks for your team.

/cc @rsc https://github.com/rsc @mikioh https://github.com/mikioh


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15021

@mdlayher

This comment has been minimized.

Member

mdlayher commented Mar 29, 2016

Hi Russ,

It appears to me that the runtime network poller functionality is exposed directly using the net.pollDesc type, which is then wrapped by net.netFD and net.conn. From what I can tell, it appears I'd have to duplicate or possibly move some code from package net to package syscall in order to enable accessing the runtime network poller there.

Specifically, I'm looking at net/fd_unix.go, and net/fd_poll_runtime.go. Is there an easier answer to this problem that I am not yet aware of?

Thanks for the quick reply, and I look forward to hearing back.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 29, 2016

Your Example looks exactly as I think it should except for the fact that
the call is net.RegisterSocket instead of syscall.RegisterSocket. Can
RegisterSocket be moved into syscall? If not, why not? Thanks.

On Tue, Mar 29, 2016 at 3:08 PM Matt Layher notifications@github.com
wrote:

Hi Russ,

It appears to me that the runtime network poller functionality is exposed
directly using the net.pollDesc type, which is then wrapped by net.netFD
and net.conn. From what I can tell, it appears I'd have to duplicate or
possibly move some code from package net to package syscall in order to
enable accessing the runtime network poller there.

Specifically, I'm looking at net/fd_unix.go, and net/fd_poll_runtime.go.
Is there an easier answer to this problem that I am not yet aware of?

Thanks for the quick reply, and I look forward to hearing back.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15021 (comment)

@mdlayher

This comment has been minimized.

Member

mdlayher commented Mar 29, 2016

I am in favor of moving it to package syscall instead, but because much of the low-level functionality to access the runtime network poller already exists in package net, I am not sure how to best access the network poller from package syscall without duplicating code or moving it to syscall entirely.

Can you suggest a viable approach to solving this problem? Perhaps the low level runtime network poller code in net could be moved to an internal package that both net and syscall use to access it?

In summary, yes, I am in favor of creating syscall.RegisterSocket instead, but I am simply not sure how to access the runtime network poller from package syscall instead of package net, where it resides now, without code duplication.

EDIT: in addition, since net imports syscall, but I would need the net.Addr type in syscall, this would create a circular dependency. Any thoughts on how to resolve this issue as well?

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 29, 2016

Can you point me at the code you changed? Maybe that will help me see why
RegisterSocket cannot easily move into syscall. Thanks.

On Tue, Mar 29, 2016 at 3:45 PM Matt Layher notifications@github.com
wrote:

I am in favor of moving it to package syscall instead, but because much
of the low-level functionality to access the runtime network poller already
exists in package net, I am not sure how to best access the network
poller from package syscall without duplicating code or moving it to
syscall entirely.

Can you suggest a viable approach to solving this problem? Perhaps the low
level runtime network poller code in net could be moved to an internal
package that both net and syscall use to access it?

In summary, yes, I am in favor of creating syscall.RegisterSocket
instead, but I am simply not sure how to access the runtime network poller
from package syscall instead of package net, where it resides now,
without code duplication.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15021 (comment)

@mdlayher

This comment has been minimized.

Member

mdlayher commented Mar 29, 2016

Here's a link to the prototype I made to make this work. Obviously, it would need cleanup, proper locking, and probably should be wrapped in a type.

https://github.com/mdlayher/go/commit/2e7d77c17dbc830135ca35eb5617e073c8fc717b

@mdlayher

This comment has been minimized.

Member

mdlayher commented Apr 4, 2016

Hey @rsc , did you get a chance to look at the prototype I created above? I'd love to hear your thoughts on it. Thanks for your time.

@mikioh

This comment has been minimized.

Contributor

mikioh commented Apr 8, 2016

@mdlayher,

I didn't have a look at your proposal/prototype carefully, and probably have no spare time for this feature (including #10565 opened by me) during Go 1.7 development cycle, but a bit. I think that the basic requirements for this feature look like the following:

  1. Need to use the net package
    • As a glue to crypto/tls and net/http packages
  2. Need to have a replacement for the syscall package
    • No use of types in the syscall package is mandatory because the syscall package is locked down and will be deprecated
    • The new package should provide a type that represents some sort of tuple known as a socket
    • The new package should provide flexibility for system adaptation not only for the existing kernels but for upcoming kernels that have fancy new network sockets
  3. Need to have a replacement for the runtime package
    • This requirement might be optional, but having a way to use runtime-integrated network poller would be better
    • The new package should provide basic event notification mechanism on the runtime-integrated network poller

#10565 just covers R1, and perhaps your proposal too.

I'd like to see a design that fulfils the above requirements, at least R1 and R2. Otherwise, the design would be the source of trouble in future, like "I cannot achieve my goal because of the lack of R2 (or both R2 and R3)." I suppose that a typical use case would be either R1+R2 or R2+R3; the former is for people who want to use the existing read/write methods in the net package with new socket types and the latter is for people who want to use new read/write methods with new socket types.

Thoughts?

@mdlayher

This comment has been minimized.

Member

mdlayher commented Apr 8, 2016

@mikioh Thanks for replying. Your requirements seem sound, but I wouldn't even know where to begin covering R2 and R3 (mine only covers R1).

Happy to help out if you have any sketches of a proposed API or something similar, but I don't have the knowledge of the internals of package runtime to create a public interface to the runtime network poller, etc.

@minux

This comment has been minimized.

Member

minux commented Apr 8, 2016

@mikioh

This comment has been minimized.

Contributor

mikioh commented Apr 8, 2016

FTR, https://golang.org/s/go1.4-syscall says that:

Note that we cannot clean up the existing syscall package to any meaningful extent because of the compatibility guarantee. We can freeze and, in effect, deprecate it, however.

not literally, but in effect.

@bradfitz bradfitz modified the milestone: Unplanned Apr 9, 2016

@mikioh

This comment has been minimized.

Contributor

mikioh commented Apr 14, 2016

@mdlayher,

I think x/net repository is a good place for the new package implementing R2, because we can copy the package into src/vendor/golang.org/x/net/ and plumb it in the net package later.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Aug 22, 2016

I will investigate options a bit here.

@mdlayher

This comment has been minimized.

Member

mdlayher commented Aug 23, 2016

Thanks, @bradfitz ! Looking forward to seeing what you come up with. Happy to help if there's anything I could do to assist you.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 26, 2016

(Still on my plate to look at.)

@bradfitz bradfitz modified the milestones: Proposal, Unplanned Oct 18, 2016

@bradfitz

This comment has been minimized.

Member

bradfitz commented Oct 18, 2016

The runtime already exposes a portable netpoll API to the net package, using uintptr as the portable FD type (like https://golang.org/pkg/os/#File.Fd), since that's what works between Windows and everything else.

I'd prefer to see a new package exposing a more Go-like (and less net/syscall-like) interface to that poller. I'd ideally like to see it done without any syscall.Sockaddr crud.

At the heart of epoll & friends is a simple API letting people register FDs they care about (and how: readability vs writability) and wait on a bunch of them at once.

We could have an API with uintptrs to watch/unwatch, a Go type for readable/writeable, and func callbacks when the edge triggers.

Does anybody want to explore prototyping that?

@mikioh

This comment has been minimized.

Contributor

mikioh commented Oct 18, 2016

I don't understand what you mean by "Go-like (and less net/syscall-like)." Can you please elaborate on that? What's your plan to accommodate user-specified socket descriptors with net package API access?

My guess is that your plan is just to provide a package like libuv for Node.js or mio for Rust. The package neither provide new IO methods such as {recv,send}mmsg nor replacements for broken APIs in frozen syscall and x/sys packages such as Sendfile for Darwin. Also the package doesn't help to inject user-specified socket descriptors with the net package via File{Conn,Listener,PacketConn} APIs because it doesn't address #10565. Is my understanding correct?

@elliotmr

This comment has been minimized.

elliotmr commented Dec 18, 2016

There is already an interface for the network poller defined, it is just that it is linked directly into the net package using go:linkname on private functions in runtime/netpoll.go. Is there some reason that the methods there (ServerInit, Open, Close, Reset, Wait, WaitCanceled, SetDeadline, and Unblock) can't be just exposed as public APIs?

This is exactly what I need for implementing an API over an AF_CAN socket type (https://groups.google.com/forum/#!topic/golang-nuts/l-1JD02_o0Q). Doing some digging this morning I found out that my options 2 and 3 are not possible because of this special link between the runtime and net packages.

@rsc rsc changed the title from Proposal: provide access to runtime network poller via socket registration mechanism to proposal: syscall: allow registration of new socket types for package net Jan 23, 2017

@mdlayher

This comment has been minimized.

Member

mdlayher commented Feb 6, 2017

FYI: mikioh recently submitted a series of three CLs which appear to be designed to implement a version of the solution proposed here.

https://go-review.googlesource.com/c/35995/
https://go-review.googlesource.com/c/35996/
https://go-review.googlesource.com/c/35994/

They are marked "DO NOT REVIEW" but I wanted to make sure they were mentioned for anyone keeping an eye on this issue.

@gopherbot

This comment has been minimized.

gopherbot commented Feb 10, 2017

CL https://golang.org/cl/36799 mentions this issue.

@gopherbot

This comment has been minimized.

gopherbot commented Feb 10, 2017

CL https://golang.org/cl/36800 mentions this issue.

@gopherbot

This comment has been minimized.

gopherbot commented Jun 14, 2017

CL https://golang.org/cl/45790 mentions this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Jun 14, 2017

The original post says:

At this time, there is no mechanism for socket types outside of the standard library to access the runtime network poller.

This is no longer true. Other packages can now construct socket fds directly and then call os.NewFile to get a *os.File. Operations on that *os.File will use the network poller. Taking the example from the top:

sock, _ := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, proto)
_ = syscall.Bind(sock, &syscall.SockaddrLinklayer{
    Protocol: pbe,
    Ifindex:  ifi.Index,
})
f := os.NewFile(uintptr(sock), "linklayer")
// c is type net.SocketConn, backed by raw socket (uses raw.Addr for addressing)
c := net.FilePacketConn(f)

If you just don't do the last line and use f in blocking i/o operations, those operations will use the network poller.

However, it's still not possible to turn a socket fd into a net.Conn/net.PacketConn/net.Listener (as in the last line), which you might want for interoperation with other software expecting a net.Conn or for timeout manipulation. That's what would be served by a new registry. The registry would need to maintain the association between (family int, sotype int) <-> (net string) and then also for each such association the corresponding (addr string)<->(sa sockaddr) conversion functions.

Given those, package net can introduce a new unexported net.Conn and net.Addr implementation that uses that functionality.

The CL I just posted (https://golang.org/cl/45790) compiles but has parts missing. Still, it should be concrete enough to make clear what I mean. I'm not 100% sure about the hooking into Dial as shown - maybe that's a bridge too far. If we did that we'd of course want to do Listen as well, but there's no DialPacket and I'm disinclined to add one just for this. I'm also a little scared of making the new types seem too much like standard types. It might be better to save Dial for standard types only.

Hooking into Dial in the example CL was really just a way to exercise the string->sockaddr conversion signature. It's possible we can drop it if we don't do Dial.

@rsc

This comment has been minimized.

Contributor

rsc commented Jun 14, 2017

If someone wants to write a CL along those lines for Go 1.10 and see if it can be done fairly cleanly, I think we agree it's worthwhile. Leaving Dial out, at least for the first version, should make it unobjectionable.

I'm going to mark this approved assuming a sketch like in the CL.

@mdlayher

This comment has been minimized.

Member

mdlayher commented Jun 14, 2017

I'm happy to give this a try. Thanks for the example CL!

@jstarks

This comment has been minimized.

jstarks commented Jul 12, 2017

@rsc, it looks to me like os.NewFile will never enable the poller on Linux (it always passes pollable = false to poll.FD.Init()), so there is still no way to use the network poller for foreign fds. Am I missing something?

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jul 12, 2017

@jstarks I think you're right; that too would have to change.

@mdlayher

This comment has been minimized.

Member

mdlayher commented Sep 2, 2017

I'm not sure I'll be able to take this on. @mikioh, are you interested?

@rsc rsc changed the title from proposal: syscall: allow registration of new socket types for package net to syscall: allow registration of new socket types for package net Nov 29, 2017

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 29, 2017

@acln0

This comment has been minimized.

Contributor

acln0 commented May 8, 2018

Hello. Is anyone working on this? If not, I am willing to do the work, be it for 1.11 or 1.12.

EDIT: In light of ea5825b, which I only now discovered, perhaps this is less relevant now. But the issue of creating a net.Conn from such an *os.File still remains, for what it's worth. If there is nevertheless interest in the socket registry, I can try to take this on.

@mdlayher

This comment has been minimized.

Member

mdlayher commented May 8, 2018

Please feel free to take this on!

@acln0

This comment has been minimized.

Contributor

acln0 commented May 8, 2018

I have thought about this some more. If I understand things correctly, I believe the socket type registry is only worth implementing if there are also going to be hooks into net.Dial (and net.Listen etc.) for it.

Since a non-blocking *os.File can now be created from a raw file descriptor, a hypothetical bluetooth package living outside the standard library can, schematically, do something like this:

package bluetooth

import (
	"net"
	"os"
	"time"

	"golang.org/x/sys/unix"
)

type Conn struct {
	laddr *addr
	raddr *addr
	fd    *os.File
}

func DialHCI(device uint16) (*Conn, error) {
	const (
		family  = unix.AF_BLUETOOTH
		sotype  = unix.SOCK_RAW | unix.SOCK_CLOEXEC | unix.SOCK_NONBLOCK
		proto   = unix.BTPROTO_HCI
		channel = unix.HCI_CHANNEL_CONTROL
	)
	rawFD, err := unix.Socket(family, sotype, proto)
	if err != nil {
		return nil, err
	}
	sa := unix.SockaddrHCI{
		Dev:     device,
		Channel: channel,
	}
	if err := unix.Bind(rawFD, &sa); err != nil {
		return nil, err
	}
	return &Conn{
		laddr: &addr{
			// put fields from sa here
		},
		fd: os.NewFile(uintptr(rawFD), "hci"),
	}, nil
}

func (c *Conn) LocalAddr() net.Addr {
	return c.laddr
}

func (c *Conn) RemoteAddr() net.Addr {
	return c.raddr
}

// Remaining net.Conn method implementations are elided for brevity, since all the
// calls are forwarded to their respective calls on c.fd, which work as expected.

type addr struct {
	// bluetooth-specific things in here
}

func (a *addr) Network() string { return "bluetooth" }

func (a *addr) String() string {
	// use fields on addr to construct this
	return ""
}

It's not difficult to imagine a similar construction for packet-oriented connections, with a straight-forward type assertion inside the library for methods like WriteTo. (EDIT: on second thought, perhaps this is not so easy after all...)

With this approach, users of the hypothetical bluetooth package would now have a usable, non-blocking net.Conn, without too much effort from either them or the package authors.

On the other hand, the nice property of being able to pass the strings returned by net.Addr.Network and net.Addr.String into net.Dial or net.Listen would probably be lost.

The existence of a socket type registry in the syscall package would potentially hide some of the details from end-users (and preserve the aforementioned property), so long as the generic net.Dial would be able to express specific requirements for the new socket. That being said, if users end up having to encode specific details about the socket in the address string passed to net.Dial (akin to the DSN passed to sql.Open), I believe the socket registry would not be an overall gain, and would lead to poorly-specified and obscure interfaces.

I hope I have not misunderstood the issues at hand. Moving forward, if the registry is still something we would like to implement, I can do the work for it. Otherwise, perhaps this issue could be closed in light of ea5825b.

Thank you.

@bradfitz bradfitz modified the milestones: Go1.11, Go1.12 May 18, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone May 31, 2018

@acln0

This comment has been minimized.

Contributor

acln0 commented Jul 21, 2018

Sorry for talking to myself here. I have thought about this some more, and I have a more concrete proposal.

For 1.12, I plan to send a CL that adds the registration mechanism to package syscall, and changes package net to make FileConn and FilePacketConn work, when the *os.File argument to these functions refers to a socket, and the type of the socket was registered with syscall.

I think hooking into Dial (or other such intrusive things) can be discussed at a later time, if and when people use the new interface and provide feedback.

Does that sound good?

@gopherbot

This comment has been minimized.

gopherbot commented Sep 20, 2018

Change https://golang.org/cl/136595 mentions this issue: net, syscall: implement socket registration mechanism

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment