support SO_REUSEPORT socket option #108
Comments
|
Without this option, I must run each instance of this support on a separate workstation (or using a separate NIC if multiple NICs are available on the same workstation). |
|
Stackoverflow has some entries about My goal is to run multiple IOCs on the same host from same username, each supporting a different CS800 controller. Advice is welcome. |
|
I don’t see any major down side to adding this on systems which provide SO_REUSEPORT when an explicit local port is specified. Just add the setsockopt() call after the bina() and inside an #ifdef SO_REUSEPORT conditional. Maybe also with a warning message if the call fails.
I does make things yet a little more unportable, though — I’m not sure that I’d want to try to add all the required conditionals and code to support Windows, for example.
… On Apr 18, 2020, at 9:34 AM, Pete R Jemian ***@***.***> wrote:
The SO_REUSEPORT <https://lwn.net/Articles/542629/> socket option allows multiple clients on the same ***@***.*** to bind to the same UDP port. With this option enabled in asyn, I expect to be able to run multiple IOCs using new support <https://github.com/epics-modules/ip/blob/969799373765c57a4992bc33f5e7e227b1f453bc/documentation/Oxford_CS800.md> for the Oxford CS800 CryoStream controller (blows cold nitrogen gas at protein sample during crystallography measurement). Currently, when I try to run a second IOC from the same ***@***.***, this error is reported to the console and no communications is the result:
# ### cs800.iocsh ###
# STATUS_ADDR "255.255.255.255:30304:30304 UDP"
# COMMAND_ADDR "10.0.0.173:30305 UDP*"
# Status Packets
drvAsynIPPortConfigure("OC_SP", "192.168.144.169:30304:30304 UDP*", 0, 0, 0)
2020/04/18 11:16:04.544 OC_SP -1 autoConnect could not connect: unable to bind to local port: Address already in use
# Commands
Is it possible to call setsockopt() (similar to what is shown <https://lwn.net/Articles/542629/>) from the st.cmd file?
Otherwise, can support for the SO_REUSEPORT socket option be enabled in asyn? The text SO_REUSEPORT does not appear anywhere in the asyn repo so I assume this option is not available yet.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#108>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECM3GADX2BXEJHEFTCKTILRNHJAFANCNFSM4MLNKFMQ>.
|
|
I don't think this is going to work. If the devices are sending UDP data to the same port on the IOC host (as the protocol file suggests), only the last IOC that was started will get all data. (Unless the devices are sending broadcasts. Then all IOCs will get all data. That's technically the same issue as with Channel Access name resolution not working properly on a Linux box running multiple IOCs if the client uses unicasts.) Have a look at how to configure virtual network interfaces on your host - which is actually pretty simple on Linux. Then every IOC can use its own IP address for the device connection and all is good. |
|
+1 for virtual NICs, thanks. The CS800 controllers broadcast status once a second by UDP to port 30304. Another problem to resolve is all status broadcasts being received by every IOC. That seems to be be resolved by specifying the IP number of the controller: At least it works when I start an IOC and two simulated CS800 controllers on the same subnet. I can test with real controllers on Monday when the beam is down. |
|
The |
|
Then a reused port is the same as multiple virtual NICs. |
|
Trying it but it fails. Changing |
|
Summary is that any and all status broadcasts will come in to asyn configured: which moves the problem a little further along to stream: how to separate the status broadcasts by their sender? The IP and MAC addresses of the sender are in the packet header and the IP address is also in the packet data. A C or SNL program could filter out the status from the desired controller but filtering by IP sender that is probably beyond the capability of a stream protocol. |
|
Ok. When you want to filter by sender address by configuring UDP "connections", you need a separate for each - then we need the
|
|
I agree. How hard is it to add? |
|
And for now, support each CS800 controller from a separate host. |
|
virtual NICs should work for now. |
|
That's much easier on the home LAN than the working beam line. They are very risk averse since they can do virus-related work now. |
|
Thanks for all the help! |
|
As Eric mentioned: the change itself is not too hard to do. A single call on the socket. Windows portability has been solved for the Channel Access case (in the UDP server). Question is if Mark wants to keep the sockets single-use by default (which might make sense) and - if so - which would be the appropriate way to configure the REUSEPORT option. Welcome! |
|
One message from Eric does not show up here - I just got the mail, don't know what happened:
|
|
Well, I always thought that I do agree that the kernel should have all necessary information to get the routing right. I never worked with such a UDP "connection", so I can't say if this setup falls under the UDP server case where all non-broadcast packages get delivered to exactly one process waiting on the shared port. Nevertheless: I don't think adding the |
|
Questions:
I can think of 2 ways to handle the API.
I will work on this once I hear a consensus on which variations make sense. |
|
Sorry, I just noticed Ralph's comment:
However, this article explains the uses of SO_REUSEPORT, and how it can be useful for both TCP and UDP. https://lwn.net/Articles/542629/ But for TCP I don't think we want to make it the default, so we do need an API change to handle it. |
|
I have created a new branch of asyn, SO_REUSEPORT. It implements option 1) above, i.e. new protocols:
On _WIN32 I used the socket option SO_REUSEADDR because SO_REUSEPORT is not supported. This article says that it does what Pete wants for UDP broadcast sockets: On all other architectures it uses the socket option SO_REUSEPORT. I have tested that it compiles on Linux, vxWorks, and Windows VS2015. @prjemian can you see if it works for you on Linux? |
|
Thanks for the quick response. I'll get to it later today. Faster if the weather goes bad but that's not likely. If this works, this can save me from a rewrite of the CS800 support in SNL. |
|
Right — here’s the salient section of the connect(2) man page — I infer from, “only address from datagrams are to be received”, that this should take care of Pete Jemian’s problem.
DESCRIPTION
The parameter socket is a socket. If it is of type SOCK_DGRAM, this call
specifies the peer with which the socket is to be associated; this address
is that to which datagrams are to be sent, and the only address from which
datagrams are to be received. If the socket is of type SOCK_STREAM, this
call attempts to make a connection to another socket. The other socket is
specified by address, which is an address in the communications space of the
socket.
… On Apr 19, 2020, at 1:36 AM, Ralph Lange ***@***.***> wrote:
Well, I always thought that connect() on a UDP socket only stores the default peer data, so that the application can use send() and recv() instead of the fully specified _from() versions of the calls. (And be more efficient, as routing does not have to be checked for every send/recv call.)
|
|
Thanks a lot, Mark. The change looks good. This will allow adding the SO_REUSEPORT flag for all client type configurations, so that the originating port of the connection can be shared. Server type ports are still exclusive. IMHO this is ok (as it would directly lead to the issue with incoming broadcasts vs. unicasts) and we can relax and wait for someone to actually need that feature. Just wanted to make it clear. |
|
Working with two CS800 simulators:
|
|
On Apr 20, 2020, at 10:49 AM, Pete R Jemian ***@***.***> wrote:
Working with two CS800 simulators:
PREFIX IP controller ID
cs800: 192.168.144.144 144
cryo: 192.168.144.113 113
UDP*&
Starting support for both from the same linux-x86_64 soft IOC (running on 192.168.144.99) using UDP*& for both status and command ports:
st.cmd configuration
No errors on startup:
IOC console output on startup
Watching for the controller ID for each instance. Both EPICS database instances receive status updates from both controllers (this is not correct).
Verified that command sent from EPICS PV to one controller is only received by the correct controller. For both controllers. (this is correct)
UDP&
Changing to UDP& (remove the broadcast configuration) and same results for status and commands. That is, each database receives all status updates (this is not correct). Commands sent to intended controller (this is correct)
This sounds like the network stack is not limiting reception to just those packets from the sender specified by the connect(). My interpretation of the connect(2) documentation is that the effect you’re seeing means that the network code is broken, but my interpretation doesn’t count for much!
:30304:30304 vs. :30304
When this part is shortened, no status updates are received by either controller
This is expected since the version without the local port specification (the second number above) just lets the kernel pick an unused port number. The CS800 can’t handle this since it has the destination port wired in.
How can this be configured so that status updates from controller 144 are received only by the EPICS database for that controller?
Fix the network stack?
Place the devices on different virtual NICs?
Tell the manufacturer to use a less bizarre communications method?
|
|
@prjemian I think the simplest configuration is to use UDP& on the status port and UDP on the command port. I don't think there is a need for the broadcast flag on the status port. The broadcast flag is to allow EPICS to send broadcasts, nothing special needs to be done to receive them. The command port does not need the & flag, it is just a normal UDP port.
@prjemian can you send the asynTrace output on the status port with asynSetTraceMask=TRACE_ERROR|TRACEIO_DRIVER so we can see the message content? I think you said in a previous message that the status message contains the IP address of the machine sending the message? If so then I think I can add to @norumwe12's list above:
|
|
@norumwe12 said:
I'm developing on a linux mint OS with most recent updates for deployment on RHEL7 at the APS. Might bear to test on the APS network before proceeding with this.
already responded to this
possible |
|
@MarkRivers : the IP address is in the UDP packet header. Can the asynRecord access that content? Here's a few packets with repeating that with The data block is 928 bytes long. Status info is in bytes 4 .. 923. Each datum is 4 bytes long and consists of 2 2-byte integers. First is a parameter code, second is the value. We're looking for parameter code 1028 (0x0404) and either 113 (0x0071) or 144 (0x0090). The controller ID is at the same offset in the data block of each packet. Row 6, 2nd datum: Bytes 104 & 105 must be 0x0404, bytes 106 & 107 must match the desired controller ID number.
That sounds like a plan to implement a filter that should be most reliable. |
|
Summary: |
|
I don't think the asyn driver can access the packet header. But it looks like what you need is the controller ID in the packet. I think you should be able to write an interpose interface that filters messages based on that. You could use asynInterposeFlush.c as a starting point. But rather than overriding the flushIt() method you would override the readIt() method. |
|
Thanks. That's the advice I needed next.
…On Mon, Apr 20, 2020, 3:06 PM Mark Rivers ***@***.***> wrote:
I don't think the asyn driver can access the packet header. But it looks
like what you need is the controller ID in the packet. I think you should
be able to write an interpose interface that filters messages based on
that. You could use asynInterposeFlush.c as a starting point. But rather
than overriding the flushIt() method you would override the readIt() method.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#108 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMFRNKMPZ3IKE64YRFLRNSTLRANCNFSM4MLNKFMQ>
.
|
|
Closed via #109 |
The
SO_REUSEPORTsocket option allows multiple clients on the same user@host to bind to the same UDP port. With this option enabled in asyn, I expect to be able to run multiple IOCs using new support for the Oxford CS800 CryoStream controller (blows cold nitrogen gas at protein sample during crystallography measurement). Currently, when I try to run a second IOC from the same user@host, this error is reported to the console and no communications is the result:Is it possible to call
setsockopt()(similar to what is shown) from thest.cmdfile?Otherwise, can support for the
SO_REUSEPORTsocket option be enabled in asyn? The textSO_REUSEPORTdoes not appear anywhere in the asyn repo so I assume this option is not available yet.The text was updated successfully, but these errors were encountered: