Dropped AVDECC connections

[RichardFoss]

I have a question regarding AVDECC connection management that I'm hoping someone can help me with:

I set up a network with 3 x Netgear switches (using 48 ports and 2 x 24 ports) and 96 XMOS endpoints. I sent CONNECT_RX commands in my own program to connect the Apple 24 channel output streams to the XMOS endpoints. Each endpoint takes one channel from a stream. I found that when I sent the 96 CONNECT_RX commands to the endpoints, only a few of the connections would be made, as indicated in the avbutils connection matrix and as evidenced by the audio streaming. However, if I increased the time between the connection commands to 300 milliseconds, the connections were established and streaming to all endpoints occurred.

I now have these same endpoints connected to Extreme switches, and with my same connection management program, am getting a few dropped connections, as evidenced by the avbutils connection matrix and audio streaming. WireShark indicates that there are CONNECT_TX commands coming back from the endpoints, and also that there are successful TX and RX responses to these commands.

I can see msrp talker advertize, listener and domain messages in the wireshark logs, and presume that these are initiated by the connection requests. However, I can't understand why so much time is needed between the connection requests. It also appears that a successful connection response isn't a reliable indication of a connection being made? I haven't been making use of the sequence ID field in the connection messages, because the commands and responses are occurring as they should. Is there a reason for the sequence ID's beyond picking up a response, and determining to which command it belonged?

Thanks, Richard.

[pinealservo]

Depending on the implementation of talkers, a talker that receives a CONNECT_TX might need to do a MAAP request to request an address, which is why CONNECT_TX has a default 2s timeout. But I don't think that's related to your problem here.

The sequence ID in ACMP is used for matching requests with responses in the request sender; this might become important if you were to repeat a command due to a timeout, and then want to distinguish whether the response came from the first or second transmit of the command.

Getting a successful CONNECT_RX response tells you that the listener and the talker agree that they are meant to establish the stream you asked for; the intention was received and agreed to at both ends. However, it doesn't include the establishment of reservations--it's typically the beginning of kicking off the reservation establishment process, in fact.

When you send all your CONNECT_RX commands tightly bunched together, they're likely to start their SRP reservation process at roughly the same time. SRP is built on MRP, which is based on a periodic transmission model. It's also a far more CPU-intensive protocol than most other similar bridge-to-bridge management protocols, and depending on how all the devices are implementing MRP, you may have got all the devices synchronized to send their MRP commands simultaneously at the same interval. This can overload the management interface on some switches if there's enough MRP traffic hitting all at once and cause its MRP engine to miss cycle deadlines, which can shut down streams or fail to establish them. Staggering the startup by 300ms might be enough to space out the resulting MRP cycles and give the bridge more breathing room to get the commands processed.

This is just a hypothesis; to verify what's going on, you'd have to look at the management interface to the bridge to see what the SRP states on the various ports are. That might give you some clues, but you may need to dig into the details of MRP traffic on the various ports.

Another experiment you might try is making just a subset of the connections as quickly as possible until you reach the point where you start seeing failures. Then scale back a bit and try making the first set, delaying 300ms, and then making another set. Does it still fail? If so, how far do you have to scale back with two groups of the same size before you don't get a failure? Does adjusting the delay change this? Do the two groups sum to significantly more than you can get with all-at-once? I suspect you can come up with a set of spaced groups that will be able to establish the ACMP connections very quickly but still get the SRP traffic spaced out enough to keep the switches happy.

This is certainly not an ideal situation, and these limitations were not intended to be in SRP (if that is indeed what's going on... please verify), but you should be able to work around any limitations to get the network scaling to where you need it to be.

[andrew-elder]

Hi Richard,

Do the XMOS devices have 100 Mbps Ethernet ports on them? I can well image a PC application running a "full" speed on a 1 Gbps port could swap the XMOS 100 Mbps network stack. All the 1722.1 CONNECT packets are broadcast as well, so that is a lot a traffic flying around!

I am surprised that a 300 millisecond delay is required though. I would have expected something much shorter than that to work. Do you implement the inflight queue and retry timeout stuff? I would suggest you do implement that, along with the incrementing sequence ID and then you can easily observe any lost messages.

You could also try doing wireshark on a single connection operation. How long after the 1722.1 transaction does it take for audio to start flowing? BTW there is a 1722.1 controller library here if you are interested https://github.com/audioscience/avdecc-lib.

Good luck!

[RichardFoss]

Thank you very much for these comprehensive comments Levi and Andrew, much appreciated!

Unfortunately the endpoints and Extreme switches are on a remote site, so my analysis has to take place via WireShark logs. I have filtered for AVDECC and mrp-msrp messages and found some interesting timings.

I connect four 24 channel streams provided by Apple's virtual entity to the various endpoints. One 24 channel stream feeds 24 endpoints, each endpoint extracting a channel. After the first CONNECT_RX command, a Talker Advertise message is sent by the Apple. Following this, there are two Listener messages with the Asking Failed declaration type from an Extreme switch and only after about 400 milliseconds is there a listener message from the switch with a Ready declaration type. At this point a second CONNECT_RX has been sent, and predictably, many listener messages with the Ready Failed declaration type follow.

When a Talker advertise for the second stream is sent, the ensuing listener message with the appropriate stream ID and Ready declaration type only appears after about 800 milliseconds. For the third stream it is back to about 400 milliseconds, so this is not a progressive time lag.

So it appears that even at 300 millisecond intervals, I am sending connect messages at a rate that is faster than the complete stream connection mechanism. I am going to try a larger interval first, and then try the subset approach that you suggest Levi.

Andrew, I am using the XMOS xCORE-200 chips in the endpoints, and they have gigabit capability. At this point I am thinking that the timing issue is within the switches. The 300 millisecond interval worked well when I was using 3 Netgear switches. Now there are four Extreme switches and I assume that worst case each of four switches has to go through the resource determination process. Regarding inflight queue and retry, do you mean that I just capture the connect responses, and if the response is problematic, retry the connect request for the message with the appropriate sequence ID?

Thanks for the library reference - yes have referred to it many times. For this application I'm running a node.js server and sending the messages via a pcap module.

Thanks again for the help!

[andrew-elder]

Four streams should not be too many at all from a switch point of view. After the first stream is connected all the endpoints should observe the Talker Advertise and should be able to immediately declare Listener Ready after a CONNECT_RX command. The fact that they don't declare Listener Ready makes me think that perhaps the endpoints are not caching the Talker Advertise attributes for all streams. If that is the case, they would probably have to wait some number of milliseconds for the switch to re-declare the Talker Advertise before they could declare Listener Ready.

I have also observed some very slow transitions of Listener Asking Failed to Listener Ready through an Extreme switch. I have wireshark traces that show up to 5 seconds for the transition to occur. However the stream does start successfully in the end.

See Figure 8.2—ACMP Controller state machine (pg 279) of IEEE 1722.1-2013 for information about ACMP timeouts and retrying "lost" commands.

[RichardFoss]

Andrew, looking at the first CONNECT_RX command I send to an endpoint - what I see in WireShark is the CONNECT_RX command from the Apple (my program), a CONNECT_TX command back from the endpoint, a CONNECT_TX_RESPONSE sent back by the Apple (not from my program, but I assume from the avbutils virtual entity - the Listener GUID is the endpoint), then a CONNECT_RX_RESPONSE sent by the endpoint. This same scenario plays out for each of the 96 connections I make.

After the second CONNECT_RX to the second endpoint, I get a listener ready from the switch after about 200 milliseconds. After the third CONNECT_RX to the third endpoint, the listener ready from the switch appears after about 20 milliseconds - so following your expectation. This is followed about 10 milliseconds later by a listener ready failed. There were two endpoint MAAP ANNOUNCE messages between the two listener messages. From then on, there are only listener ready failed messages until the next stream connection. There we have a couple of listener ready failed messages related to the first stream ID, followed (800 milliseconds after the Talker Advertise) by a listener ready relating to the second stream.

What is noticeable about the WireShark log is the large number of MAAP announce and discovery protocol entity available messages coming from the many endpoints. Given your expectation of a fast listener ready response, and seeing it actually happen in one case, I'm now wondering about the usefulness of increasing the connection request interval. Wondering if the issue is elsewhere.

I will have a look at Figure 8.2 - thanks!

[pinealservo]

SRP can be a difficult protocol to figure out, since it doesn't work like a standard request/response protocol. A while back, I wrote with the Avnu SWAPI subgroup a tutorial document that might help get you up-to-speed on the details: http://avnu.org/wp-content/uploads/2014/05/AVnu_Stream-Reservation-Protocol-v1.pdf

Even with that, it may take some head-scratching and poring over Wireshark traces to figure it out. Feel free to ask questions if anything isn't clear.

Some of the delays you see with SRP messages are probably due to implementations following the rate limiting behavior required by the specification. There are two options for rate limiting: one waits for a periodic transmit opportunity before sending anything; the other, which is only valid for point-to-point links (which applies to ethernet bridging), is to limit transmission to 3 opportunities per 300ms window on a link. Either way, it's normal to see delays of hundreds of milliseconds related to SRP.

Most of my XMOS experience is with their older audio board which used a different version of their AVB stack and was limited to 100Mb. I haven't exercised their AVDECC code very much either, so I won't be a whole lot of help on its quirks, although I could check the version of the code available on github to see what it's programmed to do.

I'm not really sure why some implementations decide to declare a Listener Failed as soon as they get an instruction to connect to a stream; from my experience, it's more straightforward to wait for the Talker Advertise registration before declaring Listener Ready. I'm pretty sure I had a discussion about this with Andrew or someone a while ago, but I don't recall what the outcome was aside from it apparently not changing my mind about it.

What you may need to do is figure out how to monitor the SRP reservation status in your program and use that as a further indication of what state the stream bring-up is in. I would also monitor the PTP status to ensure you're not having intermittent problems there. For the Extreme switches, you might also see if you can have someone hook up a serial console that you can run status commands on to see what their internal state is for each port--this can be very helpful to understand why they're sending what they're sending.

[RichardFoss]

Thanks Levi, I had previously downloaded your SRP paper - very useful!

Thanks also for rate limiting information and the suggestions relating to monitoring. I do have an Extreme switch in my office, so I can view its port state changes in a smaller network.

Just to confound the issue, the guys on the remote site have restarted avbutils, and now all the endpoints are connecting via my program. As a first step now, I intend learning more about the working of avbutils and how it will respond when there is another controller (my program) making connections on the same machine. If anyone can help in this regard, I would much appreciate it.

It would be great to get further information about avbutils, for example why only particular configurations of channels can be used to stream audio.