AIR Runtime on macOS corrupts SecureRTMPChannel #2625
Comments
|
@ajwfrost Is there a particular reason this serious bug report has received zero interaction in almost one month's time? |
|
Hi Apologies for the delay, we did see this one but there have been a lot of other activities going on; there is quite an effort to set up a reproduction for this issue so it's not yet been investigated. If you're able to provide any sample code/SWF file that we could use to run this on a mac and to make it easier to reproduce, it would greatly help. thanks Andrew |
|
Hello @ajwfrost, Thank you very much for your reply. Please help us understand what you have, and what you are missing, so we can help fill in the gaps so you can quickly recreate this. When a fresh build of the AIR Runtime is compiled, are there regression tests of the RTMP and RTMPS communication protocols that must pass before release? If so, I am wondering, do these tests actually connect out and transfer data as part of the test? In our tests, almost everything fails, which is why I was hoping this would be easy to recreate with only the bare minimum of existing tests that you might already have at your disposal. I think the fix for the issue will be in the low level RTMPS communication routines of the AIR Runtime, in an area of the code which is specific to macOS, because the application runs perfectly on Windows. RTMP works perfectly on macOS, but RTMPS hangs almost immediately, sometimes not even completing the handshake that bootstraps the encryption by negotiating the cipher suite and exchanging keys. Therefore, the problem must be in the area of encryption and decryption, since RTMPS is just the encrypted version of RTMP. In order to actually use RTMPS the way we do, you would need a server that speaks the protocol, and the SWC that activates the RTMPS protocol in the AIR Runtime. Do you have a preference in this regard? In our case, this layer is the Adobe LiveCycle Data Services (LCDS) product, and of course, we only have the binary version. Do you have access to the Adobe LCDS product, and if so, do you use it when regression testing the AIR Runtime? Adobe LCDS was distributed with sample applications. I haven't tried those yet, but I assume that those will be a good starting point for recreating the issue quickly, if you can use LCDS on your end. We are ready to assist once we understand what you need to proceed. Thank you, Matthew |
|
Hi
This is our main problem... we do not have any instance of LiveCycle or AMS which means that we aren't able to run the RTMP based testing. It's an area we've not really touched and it's not actually widely used now.. I notice that you've mentioned the issue also existed in the 32.0 runtime, which at least makes sense in terms of it not being a regression that we'd inadvertently introduced. Given your descriptions above, probably the best thing is if I can talk to Adobe and see if they can support us with LiveCycle, or if this is also something we could test with AMS then we can talk with Veriskope.. Although it might actually be that the quickest/best option is for us to work directly with you for debugging, if you have the environment and reproduction already set up. Your analysis has highlighted the particular area of code where we'd need to do the digging so this could be an option perhaps? If we sent you a build of the macOS runtime that output a lot of logging for this, we may be able to pinpoint the issue or progressively narrow it down to fix it..? thanks Andrew |
|
Yes, we'd be happy to run the debug build here and send you the logs produced. |
|
@ajwfrost I have more information for you. There are actually two bugs. The client side is falling back to RTMPT because of a second Adobe server side bug that prevents RTMPS from connecting. It is the RTMPT fallback strategy which does not work well on the macOS side, while the Windows platform falls back to RTMPT successfully. |
|
Okay thanks -> curious as I'm not 100% sure what code there is platform-specific. We'll look into it and try to get you a debug build of the macOS runtime shortly.. |
|
@ajwfrost It has been close to 6 months since your last message about getting us a debug build. We were looking forward to using your debug build of AIR for macOS to assist you with debugging. As the macOS operating system continues to advance, the AIR runtime for macOS seems to be getting worse as time goes on. RTMP and RTMPS connections are becoming unstable and practically usable. Is it intended that these protocols will be maintained? |
|
@busitech apologies for that! This one had completely fallen off our radar I'm afraid, do please feel free to chase up on things. From our perspective we would have hoped that these protocols would continue to work as-is, so if there are changes in the OS that are impacting them, then I'd hope we can work around them. The other option we'd be looking at is to pull the RTMP-related functionality out of the private Adobe source code so that it could be implemented in a separate, open source library (and kept up to date with what Veovera are doing). Anyway -> let me kick this activity off again and try to get you something that outputs more details around this area... thanks |
|
Hi @busitech Would you be able to download a test version of the AIR runtime from the below link, which has a fair amount of debug trace added around the RTMP/RTMPS functionality. This may not immediately help, but it should show us whereabouts things are going wrong so that we can add some more targeted debugging in the specific area. thanks |
|
Hello @ajwfrost Thank you for sending the test version of the AIR runtime. It produced the debug trace log shown below: |
|
Thanks for that. So it looks like we're first trying to connect using "native" SSL within the RTMP handling code, but that this fails: Result 8 is a self-signed certificate... we did change some handling here a while ago for people to be able to decide for themselves whether to accept self-signed certs. Looking at the code here though, it originally accepted this (within the RTMPS handling) and we have the same override. So actually, with an SSL connection result of 8, we should have accepted that as connected... Although I think I've also just spotted a separate bit of code that has picked up that response value and causing the socket to shut down... We can review this part again and see if there's an issue with the connection being killed off due to the self-signed certificate on the server. thanks |
|
@ajwfrost We won't be able to work on the problem this issue was opened to solve until this new problem of not being able to connect at all in 50.2.4.0 is resolved. We opened this issue when 50.2.1.1 was still current, and that version would connect just fine but then the connection would soon go down randomly after a few minutes. We are eager to work on that issue, but this new issue in the later build will need to come first since it prevents connection. I might be able to set up a server with a real certificate and possibly get a head start on that issue, if that allows 50.2.4.0 to connect. If it would be easy for you to send a debug version of 50.2.1.1, that would also work. Thank you! |
|
Okay thanks, understood - I did wonder at this because you'd mentioned it was a more sporadic issue and had been around for a while, sorry I'd missed the fact that we'd just made it worse and immediately reproducible! I'll upload an updated version of the latest runtime that allows the SSL connection, and we can then start delving into why it may be dropping after a while. thanks |
|
Hi again |
|
@ajwfrost It looks like no progress was made with today's version. Nevertheless, thank you for giving it a shot so quickly. |
|
In the mean time, I have installed a CA-signed certificate on a server, successfully connected, and was able to reproduce another issue, but I'm not sure if it is a new issue, or our original main issue. What I found easy to reproduce today is an infinite loop and hang when a debugger is connected, such as IntelliJ IDEA, and RemoteObject calls are made. I reproduced this issue when connecting with both RTMPS and RTMP using the AIR Runtime v50.2.4.0. Previously, when using the AIR Runtime v50.2.1.1, we were able to revert to RTMP to avoid our main issue. This makes me wonder if this could be a new issue, or just a really easy way to get the same issue to happen, that now affects both network protocols. In any event, this one is extremely consistent, I can repro in two minutes. I hope what is below helps you get closer to the root cause, and we can test further as needed. After RTMP or RTMPS connection, I see this line in the console being logged continuously, even though no network traffic is occurring (the message implies sending). You will notice how these messages are milliseconds apart: After one or two successful server RemoteObject calls, all debug messages stop being logged, and CPU utilization goes sky high on the adl process. There is apparently an infinite loop in progress now, inside the runtime. The server has just finished responding to a request from the client, and the client received the message, processed it, and updated the UI. The bug is sensitive to timing, because the same number of tries is not always The debugging functions are stopped or hung now, and the app is totally malfunctioning. While in this state, the application is still able to send and receive Command messages (subscribe, unsubscribe) in response to certain UI events, and the server acknowledges, and even one remoting call from every remote object will send, but incoming remote object messages seem to be going into a queue that is no longer processing what is being added to the queue. When the mouse hovers over the grid control, the background color change will follow the mouse as it usually does, so the UI itself is not frozen. The app seems hung with respect to incoming messages. Any UI events which normally fire more outbound network traffic will do so, and the server even responds, as long as the object that sends the message does not have an outstanding request. Once a given remote object has sent a message that expects a response, the response is received from the server, but since the message never gets processed, that remote object becomes hung too. The only thing the debugger can do now is kill the app. Pause doesn't work, and of course breakpoints don't work either. I am having trouble reproducing this if I start adl from the command line, but since that one trace message is constantly being logged, the logging fills up Scout's memory over time, so we don't have a very good shot at using Scout over a long period of time. I turned off some of the unnecessary recording just to get more time, and I'm going to leave a session run overnight and see what happens. Here are the last trace messages that are seen in the console: |
|
Thanks for the quick feedback! The "Sending SWF hash" is actually not sending a hash (the '0' value stops this) so I've moved that so it's only reported if we do actually send a hash. This forms part of the 'idle cycle' i.e. the stuff we check within the runtime's main thread loop which also drives the AS3 events and animation cycle, hence it appearing quite a lot! The issues you're describing sound like something is stuck within the 'read' thread, i.e. we are receiving data, maybe adding it to a queue but not processing that, and maybe then something is in a tight loop... Would it be possible to enable core dumps on your machine, and then whilst it's hanging, kill the process from a terminal? you should be able to click on "reply" on the above message link to be able to then upload a dump file to return so that we can check the call stacks... thanks |
|
You have RTMPS working with this update, thank you! I have a full core dump saved (9.5GB), and I have sent you the backtrace of all threads, by replying to the message. Please let me know if you need more information. |
|
@ajwfrost I believe we have the intermittent issue recreated, and the log messages captured have explained more about what is going on under the covers. The NetConnection logs an "Address changed" event, but the address being connected to during our test was localhost, and this address never changes. In the log, we also notice that the memory address of the NetConnection IO thread and the address of the next "starting read thread" are always the same values each time the same messages are logged, even though there is also a message indicating that the read thread is quitting. I'm not sure if this is interesting or not, but I thought I'd mention it. |
|
Hi Thanks for checking this, apologies for the delay due to the Christmas/New Year period! With that hint about the "address changed", it seems a little odd, because the socket should only be reset if we're no longer able to pick up the remote connection. The callback comes from the OS itself but we should ignore this as long as we're still active - and there's even code there that checks if it's a local/loopback address, in which case it should definitely then continue! So, a little more debug needed I'm afraid .. below is an updated version with some additional debugging around this area. many thanks Andrew |
|
Andrew, I have your updated debugging information: I wanted to clarify that the issue happens to both localhost connections and connections to external addresses, in case there is any doubt about that. Also, just a reminder that we also shared the stack trace which explains how the runtime crashes when the debugger is attached, which is a separate issue. We have to run the adl command from the command line to test with the later versions of the runtime. (Attaching IDEA causes a hang within a few seconds of running the program.) Thank you. |
|
Thanks for this. So the bit where it says We've added a bit more checking of the network situation here and trying to find out exactly why it's dropping out of the loop. And we're not sending out that AS3 event now. Sorry to ask for another attempt, thanks for your time on this so far! The crash with debugger we will look at shortly, that may well just be because of changes we've added in for this debugging session.. thanks |
|
Hi @busitech - just wondering if you'd been able to capture some logs of this going wrong? We've done some testing here, the network changed event happens e.g. if we add a VPN connection, but the original network path still shows as valid (i.e. we've not been able to get it to show It may be an IPv4 vs IPv6 thing, but if you have the logs it would be handy, I would hope this is the final one needed! and I was then hoping we could get a fix in for our next release, which I've been postponing slightly in case we can squeeze this in! thanks |
|
Hi @ajwfrost we went to work immediately on the last release you provided, and we are wondering if one of the changes you made could have resolved the issue, because we haven't seen the issue again since installing that last debug version you provided. |
|
Interesting! I will double-check the diffs, but my recollection was that the only functional change we made there was to swallow that It would still be useful if we could see the log around that "NETWORK CHANGED" message, so that we can double-check how the IPv6 addresses are being handled - I still can't quite see how we would have ended up with thanks |
|
Not only are we not seeing disconnects in this version of the debug runtime, we also haven't seen the messages you're looking for either, which is why we've had nothing to send your way. The interesting log entries are not showing up now, but we're watching for them nonetheless, and will send one if we see it. You might want to send a debug version with more problems and less changes that are acting like fixes, because it is like we've lost our ability to recreate the issue. |
|
Hello @ajwfrost , We have some log data for you! |
|
Good evening @ajwfrost Here are a couple more with the timestamps enabled. We're unsure why the problem seems to be back. |
|
Thanks for those! It seems fairly certain that this code isn't working how we would want it to: This is handling some change in the local network conditions and should be giving us the IP addresses of the available network interfaces, and then checking that the one we were using for our rtmp socket is still avaialble. The problem seems to be that our existing socket has a local address of Anyway - let us check on this aspect, I think we should have enough details to find/fix it now! Many thanks for your help with the debugging :-) |
|
Do you need us to send logs for connections to external hosts as well? We are only debugging with the loopback interface because it is convenient, but do not want to limit the scope of the debugging to only loopback, which is not used in the real world... |
|
I think at the moment we're fine thanks - a quick test here shows that we appear to not pick up all of the inet6 addresses on macOS when we query the network adapters.. I suspect the problem is the same regardless of where the remote end of the socket is connected. |
|
Having said that of course .. would it be possible to get the results of the below command run on the client Mac machine please? It looks like this may come down to a 'feature' in AIR that we don't provide the IP details for link-local network addresses. So it may be that you have something like "::1%lo0" rather than just "::1" which might explain why ::1 wasn't in that list... And when you talk about external hosts, are these external to your company, or would they still be local network addresses? thanks |
|
Yes, we have the issue connecting outside our company to remote servers as well as those inside, and those on our local machines. |
|
Thanks for all this. So, we've found the problem, although I won't pretend that we understand why it's failing! For this check following a network changed event, we are using a macOS method So -> we can switch to using a lower-level method to pick up the network addresses, which seems to work in our test app (using the unix Unfortunately not quite in time for the release which is about to come out, but we'll follow up soon with a macOS update I think as there is another mac-specific fix being investigated currently.. but once the code is updated, I can upload a fix candidate perhaps.. thanks |
|
Are you able to see a pathway by which the problem identified would affect connections to hosts that are not local? If not we will need to start looking for another issue which affects remote servers. |
|
Yes .. essentially, the code that's meant to work out whether the network interface being used for the RTMP connection is still available, uses a function to check what local interfaces are still present/available. It may work okay for IPv4 but for IPv6 for some reason the query is failing to retrieve the full set of interfaces. So any connection, regardless of the actual remote/server location, is likely to be impacted by this. The initial connection works fine, it's just whenever macOS detects any network change, the AIR code then mistakenly thinks that the connection will be cut and so disconnects and reconnects. thanks |
|
That is awesome, thank you very much. |
|
Hi So a fix proposal is below.. This is still outputting a lot of logging for us, just to check! If you're able to capture the logs (actually a lot of I'm hopeful that the connection will just continue and this error will then go away... thanks |
|
In the latest version, the channel still becomes disconnected. |
|
Hmm... so that looks like it's hitting our new code, although the NSLog output isn't coming out there. The only thing from looking through it is to see whether the ::1 network adapter has gone down .. is that a possibility? would you be able to put a monitor onto the local network interface to see whether it drops? Not sure if you open Settings -> Network whether you can just check whether the green light periodically flickers, or there are suggestions in the below link: To check the actual network interfaces that you have, can you please try running the below command-line tool? Firstly just whenever you're about to start the AIR app, and then perhaps just after it's failed? If you run this from a terminal window, you should see the log listing all the interfaces; if "::1" isn't in the final list at the end, then please check the earlier output log to see what it says re the "lo0" interface.. thanks |
|
Your utility was not built for our Intel CPUs, so it does not run here. We have seen dozens of disconnects while simultaneously sending ICMP packets to the localhost interface every 500ms with zero packets lost. Granted, 500ms is "forever" from the perspective of an operating system kernel, but at least we have proven that nothing humanly perceptible is happening on the interface. |
|
Hi Apologies, I hadn't spotted that I needed to switch the build type there... But I think that actually, this may not have much value, given you're seeing the interface when you run the 'ifconfig' command, and with your above testing which shows that it must be a very transient issue, if this is the case that the status switched briefly. Probably what we need to do is:
Will try to get a build out to you for checking this within the next day or two.. thanks |
|
We have started doing tests that do not involve a local interface, and instead connects to a remote server via the en0 interface. We saw a disconnect this evening while the same client computer running the AIR app was also connected to the very same server via ssh, which did not disconnect. We were also sending ICMP packets every 400ms (reduced from 500ms) with zero packets lost across the disconnect of AIR. Below is the log. There were no address change messages captured in this log. We did see the Therefore, we have learned via this test that disconnects do occur even when they are not triggered by address change events. |
|
Andrew, have you been able to make a build which can process the "network changed" events appropriately? Since we last spoke, our research has discovered some very helpful information. A recent update to macOS has introduced a new architecture for communicating with other Apple devices. macOS is now constantly creating and tearing down utun (user space network tunnel) interfaces via background processes. Some of these processes are named remotepairingd and remoted. The setup and tear down of these utun interfaces cause many new "network changed" events to be sent to all applications monitoring the status of interfaces. The macOS console logs reveal copious amounts of details surrounding these events. We really need an AIR Runtime Framework that handles these events correctly. There are times when the application can not stay open for more than a minute or two without becoming disconnected. |
|
Hello @ajwfrost, Please let us know how we can proceed and what to expect. Even a temporary workaround that disables the network change handler would help. Thank you. |
|
Hi @busitech - thanks for the information above about the interface changes, we've not yet had a chance to look into this.. the expectation though (I would have hoped!) is that the functions we use to get the current information about available interfaces would still show the one that's being used for your RTMPS connection, so we shouldn't then disconnect... So I'm slightly concerned that we may be trying to fix something that then turns out not to be the underlying problem! But, your suggestion of a temporary workaround to disable the network change handler is a good one, and fairly easily implemented. I'll try to get a test runtime version created for this and upload it here... thanks |
|
This issue has been open for 10 months. If the resources are not available to finish the work, please open the source code so we may maintain it. The disconnects every 60 seconds are unbearable. |
|
Hi - sadly we're not able to open up the source code, it's not really ours and the license terms are fairly strict! But I had meant to ask, could you please check whether the AIR 51 release is more stable if you set the We've got a documentation update pending, as this was a last-minute addition, but you should just be able to do: and then we won't start interrupting the sockets with all the events that are happening on macOS. Whether this is then going to help or not, depends on whether those network interfaces are actually being interrupted by the OS. It's still not clear to us why the machines are not reporting any 'localhost' availability when this is happening for you, we can get the network changed event (by starting/stopping VPN) but always see the localhost address available... thanks Andrew |
|
Okay, we'll try the new setting, and provide you with the results. On November 29th, you had suggested the possibility of separating the RTMP feature out into an open source library, and I have been wondering if you ever spoke to Adobe about this. I highly doubt there are any interruptions at the OS level. I am able to keep connections open for weeks using other protocols and applications. Avoid focusing too much attention on localhost, because the same issue happens on every network interface we connect out with. |
|
Turning on disableNetworkChanges has circumvented the disconnect problem. We opened a connection on Friday, and on Monday it was still connected and usable. Have you updated your development environment to the latest Sonoma release to make sure you have the same operating system we do for testing this issue? This should be giving you very frequent disconnects without having to cycle a VPN connection. |
|
That's good to hear! I will double-check, I thought our systems were up to date. But we weren't seeing that same issue with the disconnections. We're just using wifi, in case you're using an ethernet cable for this, maybe that makes a difference? thanks |
|
@ajwfrost When will a permanent fix for this issue be seen the AIR runtime? The permanent fix should be one that does not require special options to be set in our source code, or a special build of the AIR application to be released. We are expecting that end users would simply be able to update their AIR runtime in order to access this fix for macOS, while using presently deployed versions of our AIR applications. We do not intend to make any permanent changes to our development environment or to have to recompile our various AIR applications in order for users to obtain a stable environment for their existing AIR applications in later macOS releases. |
|
Hi ** probably you can skip all this text and just jump to the 'update' section below.. but I'd written it and don't really want to delete it... ** The issue for us is that we don't see the same behaviour as your systems are showing. You mentioned earlier:
We don't see this though, we are not getting any spurious "network changed" events. We've checked on two different macs which have the latest software on them; the only things I am wondering that may be different are (a) are you using ethernet cables? as we are only on wifi; and (b) we are using arm64-based devices rather than intel.. But, it was also clear from the earlier testing/profiling that when your systems are sending this "network changed" event, the list of available network interfaces did not include the one that was being used for the RTMP connection, which is why there was then a manual tear-down of the socket. I don't really like changing the existing behaviour of the runtime without having some opt-in or opt-out type mechanism, but we could look perhaps at the RTMP code ignoring these network changes and just relying on the underlying OS socket implementation to let us know if there's an issue. Based on your findings with that new AS3 flag, this is all that would be needed. The risk of course is that someone else might be relying on the early detection of a changed network in order to rapidly respond/reconnect. So it's really just a case of two options:
Of course, it would be much nicer if the OS didn't send these notifications or didn't miss out the required network information when we queried it, and if we had a way to reproduce it here then we would be more likely to see if there's a better workaround we could do (such as perhaps delaying a few milliseconds before checking the list of network interfaces - since from what you say, and the behaviour of other apps, this is just a glitch that shouldn't actually impact a socket connection). UpdateJust doing a little more searching to see if we could find any reasons behind these messages, and we found that the same problem had been caused in Chrome. After a lot of analysis, they had essentially added an "ignore network change notifications" but in a limited/targeted way. So, we should be able to do the same... Seems like the issue causing these notifications is related to the wider use of Apple connected devices on the local network, hence us not reproducing it here in the dev environment. Let me get someone to inspect the changes that the Chromium team did, and try to do the same thing as default in AIR... thanks |
|
@ajwfrost It is incorrect to assume that macOS is broken, and that no sense can be made of the events you receive. These two assumptions are leading you to the false dichotomy of ignore vs. do not ignore. The "ignore" concept leads to a serious design flaw, the one implemented by Chromium. My belief is that after working with Apple to understand their changes, such as by opening a case with Apple Developer Support and discussing the APIs you're using to monitor the interfaces, and the events received as a result of the utun interface changes, you will be able to achieve the original functionality without ignoring events, while fixing the bug we have reported at the same time. The bug is tearing down perfectly good connections. The most important next step on your end is to do whatever it takes to reproduce the issue, and capture full details in your debugger. I would imagine that the interface being changed is part of the event. All we have in the office for Apple products are two iPhones on WiFi and two Intel iMacs on Ethernet. If in the end, you believe that Apple has introduced a bug in the APIs you are using, please, for the sake of all mankind, report the issue so they can fix it! : ) |
The AIR Runtime corrupts SecureRTMPChannel connections on macOS, but not on Windows. Our application makes RemoteObject calls to the server over this channel, which begins to hang forever on macOS after a few transactions. The initial connection is usually successful, and sometimes even a few successful remote calls can be made before the hang occurs.
Compiler: AIR SDK 32.0.0.116
AIR Runtime: 32.0.0.116 to 50.2.1.1
Platform: macOS Ventura 13.3.1
The text was updated successfully, but these errors were encountered: