[android] cannot open websocket when using mobile data

[gsgou]

Describe the bug
Can connect using wifi data but not using mobile data (API: StartContinuousRecognitionAsync()).
Checked that the app has permission to use mobile data.
android.permission.INTERNET is granted.
This bug was introduced a couple days ago (7-14 days) though the package and the logic was not updated.
Perhaps something broke it in the backend or maybe my phone operator..

Expected behavior
User should connect when using mobile data.

Version of the Cognitive Services Speech SDK
Microsoft.CognitiveServices.Speech v1.18.0

Platform, Operating System, and Programming Language

• OS: Android 10
• Hardware - ARM, Pocophone F1
• Programming language: C#

Additional context
ErrorCode: ConnectionFailure
ErrorDetails: Connection failed (no connection to the remote host). Internal error: 1. Error details: Code: 101. SessionId: dd0300bb8eb8abf4b9ebee28bbea865b

[gsgou]

setting Proxy had the same behaviour:
speechConfig.SetProxy("178.63.17.151", 3128);
https://fineproxy.de/kostenlose-proxy-server-liste/#proxy-list-table

ErrorCode: ConnectionFailure
ErrorDetails: Connection failed (no connection to the remote host). Internal error: 1. Error details: Code: 101. SessionId: de5c2295e7917744be272fa336e72d9d

[gsgou]

@rhurey i was able to set PropertyId.Speech_LogFilename and get some log data for review.
speechsdklog.txt

I appreciate any helpful Feedback. I can't publish a mobile app with this issue open.

[Abdull]

I can reproduce this bug with my Android device.

[brandom-msft]

@gsgou Thanks for raising the issue and also for including the log. I'll raise this with the team and reply back when I have more.

[gsgou]

Thank you @brandom-msft for your Feedback. I am gonna have soon some Android beta users.
Writing code now to collect the extensive logs from users with this issue and will provide
additional info here when i have more data available.

[brandom-msft]

@gsgou Please do send any additional logging when you have it.

It's puzzling that this does work on wifi but then fails on mobile. Have you verified the internet connection is working on mobile for the customers that see this?

Alternatively wondering if it is due to a permissions issue. Some mobile settings have permissions similar to "allow connection over wifi but not cellular", curious if there's something like that going on. This article has details (while the Android version may differ the idea would still apply.)

[gsgou]

@brandom-msft i went through every single option because i can reproduce it in my phone every single time. Its even more puzzling that the same code used to Work before 14 days and i Had No OS Updates.
Only could have changed is the Server or adjustnents from my phone operator.

[rhurey]

The error code 101 maps to "Network not reachable" from the underlying platform.

Shot in the dark: The cell network has IPV4 support right? The Speech SDK is IPV4 only.

[gsgou]

@rhurey, @brandom-msft tks for the Feedback, the cell network supports IPv4.

I had a flight to Spain today and since a different network is used I tested again on my device and the service works fine here.

So If there hasnt been Backend Updates then indeed could be related to my cell network as the connection stopped working at some point of time without recovering since then. I all also try to use the sim card with an iOS device or another android device. Telekom network in Germany is very popular.

Maybe the logs can help further. Beginning of next week i ll publish a Beta on few users and maybe will provide me some further logs. I ll collect also the operator name in that case.

[gsgou]

@rhurey, @brandom-msft since the same configuration works in Spain, when i am in Germany i ll test with http://websocketstest.com

Might be related to carriers setting up broken transparent proxies according to these posts:
https://stackoverflow.com/questions/5557776
https://stackoverflow.com/questions/53219350

[rhurey]

Thanks for the info. I've also reached out to one of our teams in Germany to see if anyone from there has the same provider and can try to see what results they get.

Also, can you try to see if the JavaScript based demo here works since it will use a different websocket layer to connect (Namely the one the browser provides.)

[gsgou]

@rhurey I can test the Javascript when i am back in Germany first week of October or earlier in case another of our testers can reproduce it.

For completness my mobile network carrier is from Telekom on 4G while the operator is named Congstar.

[gsgou]

@rhurey the same error is still occuring on my operator and location.

We have very few test-users sofar. We could test that on another Android device with congstar. The error did not occure. Maybe a different network configuration as the tester was in another remote german city located.

I tested the Javascript Demo on my device and with chromes websocket layer everything works fine.

[gsgou]

@rhurey we did a soft-launch of our project using the speech-sdk. I ll provide here more logs if some users reproduce it and send us their logs. Has the team being able to reproduce it already?

[gsgou]

@rhurey was the team able to reproduce it? We are growing slow in the stores but as soon as more logs are there or users that denied send them i ll share more info here. The project can be found as Scribbn and uses the SpeechSDK: https://www.scribbn.de/

[rhurey]

No, we haven't been. I've asked for some more clarity on what's been tried.

Did you have any luck with the http://websocketstest.com site?

[gsgou]

@rhurey websocketstest.com and the azure java script demo website work well with the browser version shared above.
On the same network in my region the SDK errors with 110. In roaming in different cities in Spain the SDK succeeds.

[rhurey]

Let me just re-cap this all to try and make sure I've got everything.

Works:
Connections to the Speech Service from the Azure demo in a browser on Telekom mobile.
Connections from the app on a different operators network.
Connections from the app on Telekom mobile if we could time travel backwards a few months.

Doesn't work:
Connections from the app on Telekom mobile today.

Does the app make any other network connections that do work on Telekom mobile? I'm trying to differentiate a general network issue vs a websocket issue.

I'd looked into the SDK log above with the hope that maybe adding more logging around web socket upgrade would be worthwhile, but it looks like opening the raw socket is failing.

[gsgou]

@rhurey correct all your points.
The Mobile Virtual Network Operator is Congstar and the carrier Telekom.
The issue persists also with v1.19.0 and was introduced without app related
changes in the middle of August.

It is explicitly a websocket issue. At the same time the audio is transcribed
the app is uploading succesfully the audio file over https to our servers for
further AI operations. Android Connectivity API shows also that there is
internet: https://github.com/xamarin/Essentials/blob/main/Xamarin.Essentials/Connectivity/Connectivity.android.cs

Attached you ll find the logs from the same device with v1.19.0 in case something changed.
I will provide more logs when available but i don't expect to be different.
speechsdklog_v1.19.0.txt

[rhurey]

I'm not sure it's websocket as much as the way the Speech SDK has implemented networking. At it's base the SDK uses a C layer to handle network connections, and that layer is failing to open a socket to the remote hose with an error of "The network is unreachable" (Errno=101). This is before attempting a websocket upgrade.

Since the same app is making other http calls successfully (from the same device I assume), and the app (on the same device) works with a different carrier, I've been trying to see if a carrier can restrict socket access from C independently of higher level API's, but haven't found anything yet to confirm or refute.

[gsgou]

Thanks @rhurey for your effort. If you need me to run some beta nuget for additional diagnostics or provide remote access for debugging purposes to your team simply ping me. In the meantime i ll try to contact the operator in case they provide some feedback.

[gsgou]

@rhurey , @pankopon we are seeing the error appearing 7 times in 200 users. We can't promote yet the project as we could experience a lot of negative reviews. Also seen once in iOS. Our users tend not to send the logs collected including used mobile operator but it is the known 101.

Is it possible to extend in the meantime the Speech-to-text REST API for short audio to more then 60 sec?
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/rest-speech-to-text#speech-to-text-rest-api-for-short-audio

We would have to consider internal a deadline to switch in another transcription service if there is no effort planned on this issue. I understand the complexity but i have not seen any resources allocated on this. If we can help in testing feel free to contact us.

[dargilco]

Hi @gsgou, this is Darren from the Speech SDK team (@rhurey and @pankopon are my colleagues). I can reach out again to our sister team in Munich to ask if anyone is using Congstar on Telekom. If we have folks, will you be able to onboard them to your app preview program? It seems to me the only way to make progress is to have one of our engineers in Munich reproduce this first with your app, then we can take it from there and iterate with them to try out some test code to further diagnose the issue.

I doubt there is a way to increase the 60-seconds limit for a particular subscription, but I can ask and get back to you.

Darren

[dargilco]

@gsgou we have a colleague in Munich who likely has mobile access to Congstar & Telekom . Can you please e-mail me (dcohen at microsoft dot com) so we can loop him in to self-host your app?
Thanks,
Darren

[fmegen]

@gsgou would it be possible for you to lookup the IP addresses of your mobile while on the operator network and validate that it indeed a public ip address and not some private ip address that requires an operator proxy to reach the internet.

private ip address ranges:
10.0.0.0 - 10.255.255.255
172.16.0.0 - 172.31.255.255
192.168.0.0 - 192.168.255.255

thanks a lot!

[gsgou]

@fmegen it will take more then a month as i am in Spain now Sith roaming and everything works fine but i ll remember to do it and report. I have tried setting a non operator Proxy and the issue was showing again while other Apps where connected with the same proxy.
speechConfig.SetProxy("178.63.17.151", 3128);

[gsgou]

@gsgou we have a colleague in Munich who likely has mobile access to Congstar & Telekom . Can you please e-mail me (dcohen at microsoft dot com) so we can loop him in to self-host your app?
Thanks,
Darren

Email sent. Thanks for allocating resources and time on this Darren.

[dargilco]

We are following up directly with @gsgou over e-mail. We'll update this GitHub issue if/when there is any resolution.

[fmegen]

@gsgou thank you for the information.

[glecaros]

Closing as communication was moved to email