Error when holding call: Unable to getUserMedia: getUserMedia(): TypeError, constraints requests no media types

[eduardothiesen]

Describe the bug
On previous versions calling call.hold() was working, now in version 1.0.0 when I try to hold a call I receive the following

Unable to getUserMedia: getUserMedia(): TypeError, constraints requests no media types

exception = {InvalidStateError}
_stackTrace = null
code = 2
name = "INVALID_STATE_ERROR"
parameter = null
value = null
message = "Invalid status: getUserMedia() failed"
status = "getUserMedia() failed"

but, different from answer and call methods, the hold method does not have parameters to pass the user media.

System Infomation()
Flutter 3.24.1 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 5874a72aa4 (2 weeks ago) • 2024-08-20 16:46:00 -0500
Engine • revision c9b9d5780d
Tools • Dart 3.5.1 • DevTools 2.37.2
Target OS and Version: Android 14

Happening also in the example project

[Volsavr]

The same behavior in our app after upgrade to v1.0.0. Seems 'hold operation' and 'renegotiation after network switch' do not work (at least on android). Looks like it happened after "Upgrade to video call implementation" (#462), but it requires further investigation...

[mikaelwills]

Upgrade to video call implementation was mine, i'll have a look into this!

[mikaelwills]

So the issue seems to be that no media constraints or rtcOfferConstraints are passed in any options maps when hold or undhold is performed.

So in rtc_session hold() when it calls sendReInvite I added the following in the map:


_sendReinvite(<String, dynamic>{
        'rtcOfferConstraints': <String, dynamic>{
          'mandatory': <String, dynamic>{
            'OfferToReceiveAudio': false,
          },
          'optional': <dynamic>[],
        },
        'mediaConstraints': <String, dynamic>{'audio': false},
        'eventHandlers': handlers,
        'extraHeaders': options['extraHeaders']
      });

getUserMedia() then successfully got 0 tracks.

In the resulting invite to the PBX audio was a=sendOnly and the other client started receiving hold music.
So this is as expected and things are fine here.

Although I faced issues when performing unhold()

In rtc_session unhold() when it calls _sendReinvite() I added into the options map:



_sendReinvite(<String, dynamic>{
        'rtcOfferConstraints': <String, dynamic>{
          'mandatory': <String, dynamic>{
            'OfferToReceiveAudio': true,
          },
          'optional': <dynamic>[],
        },
        'mediaConstraints': <String, dynamic>{'audio': true},
        'media'
            'eventHandlers': handlers,
        'extraHeaders': options['extraHeaders']
      });

The local stream created from getUserMedia() would contain one audio track
but in the following code when adding that audio track to the connection, the print out after adding to check whats in the connection still showed 0. I dont understand why adding the track didnt add the track.

 localStream.getTracks().forEach((MediaStreamTrack track) async {
            if (track.kind == 'video' && hasVideo) {
              await _connection!.addTrack(track, localStream);
            }
            if (track.kind == 'audio') {
              print('adding audio track to connection');
              await _connection!.addTrack(track, localStream);
              print(
                  'tracks in connection after adding: ${_connection!.getLocalStreams().first?.getAudioTracks().length}');
              print(
                  'connection overall tracks count: ${_connection!.getLocalStreams().length}');
            }



Now further down when the localDescription is made to get the SDP to send off… because there are no local audio stream tracks the resulting SDP audio a=sendOnly still… when it should be a=sendRecv

My understanding that in _createLocalDescription() when you call _connection!.createOffer(constraints) if you say in the constraints that you want audio but your connection contains no audio tracks the resulting SDP will still say audio a=sendOnly.

The SDP created looks at what tracks are currently in the connection and what your constraints say?

Currently I can’t figure out why the tracks aren’t being added to the stream when I call _connection!.addTrack(track,localStream);

[dmagic99]

Any solution? I also facing the same.

[mikaelwills]

Further progress with this but not quite finished yet.

What i was doing wrong in my last comment above is when you put a call on hold, i believe you shouldnt be remove or adding any video or audio streams.

So don't mess with any mediaConstraints you only change the rtcOfferConstraints to OfferToReceiveAudio: false for hold and true for unhold that gets passed into _createLocalDescription() so an SDP comes out with the audio tag a=sendOnly (for hold) and then a=sendRecv (for undhold), again leaving the streams in the connection untouched. Id would be great if someone with better understanding than me could confirm this.

The initial error of:
Unable to getUserMedia: getUserMedia(): TypeError, constraints requests no media types

Is called because in _sendRevinite() when a hold is called its trying to get new audio or video streams when I believe it shouldnt be.. if its a hold or unhold, like i said above we shouldnt alter the streams in the connection.

So in my latest work if put into the options when making an un hold or hold 'holdUnholdRequest': true

if this is true inside the _sendReinvite() it skips over getuserMedia and adding anything to the streams completely and just goes to _createLocalDescription() and sends off the reinvite.
With this ive noticed hold works a lot more smoothly than in my last investigation.

But i still have an issue when coming to unhold.
Unhold sort of works but im noticing the iceConnectionState changes to Disconnected, triggering 1645 of rtc_session.dart
which calls _iceRestart() which sends another renegotiate with no offer constraints or any mediaConstraints.

Now we come back into _sendReinvite() without my holdUnHoldRequest true
So it goes to getUserMedia but i has no mediaConstraints so we're back to the original error of TypeError, constraints requests no media types.

So in sendReinvite ive said the rtcOfferConstraints to the files _rtcOfferConstraints so that iceRestart picks them up.
And also in iceRestart i added mediaConstraints of audio: true and checked the connection for video tracks to see if its video, if so video: true as well.

This seems to get further with unhold but somewhere it goes around again sending another invite and setRemoteDescription fails because the state is stable and its unnecessarily trying to do another invite..

Will do further investigation soon.

[eduardothiesen]

Any news on this?

[mikaelwills]

Life and work got super busy there for a second but work has me back on our sip dev side of things. Spending all day today and tomorrow and possibly next week on it. Should be able to iron this out.

[eduardothiesen]

Hey @mikaelwills thanks for the update, totally understand how things can get busy! Really appreciate you taking the time to work on this, hopefully it won't take too much of your time 😅

[mikaelwills]

Ha no worries, It's still my responsibility to fix it since i broke it.

So far with todays work from it.. my plan is to step back from all of the above.
Im tempted to rename my existing sendReInvite() to sendVideoUpgradeReInvite()

Then get the old sendReinvite() back from before my video upgrade commit.

This way hold will work just the way it did before and i could do any video upgrade in the new function seperating any concern.

With this hold is working perfectly fine now.
EDIT spoke a little too soon on this line, in 3cx it seemed to work but testing on FreePBX it wont unhold.

Edit hold is officially back , just ammending my video upgrade

[mikaelwills]

@eduardothiesen submitting the PR today!

[eduardothiesen]

@mikaelwills Thanks, brother!

[mikaelwills]

@eduardothiesen Ah few more things to smooth out but hopefully will be up son