Add ability to switch between voice & video in native matrix 1:1 calls

[nadonomy]

Your use case

What would you like to do?

When you make a call in Element, if it's a call between 2 people we use native Matrix calls. However, once the place is called, if you want to switch between voice or video you need to hang up and place a new call. It should be possible to switch between voice & video after placing a call.

How would you like to achieve it?

@SimonBrandner has offered interest in developing this on Web, pending us working on designs and gaining confidence we want the feature.

@jakewbate it'd be great if you can figure out the details on this. The things which spring to me are:

1. Any solution shouldn't break compatibility with other clients, as we can't guarantee concurrent resources on iOS & Android. Worth leaning on expertise from @dbkr to see the feasibility of this, or if we need to take a cross-platform approach..
2. We need to figure out when you can toggle between voice/video. Is it while the call is ringing? Or only after the other participant has answered?
3. We need to figure out call tiles. Could they update in real-time in the timeline if they change? Once a call finishes, should they just persist as their last known state?

Have you considered any alternatives?

No response

Additional context

No response

[SimonBrandner]

@jakewbate, feel free to DM me directly (@simon.brandner:envs.net)! I can provide any necessary info about this from the code point of view

[jakewb-b]

Based on my discussion with @SimonBrandner and @dbkr, we propose the following solution:

• A video call can already be 'downgraded' to a voice call essentially just by turning off camera, so there is no urgent requirement for any change. (This does not force the other user to also turn their camera off, but I don't think this is a greatly needed use-case, and arguably not even a desirable one)
• A voice call is essentially just a video call just with both cameras turned off. Given that, this means that it can be easily 'upgraded' to a video call by simply turning the camera on. This requires very little extra functionality or UI.
• Therefore, on Element web, using the 'voice call' button (telephone icon) would (from the user's POV) be the same as starting a video call but simply with both cameras defaulting to off.
• As the user placing a voice call, I can choose to turn my camera on while the call is ringing, but the other party's camera will still default to off. They can turn it on as soon as the call starts if they wish.
• The video call (camera icon) remains in place, and starts a video call with both cameras defaulting to on.
• Notifications will remain specific to the type of call, so the recipient knows what to expect when they answer.
• The current tiles will also remain the same, and simply reflect the type of call that was originally placed, regardless of whether cameras were turned on or off part-way through.
• For cross-client compatibility, we can simply disable the 'turn camera on' icon when receiving a voice call from a client where this is not implemented.

Ideally, this same solution will be adopted on other platforms, but this is not critical.

I will work with @gaelledel next week to review the UX and confirm any UI changes. They should be very minimal.

[SimonBrandner]

Therefore, on Element web, using the 'voice call' button (telephone icon) would in fact start a video call but simply with both cameras defaulting to off.

To be clear, from the WebRTC perspective we will still start a voice call - audio track only. But that doesn't concern the UI. From the UI perspective it will appear as what is suggested above

[nadonomy]

Therefore, on Element web, using the 'voice call' button (telephone icon) would in fact start a video call but simply with both cameras defaulting to off.

The tile could change to a more generic icon and just the text 'Call', rather than 'video' or 'voice' call removing the need to update them mid-call. However keeping the current tiles and simply leaving them in the state that the call was originally placed in might be preferable, particularly so the call recipient knows whether their camera will be on or off when they accept the call. <-- @gaelledel to input.

Interested to see how this proposal would impact push notifications too.

Anecdotally I really like to know if someone is placing an explicit voice or video call before answering. I might be somewhere inappropriate to answer with video, for instance.

[SimonBrandner]

Anecdotally I really like to know if someone is placing an explicit voice or video call before answering. I might be somewhere inappropriate to answer with video, for instance.

The difference should definitely stay visible in the notification 👍

[jakewb-b]

Therefore, on Element web, using the 'voice call' button (telephone icon) would in fact start a video call but simply with both cameras defaulting to off.
The tile could change to a more generic icon and just the text 'Call', rather than 'video' or 'voice' call removing the need to update them mid-call. However keeping the current tiles and simply leaving them in the state that the call was originally placed in might be preferable, particularly so the call recipient knows whether their camera will be on or off when they accept the call. <-- @gaelledel to input.

Interested to see how this proposal would impact push notifications too.

Anecdotally I really like to know if someone is placing an explicit voice or video call before answering. I might be somewhere inappropriate to answer with video, for instance.

Thanks. The more I think about it, the more I'm leaning towards keeping the current tiles and notifications as they are, so you know what is incoming and what will happen when you press accept.

With this, there is the potential that I start a voice call, then switch my camera on while it is ringing, so the notification for you is still 'voice call' and when you answer you might be surprised to see my video on. However, your video is still off, so I think while this may be a little unexpected it shouldn't be a great concern. If it is, we could just prevent the user from enabling video while the call is ringing.

To be clear, from the WebRTC perspective we will still start a voice call - audio track only. But that doesn't concern the UI

@SimonBrandner - thanks! I updated that line just to make it more clear.

[SimonBrandner]

With this, there is the potential that I start a voice call, then switch my camera on while it is ringing, so the notification for you is still 'voice call' and when you answer you might be surprised to see my video on. However, your video is still off, so I think while this may be a little unexpected it shouldn't be a great concern. If it is, we could just prevent the user from enabling video while the call is ringing.

Thinking about this again, I think we can't allow you to turn on your camera while ringing, since at that point we don't know if the other side will be able to handle that change

(though from the spec perspective this will probably become mandatory in the future)

[jakewb-b]

With this, there is the potential that I start a voice call, then switch my camera on while it is ringing, so the notification for you is still 'voice call' and when you answer you might be surprised to see my video on. However, your video is still off, so I think while this may be a little unexpected it shouldn't be a great concern. If it is, we could just prevent the user from enabling video while the call is ringing.

Thinking about this again, I think we can't allow you to turn on your camera while ringing, since at that point we don't know if the other side will be able to handle that change

(though from the spec perspective this will probably become mandatory in the future)

That's fine, I don't think that's essential by any means. I've struck it through for now, and we can re-consider it in the future. I've also updated the comment above to confirm that notifications and tiles will remain specific to the call type. This is also relevant so that if I miss a call and hit the 'call back' button that appears in the tile, it is clear what type of call I will be making.

[SimonBrandner]

This is now a part of the js-sdk, so the only part left here is the UI for this

[viasux]

Based on my discussion with @SimonBrandner and @dbkr, we propose the following solution:

* A video call can already be 'downgraded' to a voice call essentially just by turning off camera, so there is no urgent requirement for any change. (This does not force the other user to also turn their camera off, but I don't think this is a greatly needed use-case, and arguably not even a desirable one)

* A voice call is essentially just a video call just with both cameras turned off. Given that, this means that it can be easily 'upgraded' to a video call by simply turning the camera on. This requires very little extra functionality or UI.

* Therefore, on Element web, using the 'voice call' button (telephone icon) would (from the user's POV) be the same as starting a video call but simply with both cameras defaulting to off.

* As the user placing a voice call, I can choose to turn my camera on while the call is ringing, but the other party's camera will still default to off. They can turn it on as soon as the call starts if they wish.

* The video call (camera icon) remains in place, and starts a video call with both cameras defaulting to on.

* Notifications will remain specific to the type of call, so the recipient knows what to expect when they answer.

* The current tiles will also remain the same, and simply reflect the type of call that was originally placed, regardless of whether cameras were turned on or off part-way through.

* For cross-client compatibility, we can simply disable the 'turn camera on' icon when receiving a voice call from a client where this is not implemented.

Ideally, this same solution will be adopted on other platforms, but this is not critical.

I will work with @gaelledel next week to review the UX and confirm any UI changes. They should be very minimal.

I have an issue with the fact that joining a video call automatically turns on your camera. If you join a video call in any other program, it's not going to turn on your camera without your permission. What if I just want to watch someone else's video without turning on my own? To do so, I'd have to join the call, cover my camera, and then turn off my camera. Also from a privacy/security perspective, having to expose your video signal (resolution, lighting, etc.) in order to even participate in a video call is not ideal.

[jakewb-b]

I think it's important to bear in mind that this ticket relates to 1:1 calls, so it's less about 'joining' an existing call, and more 'accepting' an incoming call.
Given that, I don't know if I agree with your point that if I accept a video call in other programs it wouldn't turn my camera on by default. In fact, if I accept an incoming video call in WhatsApp or Facetime, I will start the call with my camera on. That seems logical, as it is made clear that I am accepting a video call. Having to accept the call and then turn my camera on would be poor user experience.

I do agree that there is a case for providing two options when answering a video call: 'answer with video' or 'answer with voice only'. I don't think that's a bad idea, but I'm not sure it's in the scope of this ticket. Even in today's implementation, if someone video calls you your only options are to accept and go straight into a video call, or to decline and call back with audio only, so this ticket doesn't change that either way.

[SimonBrandner]

Just to note: from the technical perspective adding two buttons Answer with video and Answer with just audio should be super simple, if designs were provided

[viasux]

Just to note: from the technical perspective adding two buttons Answer with video and Answer with just audio should be super simple, if designs were provided

do you accept design contributions?

[SimonBrandner]

Just to note: from the technical perspective adding two buttons Answer with video and Answer with just audio should be super simple, if designs were provided

do you accept design contributions?

I am awaiting the Element design team to get more time on their hands to figure this out