Streaming Media Manager 5.0 #56

asm09fsu · 2016-11-17T19:23:15Z

This proposal is geared at revamping how the Streaming Media Manager (SMM) class manages audio/video streaming. This revamp will also include lifecycle management for these streams as the overall app/proxy lifecycle changes.

henry-fan · 2016-11-23T08:48:01Z

There is one case which cannot be covered based on our observation. Since iOS10 can auto-detect AppLink apps, there is one scenario where app gets registered and then move to HMI_FULL on user selection, at the meantime app remains background (app enters state machine with initial state Backgrounded), we couldn't do anything about it and user gets a constant "waiting device" screen. The change should also be able to handle this case

asm09fsu · 2016-11-23T09:04:52Z

@henry-fan you are right that is a use case that needs to be handled. What would you expect to see in this use case, as the app is backgrounded and the encoder cannot be used? I am thinking an Alert with tts.

henry-fan · 2016-11-30T03:30:46Z

@asm09fsu it might be the only choice we have.

asm09fsu · 2016-11-30T17:02:38Z

proposals/nnnn-streaming-media-manager-update.md

@@ -52,7 +52,7 @@ There will need to be multiple state machines that need to work together to deal
 3. Limited
 	- App was open, however the user has moved to another screen on the head unit. This app is still the primary streaming app. The Streaming State must not move from **Ready**, however if the Phone State is moving from **Active** to **Resigning Active**, we must send 30 frames.  
 4. Full
-	- App is open, and is currently in focus. If the Streaming State is currently **Stopped**, we should proceed to **Starting**, so long as the Phone State is not **Resigning Active** or **Backgrounded**. If moving to this state and the Phone State is currently **Regaining Active** or **Active**, and the Streaming State is currently  **Ready**, the Streaming State should be moved to **Stopped** followed by **Starting** (restarting the streams).
+	- App is open, and is currently in focus. If the Streaming State is currently **Stopped**, we should proceed to **Starting**, so long as the Phone State is not **Resigning Active** or **Backgrounded**. If moving to this state and the Phone State is currently **Regaining Active** or **Active**, and the Streaming State is currently  **Ready**, the Streaming State should be moved to **Stopped** followed by **Starting** (restarting the streams). If moving to this state and the Phone State is currently **Backgrounded**, send an RPC to alert the user to open the app (current suggestion is an Alert with TTS).


@henry-fan is this a desired inclusion to handle the missed state?

asm09fsu · 2016-11-30T17:03:32Z

proposals/nnnn-streaming-media-manager-update.md

@@ -63,23 +63,23 @@ There will need to be multiple state machines that need to work together to deal
 	- Streaming sessions and encoder are ready. This state is only active so long as HMI is **Limited or greater** and Phone State is **Regaining Active** or **Active**. If at this state, and Phone State changes to **Resigning Active**, send ~30 frames. If at this state, and HMI changes to **Background or less**, move to **Stopped**.

 ##### Sending Video Data
-There will be a handler, `videoDataHandler`, that will be within SMM that will expect a return of a `CVImageBufferRef` for video.


@kshala-ford How does this look? Since we are only assuming the developer is using OpenGL, I have changed how we get the video frames from a developer.

kshala-ford · 2016-12-04T12:37:23Z

Unsure if this is covered. What happens to the stream when constantly receiving push notifications? I can imagine scenarios receiving lots of notifications coming from all my facebook/WhatsApp groups. Is it planned to continue the video streaming in that situation?

asm09fsu · 2016-12-05T23:59:05Z

@kshala-ford On iOS currently, notifications show up in a non-evasive manner (banners at the top). This does not change the state of the current foreground app. If an application displays a UIAlertView or UIAlertController, the app will receive a UIApplicationWillResignNotification notification, which will be handled in a similar manner as is described in the doc (sending some frames to the HU or an audio alert, or something.

…ecycleManager.

joeljfischer · 2017-03-01T19:30:45Z

proposals/nnnn-streaming-media-manager-update.md

+
+
+## Motivation
+We currently have a SMM which provides the basic functionality for streaming audio and video, however much of the management of this class is still up to the developer, and may vary by implementations. In order to allow developers to deliver high quality mobile navigation experiences, the SMM should be revamped to remove much of the bloat and management of the sessions internally and allow the developers to easily use this class for mobile navigation applications. If you need a real reason for why this should be considered, look at the Detailed Design section for all the possible complexities we expect every developer to implement.


If you need a real reason for why this should be considered, look at the Detailed Design section for all the possible complexities we expect every developer to implement.

Are you currently giving me the fake reason? 🤔

joeljfischer · 2017-03-01T19:41:11Z

proposals/nnnn-streaming-media-manager-update.md

+There will need to be multiple state machines that need to work together to deal with all the possible scenarios the app can be in on both the phone and head unit. Luckily, HMI states are already provided so we will not be building a new state machine, but will list the possible scenarios for different states below.
+###### Phone (new state machine)
+1. Resigning Active
+	- The app is moving to the background. If the streaming state is **Ready**, and in HMI **Limited or greater**, we should send ~30 frames so that we can alert the user to reopen the app. The streams will remain open. Can move to **Backgrounded**.


Why 30 frames?

30 frames guarantees that we send enough to be visible on the head unit. This can be changed, but in testing this seems to be a good amount.

joeljfischer · 2017-03-01T19:43:40Z

proposals/nnnn-streaming-media-manager-update.md

+2. Backgrounded
+	- The app is currently in the background, and cannot use the encoder to send over frames. Regardless of HMI changes, we cannot stream video. The stream should remain open if currently open. If the HMI changes to **Background or less**, we should close the sessions, and the Streaming state will move to **Stopped**. Can move to **Regaining Active**.
+3. Regaining Active
+	- The app is currently regaining it’s position as the foreground app. If we are currently in HMI **Limited or greater**, and the Streaming state is **Ready**, we will restart the streams by moving to state **Stopped**, and then **Starting**, followed by **Ready**. Can move to **Active**.


Wouldn't this just be that the streaming state machine would just move to starting? It should already be in stopped, and ready is conditional on the streaming state machine, not this.

No, as we must stop and restart the sessions by sending an end session and then a start session.

Then why does 2. Backgrounded say it's moved to stopped? Is it moving from stopped to stopped again?

The streaming state is independent from the app and hmi state. The streaming state moves to stopped only if the HMI state goes to background or none. In all other instances, the stream stays open. Not sure where you are getting stopped -> stopped?

joeljfischer · 2017-03-01T19:44:11Z

proposals/nnnn-streaming-media-manager-update.md

+3. Regaining Active
+	- The app is currently regaining it’s position as the foreground app. If we are currently in HMI **Limited or greater**, and the Streaming state is **Ready**, we will restart the streams by moving to state **Stopped**, and then **Starting**, followed by **Ready**. Can move to **Active**.
+4. Active
+	- The app is in the foreground. We are able to stream if needed, permitting we are in the proper HMI. Can move to **Resigning Active**.


permitting

"assuming"?

joeljfischer · 2017-03-01T19:52:11Z

proposals/nnnn-streaming-media-manager-update.md

+FULL        | Yes                   | No
+
+##### Sending Video Data
+The way in which video frames are sent over will not change from the current implementation. This function will require input of a `CVImageBufferRef` for video. It will be up to the developer to decide how to frequently to call this function. The developer will be able to know if the currently connected stream is encrypted with a `BOOL` `isEncrypted`, a `CGSize` `screenSize` to let them know the current Head Unit screen size, and a `CVPixelBufferPoolRef` for using the encoder’s pixel buffer pool, as is currently available in SMM.


I assume isEncrypted et. al. are actually encrypted with the getter prepending the is?

joeljfischer · 2017-03-01T19:54:48Z

proposals/nnnn-streaming-media-manager-update.md

+```
+
+##### Sending video data when iPhone is moving to background
+Since iOS cannot use the encoder when the phone is in the background, to conserve battery and CPU, we must find a solution that will drive the user to reopen the app. The suggested solution is to create a single frame that is generated when SMM’s encoder is initialized, and send that over as the Phone State changes from **Active** to **Resigning Active**. This should be built as a non-customizable screen, so that we can rely on consistent behavior among streaming applications for the time being. The current idea is to have a black screen with the text “Please re-open <#App Name#>”. We have the option to provide translations for all languages that SDL supports, if we wish to go that route. We can also use the app icon, if we wish instead of the app name, as the app name could potentially be quite long.


It's not 100% clear, but it looks like you're saying this would be built into the SMM?

I would say both name and icon could be helpful. Do we have an idea for how this might be implemented?

This could be implemented in SMM, but that's the point of the proposal is to see if that is ideal. We could do an XIB, or dynamically generate an image. Just depends on if we want to give the developers to ability to customize. The only problem with customization is we must make sure it is NHTSA compliant for US market.

joeljfischer · 2017-03-01T19:56:38Z

proposals/nnnn-streaming-media-manager-update.md

+Since iOS cannot use the encoder when the phone is in the background, to conserve battery and CPU, we must find a solution that will drive the user to reopen the app. The suggested solution is to create a single frame that is generated when SMM’s encoder is initialized, and send that over as the Phone State changes from **Active** to **Resigning Active**. This should be built as a non-customizable screen, so that we can rely on consistent behavior among streaming applications for the time being. The current idea is to have a black screen with the text “Please re-open <#App Name#>”. We have the option to provide translations for all languages that SDL supports, if we wish to go that route. We can also use the app icon, if we wish instead of the app name, as the app name could potentially be quite long.
+
+##### Sending Audio Data
+Since audio data is sent sporadically, we will have a function that will send audio data only when necessary.


Doesn't this already exist? I'm confused if this is a comprehensive overview or just showing the changes?

This is an entire rewrite of SMM, so this will contain an overview.

joeljfischer · 2017-03-01T19:58:22Z

proposals/nnnn-streaming-media-manager-update.md

+```
+
+## Impact on existing code
+The impact on SDL is that the previous version of `SDLStreamingMediaManager` and all code associated will no longer work. This will require a developer to adapt to using this new method of streaming, but will greatly improve the efficiency of Mobile Navigation Apps.


This seems mostly additive, public API-wise. What behavior prevents current code from working with this change?

This is not only additive. We are completely removing the ability to start/stop the sessions, because we want consistent lifecycle between all streaming apps.

BrandonHe · 2017-03-02T06:20:44Z

@asm09fsu
I am trying to understand the state switching. Too much states matrix from HMI and App to understand, could you add State Machine Diagram(s) to help us more clearly understand this proposal? Thanks!

Created nnnn-streaming-media-manager-update

85549ba

asm09fsu added ios protocol and removed protocol labels Nov 17, 2016

asm09fsu added 2 commits November 30, 2016 00:39

Updated to add a scenario for HMI/Phone state not included.

22a3c76

Updating how video data is sent.

473627e

asm09fsu commented Nov 30, 2016

View reviewed changes

asm09fsu mentioned this pull request Dec 7, 2016

Update Streaming Media Manager to handle all App & HMI States smartdevicelink/sdl_ios#478

Closed

asm09fsu mentioned this pull request Jan 28, 2017

Streaming Media Manager Updates smartdevicelink/sdl_ios#514

Merged

3 tasks

asm09fsu added 3 commits January 30, 2017 14:05

Updated to include stream lifecycle charts. Updated isPaused code.

fdfa584

Updated to add in support for touch manager improvements

3f5c208

Fixed property annotations

df19ce5

asm09fsu mentioned this pull request Feb 10, 2017

Streaming sessions are not closed on proxy disconnect smartdevicelink/sdl_ios#386

Closed

Updated to talk about incorporating a new class, SDLStreamingMediaLif…

66577a1

…ecycleManager.

joeljfischer reviewed Mar 1, 2017

View reviewed changes

asm09fsu changed the title ~~[WIP] Streaming Media Manager 5.0~~ Streaming Media Manager 5.0 Mar 7, 2017

asm09fsu added the review ready label Mar 7, 2017

theresalech merged commit d1f11dc into smartdevicelink:master Mar 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streaming Media Manager 5.0 #56

Streaming Media Manager 5.0 #56

asm09fsu commented Nov 17, 2016

henry-fan commented Nov 23, 2016

asm09fsu commented Nov 23, 2016

henry-fan commented Nov 30, 2016

asm09fsu Nov 30, 2016

asm09fsu Nov 30, 2016

kshala-ford Nov 30, 2016

kshala-ford commented Dec 4, 2016

asm09fsu commented Dec 5, 2016

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017 •

edited

joeljfischer Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

joeljfischer Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

joeljfischer Mar 1, 2017

asm09fsu Mar 1, 2017

BrandonHe commented Mar 2, 2017



		## Motivation
		We currently have a SMM which provides the basic functionality for streaming audio and video, however much of the management of this class is still up to the developer, and may vary by implementations. In order to allow developers to deliver high quality mobile navigation experiences, the SMM should be revamped to remove much of the bloat and management of the sessions internally and allow the developers to easily use this class for mobile navigation applications. If you need a real reason for why this should be considered, look at the Detailed Design section for all the possible complexities we expect every developer to implement.

Streaming Media Manager 5.0 #56

Streaming Media Manager 5.0 #56

Conversation

asm09fsu commented Nov 17, 2016

henry-fan commented Nov 23, 2016

asm09fsu commented Nov 23, 2016

henry-fan commented Nov 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kshala-ford commented Dec 4, 2016

asm09fsu commented Dec 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asm09fsu Mar 1, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BrandonHe commented Mar 2, 2017

asm09fsu Mar 1, 2017 •

edited