UWP Unable to OpenMedia multiple times on the same MediaPlayer

[eedmond]

I have a project (using WinRT video API) and am finding a difference in behavior between in editor and on a HoloLens in a UWP app package. We have a MediaPlayer gameobject that is intended to handle whichever video a user selects. However, I am finding that (unlike in editor, which works as expected) only the first video plays properly. If I select another video, which calls OpenMedia with a different URL, the media player is stuck on the frame last drawn from the previous video. I see in the MediaPlayer code, that OpenMedia internally calls CloseMedia first, so it appears I'm not missing a cleanup call from what I can tell.

Any tips on addressing/debugging this issue? Unfortunately, it's quite difficult to get live debugging in a UWP package. I am using ApplyToMesh to render the video, so perhaps the issue is somewhere there.

[eedmond]

I've written a minimum repro scene with a gameobject with a MediaPlayer and a child with ApplyToMesh. I have 2 buttons to load different videos. If I load the first video, it plays fine. However, I can't play another video. It seems it's in a bad state after calling OpenMedia the second time.

[AndrewRH]

Hi,

I haven't been successful at reproducing this issue yet, so perhaps you could help us with that by providing a bit more information:

Please could you let me know which version of AVPro Video and Unity you're using?

Also is this problem reproducible using our sample media? If not, what kind of media are you loading?

And finally, if you do a Desktop UWP build do you have the same issue?

Thanks,

[eedmond]

Thanks, Andrew. Interesting to hear it's not reproducing for you. We're using Unity version 2019.4.23f1 (which has caused us some issues already, so I wonder if this might be the cause), and AVPro Video version v2.1.3-Enterprise. I am also on a non-standard build of HoloLens, which also might explain the issue.

I'll reply back when I try a Desktop UWP build and the sample media scenes on HoloLens and let you know if they worked.

Thanks,
Eric

[eedmond]

I'm curious which version of Unity and which build of HoloLens you are on. I could snap to each of those and try things out again.

[eedmond]

Sorry, should have also mentioned we're using HLS video streaming, so we're not referencing local video files.

[AndrewRH]

Thanks for this information, I was just testing with local files. Also I don't have a Hololens so I'm testing Desktop UWP builds - but in theory it should be the same since it's UWP....

I was testing with Unity 2019.3.13f1

I will do some more testing and include HLS streams and your version of Unity.

Please do let me know about the Desktop build.

Thanks,

[eedmond]

From my testing, this issue does not reproduce on a Desktop UWP app, so it appears to be HoloLens-specific.

I have been adding some logs in the AV Pro code and noticed that even though WindowsRtMediaPlayer.OpenMedia runs successfully, it's never loading the videos in WindowsRtMediaPlayer.Update. It just spins with _isMediaLoaded, HasAudio, and HasVideo are all false endlessly.

Let me know if there is some additional logging I could add to better root cause this.

[AndrewRH]

Thanks for these details. I'm afraid we don't own a HoloLens to test on :(

I could perhaps make a debug build of the plugin to send to you, with some additional logging so we can try to work out what is happening. I worry though that this may be a slow turnaround cycle. I'm not sure if there is a contact in the UK that could ship us a Hololens for a few days to debug this on.

It seems odd that it works on Desktop UWP, but on the Hololens which I believe is ARM64 it can load the HLS once but not again.

A few questions and things to try:

1. Are you using D3D12? If so, perhaps try using D3D11 instead (in Player Settings > Auto Graphics API)

2. Could you try removing the ApplyToMesh and see if this fixes it - you won't be able to see the result, but hopefully if your video contains audio you can hear whether the video begins to play. I doubt that it is the fault of this component though, I suspect it's something internal in the native plugin DLL

3. Could yout use the Apple Bip-Bop HLS stream to test with, as this is what we're testing with here and it rules out one variable: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8

4. Are you building with any sort of optimisations, such as RELEASE/IL2CPP? I wonder if buidling without these, or making a development build can help?

5. It's interesting that you get similar results when using both WinRt and Media Foundation. This suggests it's something fundamental in the low-level as our code for these 2 playback paths are completely different. Can you just confirm that you're changing this API in the Platform Specific > UWP section and not the Platform Specific > Windows section as this is an easy mistake to make:
   

6. Do you have any log generated from the Hololens run that you could share with us?

7. I wonder if you could test with a MP4 instead of HLS as well to see whether that makes a difference. Such as this: https://rh-testmedia.s3-eu-west-1.amazonaws.com/Simple/BigBuckBunny-360p30-H264.mp4
   For example if you play this MP4 first, can you play another MP4, and can you then play a HLS? And after playing HLS can you then still play the MP4?

I'll do some digging on this side to see if we can add some logging into a debug DLL.

Thanks,

[eedmond]

Thanks so much for the reply, Andrew. I'm travelling tomorrow, but I'll keep an eye on messages and will test things out again on Monday.

1. We are actually using D3D11. Here is a snippet of the top of our logs.

Direct3D:
    Version:  Direct3D 11.0 [level 11.1]
    Renderer: Qualcomm(R) Adreno(TM) 630 GPU (ID=0x45373230)
    Vendor:   
    VRAM:     1931 MB
Initialize engine version: 2019.4.23f1 (3f4e01f1a5ec)
D3D11 device created for Microsoft Media Foundation video decoding.

2. Earlier today I tried swapping ApplyToMesh with a uGUI, but saw similar results. I had the same thought that given what I'm seeing in the logs (the WinRTMediaPlayer never loading audio or video), it seems to be a separate issue from exactly how we're rendering it.

3. I can add this to our test, but it's worth noting that I haven't found a very consistent set of failures/successes. My test scene right now uses 3 HLS streams and rotates between them every 4 seconds. After showing each video, it toggles the entire game object on and off for 2 seconds. I'm not sure if it's consistently the case, but it seems the first video loads and plays, the next two fail (doesn't even show the default texture that I set), and after toggling the game object, all the videos seem to load. Again, this isn't perfectly reliable, but that's roughly the behaviour I'm seeing.

I give all this detail to simply say we can add this test URL, but we'll need another 2 for testing purposes.

4. That's correct on both accounts -- Release builds and IL2CPP. I'm not aware of a way to produce an app that can run on HoloLens without using IL2CPP. Is that an option?

5. Correct. My test scene has two side-by-side video players, one using WinRT API and one using Media Foundation. I can verify in the logs from MediaPlayer.OpenMedia that it opens the videos with each API. Recently, after seeing similar behavior in both players, I've removed the Media Foundation player and am just testing with the WinRT player.

6. I'll follow up with you in our email thread with a log from my test scene (with just the WinRT player).

7. I'll give this a try on Monday (next I'll be working) and will loop back with you. I'm curious if this is HLS-specific or if local videos will also fail. Definitely good idea to test this out.

[AndrewRH]

Thanks for the information and for emailing over the log file.

It seems like it's not detecting any valid tracks during track enumeration, so this is why it considers the media as not loaded. Could you try adding some more logging in as this may help us narrow down the issue.

In Tracks.cs method UpdateTracks could you see whether InternalIsChangedTracks is ever passing?

I also wonder whether it's worth making a new simpler test case that just loads the same URL over and over until it fails. Here's a script you could try:

  using System.Collections;
  using System.Collections.Generic;
  using UnityEngine;
  using RenderHeads.Media.AVProVideo;
  
  public class TestReloading : MonoBehaviour
  {
	  [SerializeField] string url = string.Empty;
	  private MediaPlayer mediaPlayer;
  
	  private float loadTimer;
	  private float playTimer;
	  private int loadCount;
  
	  void Update()
	  {
		  if (mediaPlayer == null)
		  {
			  var go = new GameObject("Player");
  
			  mediaPlayer = go.AddComponent<MediaPlayer>();
  
			  var display = go.AddComponent<DisplayIMGUI>();
			  display.Player = mediaPlayer;
			  display.ScaleMode = ScaleMode.StretchToFill;
  
			  mediaPlayer.PlatformOptionsWindows.videoApi = RenderHeads.Media.AVProVideo.Windows.VideoApi.WinRT;
			  mediaPlayer.PlatformOptionsWindowsUWP.videoApi = RenderHeads.Media.AVProVideo.WindowsUWP.VideoApi.WinRT;
			  mediaPlayer.OpenMedia(MediaPathType.AbsolutePathOrURL, url);
			  loadTimer = 0f;
		  }
  
		  if (mediaPlayer != null)
		  {
			  if (!mediaPlayer.Info.HasVideo())
			  {
				  loadTimer += Time.deltaTime;
				  if (loadTimer > 10f)
				  {
					  throw new System.Exception("Failed to load media");
				  }
			  }
			  else
			  {
				  playTimer += Time.deltaTime;
				  if (playTimer > 5f)
				  {
					  loadCount++;
					  mediaPlayer.OpenMedia(MediaPathType.AbsolutePathOrURL, url);
					  loadTimer = 0f;
					  playTimer = 0f;
					  Debug.Log("Load count: " + loadCount);
				  }
			  }
		  }
	  }
  }

Do let me know about the HLS vs remote MP4 / local file test too please.

I will also try to email your a new DLL build which contains some extra logging.

Thanks,

[eedmond]

Thanks again, Andrew! I appreciate the help.

I added logging to InternalIsChangedTracks, and in the unsuccessful cases, returns false every frame for audio and video.

I didn't get time to try local MP4's today, but I did find something really interesting that's worth noting -- I repeated the exact same scenario, but with the rest of our app disabled. This loaded 25 of 25 videos in a row without issue. The same code, with the rest of our app running, fails sporadically. It appears there's some conflicting code in our app or possible resource contention that's causing the videos to never load. We have some audio/speech recognition that might be fairly demanding and I wonder if, somehow, that's the source of the failures.

Unfortunately, I don't have logs with the new DLL. We were encountering crashes after loading a few videos each run. I was wondering if the added logging was throwing in certain cases? I reverted to the original DLL for most of the tests today.

[eedmond]

One of our theories is that our app has high memory consumption and that might cause the video API to silently fail and/or spin endlessly.