Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HLS on Safari / IOS now not working #9754

Closed
danrossi opened this issue Sep 23, 2016 · 239 comments
Closed

HLS on Safari / IOS now not working #9754

danrossi opened this issue Sep 23, 2016 · 239 comments

Comments

@danrossi
Copy link

I apologise if there is nothing you can do. I am very certain in my tests HLS has been working and suddenly is now not.

I am using the CORS proxy hack here so CORS isn't a problem. I don't need to use the CORS proxy in WebView IOS apps and even there rendering is an issue.

Is there a WebGL fix that can be applied to get HLS rendering in WebGL ? I will try the canvas renderer to see if it helps. I know there is a requirement for double drawing of the canvas to get a frame up when using drawImage but this isn't an issue with Mp4 files.

The example is here

http://dev.electroteque.org/threejs/

@danrossi
Copy link
Author

It is drawing to canvas and rendering with the canvas renderer albeit dropping frames. Something is required with webgl.

@danrossi
Copy link
Author

This is a beauty. I have modified a kpano demo a little with the proxied HLS stream.

It is working now on Safari OSX but not on IOS 9. Any ideas what the difference is ? I am not an expert with the webgl flags sadly or what they even do. On IOS9 it's still a black frame.

http://dev.electroteque.org/threejs/webgl.html

@makc
Copy link
Contributor

makc commented Sep 23, 2016

this last link is black rect / sound only in safari 9.1.3 / mac os x 10.11.6

@mrdoob
Copy link
Owner

mrdoob commented Sep 23, 2016

What's HLS?

@danrossi
Copy link
Author

danrossi commented Sep 24, 2016

@makc really ? I have OSX 10.10 and Safari 10. Well that explains the recent sabotage change by Apple. Safari 10. So if we're Safari 10 we're all good.

@mrdoob HLS = Apple Streaming. ie for live streams but also VOD. Live streaming is going to become a norm with the Terradeck gear. ie http://dev.electroteque.org/video/360/hls/ultra_light_flight.m3u8

Any ideas what the differences could be there ? I tried changing some things in the first example. The second example is raw webgl functions.

so texImage2D will partially render HLS but not on IOS yet maybe because it's not Safari 10 ?

@danrossi
Copy link
Author

danrossi commented Sep 24, 2016

Please forgive me I have reported on a few webkit bug tickets.

They've gone and sabotaged the CORS proxy hack on an IOS update. macOS also is reported to not have fixed the CORS issue in Safari either.

I have to now figure out how to work around their sabotage. I have to get that working to further test HLS working on IOS. Apple is seriously making me work up a sweat to put out their fires hahah.

If I can get OSX rendering that would be a start. I just need to figure out what is different in three.js ?

@danrossi
Copy link
Author

danrossi commented Sep 26, 2016

I'm still trying to do reduction methods to compare what is different. Once I figure that out I may get IOS working too.

Perhaps the extra shader programs within three.js is causing the rendering problems ? I'm not across it, but trying to replicate in the raw webgl example what three.js is doing in that equirectangular example. Is there example shader code for this ?

I found some raw functions here which hopefully I can replicate fully what three.js is doing. I can't just copy the shader programs directly because some variables are set externally like world position.

https://bl.ocks.org/mbostock/5446416

@dustinkerstein
Copy link

I may have just run across this or possibly a related issue. I have h264 mp4 files located on an S3 bucket and I am loading them to a video.src as a blob.

If I upload to my S3 bucket using the Amazon web-based uploader, the videos play fine when directly loaded in Mobile Safari (iOS 10.0.1), but if I try to play them using my three.js viewer then I just get a black screen. However, if I upload to the S3 bucket with Cyberduck, they play fine directly in Mobile Safari and in three.js.

It looks like Cyberduck is setting the meta-data 'Content-Type=video/mp4' and the AWS uploader is 'Content-Type=application/octet-stream'. When I manually set this meta-data to video/mp4 everything works correctly.

Strange issue, and one that caused me a lot of head scratching. Let me know if this is the same underlying issue as this ticket, or if I should create a new one.

Thanks,
Dustin

@danrossi
Copy link
Author

This is for HLS mate as in mpegts fragments or fragmented files not mp4. Mp4 is fine on both Safari and IOS. I checked the mimetype in cyberduck and it says the mpegts is correct.

The same stream is working for me in the raw webgl example just not on IOS of course. Your issue seems similar to mine though as in a black frame. No rendering at all.

@dustinkerstein
Copy link

@danrossi ok cool. I'll create a new ticket specifically for my issue. Thanks.

@phaseOne
Copy link

phaseOne commented Sep 28, 2016

I think this is the issue: www.krpano.com/ios/bugs/ios10-webgl-video-texture-crash/
I'm currently trying to implement the same workaround for three.js, but not getting very far.

@danrossi
Copy link
Author

@phaseOne. It is a safari issue as both a problem on OSX and IOS. Mp4 is working fine for IOS and OSX. I'll tinker with antialias .

Still can't work out the differences between the two. The raw webgl example still won't work on IOS though but getting it working on OSX is a start.

I'm wondering if I can find a working equirectangle shader program that I can drop in.

@phaseOne
Copy link

More info here: http://stackoverflow.com/questions/39123109/rendering-a-video-with-webgl-in-ios-10-beta-7-safari-shows-weird-purplish-co.

I realize that this is a browser bug, but implementing a workaround might be necessary given how long these WebGL bugs take to get fixed.

@NishimuraT
Copy link

In the case to use the texture through the canvas, the video was displayed in safari.
However, since there are still CORS problems in iOS, it worked only in the Same Origin.
https://bugs.webkit.org/show_bug.cgi?id=135379

I'm not familiar with the webgl, but please refer to the following.
NishimuraT/videojs-panorama@bd99200

@danrossi
Copy link
Author

danrossi commented Sep 28, 2016

@NishimuraT I think I know what you mean use the canavasrender. I did try that to do reduction method.

I think that was working but it causes severe drop frames and playback performance issues due to cpu. Obviously worse on IOS. Ipad 3 drops frames for webgl already.

@NishimuraT
Copy link

@danrossi I had a little misunderstanding. I'm sorry. I had sent a misguided thing.

@danrossi
Copy link
Author

I have confirmed the same problem with the elevr project which uses raw webgl and shaders. Will have to hack that one to work my way back. Something in there is causing rendering issues and would affect three.js the same.

@danrossi
Copy link
Author

I have confirmed in elevr it is auto clearing binding the texture which is causing it to break ie

webGL.gl.bindTexture(webGL.gl.TEXTURE_2D, null);

If this is commented out it is working.

This was also causing a problem, I'm not sure what it's for.

webGL.gl.pixelStorei(webGL.gl.UNPACK_FLIP_Y_WEBGL, true);

Maybe there is an auto clear property that is doing the same thing in three.js ?

@danrossi
Copy link
Author

I see many references to

state.bindTexture(_gl.TEXTURE_2D, null);

But don't know where it could possibly be. Is it required to do that ?

@danrossi
Copy link
Author

code is too crazy to track down what is causing the texture drawing issue. Seems to have something with this binding of texture.

@mrdoob
Copy link
Owner

mrdoob commented Oct 22, 2016

@danrossi is there any page out there that renders it properly?

@danrossi
Copy link
Author

Sorry about the noise mate. I did more AB testing. It is indeed it seems what I mentioned, as I had to comment it out for elevr demo also.

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, texture.flipY );

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, texture.flipY );

This is causing the problem. However without it the texture is upside down. I'll add another test to show it now working.

@danrossi
Copy link
Author

Another raw example I found, commented out FLIP_Y. Possible Webkit bug now ?

http://dev.electroteque.org/threejs/webgl3.html

@danrossi
Copy link
Author

The only reference to flipY is this but webkit nightly still an issue

https://bugs.webkit.org/show_bug.cgi?id=162491

If I disable flipY which I assume breaks hardware acceleration it is working ?

texture = new THREE.VideoTexture( video );
                texture.minFilter = THREE.LinearFilter;
                texture.format = THREE.RGBFormat;
                texture.magFilter = THREE.LinearFilter;
                texture.flipY = false;

http://dev.electroteque.org/threejs/hlsflipy.html

Trying to see if it's a webkit bug now. How is it possible to rotate the texture around to work around it for now ?

@makc
Copy link
Contributor

makc commented Oct 23, 2016

How is it possible to rotate the texture around to work around it for now ?

why not just flip geometry uv or do that in the shader

@danrossi
Copy link
Author

@makc I have no idea how to go about that.

Just to do heads in further. This FlipY "fix" doesn't help on IOS 9, just tested it.

Something else is a problem there now. It took days just to track this one down.

It may be all webkit's problem not Three.JS from the looks of it. As per usual so sorry about this.

I have filed a report there too https://bugs.webkit.org/show_bug.cgi?id=163866

@makc
Copy link
Contributor

makc commented Oct 23, 2016

I have no idea how to go about that.

http://jsfiddle.net/hfj7gm6t/2182/

@danrossi
Copy link
Author

It calls this. after capturing hls resize events

this.video.width = MathUtils.ceilPowerOfTwo(this.video.videoWidth);
this.video.height = MathUtils.ceilPowerOfTwo(this.video.videoHeight);

then resize

 this.camera.aspect = width / height;
 this.camera.updateProjectionMatrix();
 this.renderer.setSize(width, height);

@makc
Copy link
Contributor

makc commented Feb 13, 2021

@danrossi so what are you saying is that the video has to be power-of-two or else it goes black? are you sure it is not just bcause you have bad filters / misconfigured mipmapping?

@danrossi
Copy link
Author

danrossi commented Feb 13, 2021 via email

@danrossi
Copy link
Author

Somebody is now observing my fix works on Ipad which is the only device I have access to test. But Iphone IOS 14, the rendering won't appear. Its not an overall webgl bug like before. It's pathetic regardless though. I don't have access to an Iphone to test it.

My texture is setup like.

        var texture = new Texture(video);
        texture.format = RGBFormat;
        texture.minFilter = LinearFilter;
        texture.magFilter = LinearFilter;
        texture.wrapS = ClampToEdgeWrapping;
        texture.wrapT = ClampToEdgeWrapping;
        texture.generateMipmaps = false;

@danrossi
Copy link
Author

danrossi commented Feb 23, 2021

Somebody at the bottom here claims it might be "fixed" again in IOS 14.2 beta but on Ipad. But a client says 14.4 on Iphone IOS is still a problem. Webkit don't use HLS textures in their test files. When I first reported I had to modify them with HLS to demonstrate the problem. There may still be a problem with Iphone for who knows what reason.

https://bugs.webkit.org/show_bug.cgi?id=215908

@danrossi
Copy link
Author

Fix is in webkit but not in IOS as usual. Might not be released until 14.5. So no point testing fix on Ipad but Iphone.

@arpu
Copy link

arpu commented May 20, 2021

@danrossi do you found a workaround for iphone ios 14 ? i follow the bugreport https://bugs.webkit.org/show_bug.cgi?id=215908
tested ipone 11 ios 13 works fine ios 14 not ipad 14 works ( tested with Browserstack)

@danrossi
Copy link
Author

I'm working on this feature again but to upgrade issues with the webxr polyfill not working with current three, and my virtual controlbar not working with the old vr controller code now.

I will get back to checking HLS in IOS 14.5 but it seems some guy just reported its still not working in beta. I'm not killing myself over Apple antics anymore. It took me 3 months originally in this same thread to find shader work arounds. I don't think this is a shader issue, as its working on IPad 14.4. Its just Webkit still do not use HLS as textures in their webgl tests so they keep breaking things !

Things may change if they finally add WebXR into Safari ! Interest for VR has dropped the past few years, but people still want it, for things like street mapping etc.

I don't use Apple products and have to borrow them for testing. I'll get IOS 14.5 on an Iphone if 14.5 is even released. I think you have to wait for 14.5 to be released to truely see it working again.

@danrossi
Copy link
Author

The goal posts have been moved. Iphone 12 fix in IOS 14.6 ! Another 6 months wait. So a full year wait which is standard for IOS. All because they don't test HLS textures. They will keep making these mistakes I think. The original time was enough.

@arpu
Copy link

arpu commented May 21, 2021

@danrossi where can i find this info for 14.6 fix ?

@danrossi
Copy link
Author

In the webkit thread. I thought it was over and done with in IOS 14.5. Now you have to wait another 6 months for 14.6. I don't understand why they can't just auto update Safari ! Unless its an IOS specific problem.

@arpu
Copy link

arpu commented May 21, 2021

ah thx i do not see the new response in the thread, but still not understand why its not working on iphone 11 and ios 14.5 any idea?

@danrossi
Copy link
Author

No point trying to pull up different devices and revisions just wait for IOS 14.6. I might try and get a hold of an Iphone 12 and put 14.5 on it and run some tests. I have no access to actually see the bug yet it works on Ipad. If there is a webgl flag or shader trick I'll report it. But in 14.4 I managed to get it rendering on startup using the video resize event for HLS. It would or may start black. I posted code above. Resize the renderer canvas within resize event.

@radiantmediaplayer
Copy link

radiantmediaplayer commented May 26, 2021

Just saying hello to this thread to mention that iOS 14.6 was just released and that the issue we have been seeing specifically affecting iPhone 12 devices has been resolved on our side with that release. Hopefully it will stay that way.

@danrossi
Copy link
Author

I just asked to really be sure if the fix is in 14.6 as we waited 3 months for it to appear in 14.5 and didn't happen. That bug ticket has been opened since August 2020. And from previous situations dealing with the same problem it can take up to 2 years. That is how much Apple love their own streaming format.

@NehaKushwah993
Copy link

Getting same issue for IOS Safari.. any solution found?

@danrossi
Copy link
Author

danrossi commented Nov 25, 2021

What version are you up to now. The last issue was Iphone not Ipad. It was fixed in 14.6. Have they regressed again ? Webkit dont have HLS textures for their tests.

@SebasScript
Copy link

I am having am still getting a security error trying to stream HLS as a video texture on IOS 15.3, also trying to use a normal .mp4 as source, without streaming is not working if the file is no fetched from the same URL that the application. So the only thing that works is bundling an mp4 with the app and loading it from the public folder at the moment. I added Cors headers to the steam but it still breaks.
The Streaming works for default

@danrossi
Copy link
Author

danrossi commented Feb 1, 2022

Use Inspect to remote debug. Check browser logs. I highly doubt they killed CORS since IOS11. I havent had a complaint yet about texture rendering or CORS. I have to check some wierd problem rendering WebRTC VR streams sometime this week and will confirm. My qemu macOS setup cant get a gpu connected so cant check webgl yet.

@SebasScript
Copy link

SebasScript commented Feb 1, 2022

thanks @danrossi, I did run remote inspect on an iphone 10, and I am getting a three.js security error:

three.webgl state security error the operation is not save

But as mentioned stream to to a video DOM element works fine

@danrossi
Copy link
Author

danrossi commented Feb 1, 2022

Post screenshot of console ? Sounds like it needs to be under HTTPS. It needs that to activate orientation api with the permission request.

@SebasScript
Copy link

Screenshot 2022-02-01 at 11 22 05 PM

@SebasScript
Copy link

i am using cloudflares video streaming which uses https

@SebasScript
Copy link

I got it to work by reengineering an example from the webkit issue that did track the crossorigin bug

https://bugs.webkit.org/show_bug.cgi?id=135379
example:
https://krpano.com/ios/bugs/ios8-webgl-video-cors/

the issue was the way I tried to set the crossorigin policy on the video element to anonymous. In the Mozilla docs the crossorigin attribute for the html media element is spelled in all small caps, but in the example link it was spelled as
crossOrigin. After setting the attribute in camel case writing the security error disappeared

@danrossi
Copy link
Author

danrossi commented Feb 1, 2022

lowercase seems to be fine when setting an attribute

 video.setAttribute("crossorigin", "anonymous");

@SebasScript
Copy link

interesting if you set it directly on the javascript object with video.crossOrigin, i have to use camel casing otherwise it will break

@mzvast
Copy link

mzvast commented Jun 14, 2023

Any update on this? similar to this issue

@danrossi
Copy link
Author

It doesn't matter how many work arounds I chuck at this Webkit always manage to break HLS textures because they don't use their own format as textures in the Webkit tests. I had to make some to report the bug. It has that first frame bug with HLS again on Iphone. If it gets cached it will appear on a second load. It's had to test for. I just saw it in browserstack. You have to clear cache to keep seeing it. On Ipad and macOS its ok. Iphone is always the problem the whole time.

You can try this for HLS.

video.addEventListener("resize", () => {
//resize video to videoWidth and videoHeight or ceil power of two of the width and height. Is power of 2 still needed ?
//resize canvas
});

If you click the picture in picture icon or fullscreen which is a video rendering fullscreen hack I just implemented and exit it shows. On Iphone picture in picture just goes fullscreen. On Ipad it goes into a pip window.

https://github.com/danrossi/canvas-pip-fullscreen

I am being harassed to fix the xr polyfill nobody wants to touch but people still want it as who knows what they are going to do with it in IOS 17. So that is more important right now. It broke with three end of 2019 it seems.

https://electroteque.org/plugins/videojs/vrvideo/demos/hls/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests