HLS on Safari / IOS now not working

[danrossi]

I apologise if there is nothing you can do. I am very certain in my tests HLS has been working and suddenly is now not.

I am using the CORS proxy hack here so CORS isn't a problem. I don't need to use the CORS proxy in WebView IOS apps and even there rendering is an issue.

Is there a WebGL fix that can be applied to get HLS rendering in WebGL ? I will try the canvas renderer to see if it helps. I know there is a requirement for double drawing of the canvas to get a frame up when using drawImage but this isn't an issue with Mp4 files.

The example is here

http://dev.electroteque.org/threejs/

[danrossi]

It is drawing to canvas and rendering with the canvas renderer albeit dropping frames. Something is required with webgl.

[danrossi]

This is a beauty. I have modified a kpano demo a little with the proxied HLS stream.

It is working now on Safari OSX but not on IOS 9. Any ideas what the difference is ? I am not an expert with the webgl flags sadly or what they even do. On IOS9 it's still a black frame.

http://dev.electroteque.org/threejs/webgl.html

[makc]

this last link is black rect / sound only in safari 9.1.3 / mac os x 10.11.6

[mrdoob]

What's HLS?

[danrossi]

@makc really ? I have OSX 10.10 and Safari 10. Well that explains the recent sabotage change by Apple. Safari 10. So if we're Safari 10 we're all good.

@mrdoob HLS = Apple Streaming. ie for live streams but also VOD. Live streaming is going to become a norm with the Terradeck gear. ie http://dev.electroteque.org/video/360/hls/ultra_light_flight.m3u8

Any ideas what the differences could be there ? I tried changing some things in the first example. The second example is raw webgl functions.

so texImage2D will partially render HLS but not on IOS yet maybe because it's not Safari 10 ?

[danrossi]

Please forgive me I have reported on a few webkit bug tickets.

They've gone and sabotaged the CORS proxy hack on an IOS update. macOS also is reported to not have fixed the CORS issue in Safari either.

I have to now figure out how to work around their sabotage. I have to get that working to further test HLS working on IOS. Apple is seriously making me work up a sweat to put out their fires hahah.

If I can get OSX rendering that would be a start. I just need to figure out what is different in three.js ?

[danrossi]

I'm still trying to do reduction methods to compare what is different. Once I figure that out I may get IOS working too.

Perhaps the extra shader programs within three.js is causing the rendering problems ? I'm not across it, but trying to replicate in the raw webgl example what three.js is doing in that equirectangular example. Is there example shader code for this ?

I found some raw functions here which hopefully I can replicate fully what three.js is doing. I can't just copy the shader programs directly because some variables are set externally like world position.

https://bl.ocks.org/mbostock/5446416

[dustinkerstein]

I may have just run across this or possibly a related issue. I have h264 mp4 files located on an S3 bucket and I am loading them to a video.src as a blob.

If I upload to my S3 bucket using the Amazon web-based uploader, the videos play fine when directly loaded in Mobile Safari (iOS 10.0.1), but if I try to play them using my three.js viewer then I just get a black screen. However, if I upload to the S3 bucket with Cyberduck, they play fine directly in Mobile Safari and in three.js.

It looks like Cyberduck is setting the meta-data 'Content-Type=video/mp4' and the AWS uploader is 'Content-Type=application/octet-stream'. When I manually set this meta-data to video/mp4 everything works correctly.

Strange issue, and one that caused me a lot of head scratching. Let me know if this is the same underlying issue as this ticket, or if I should create a new one.

Thanks,
Dustin

[danrossi]

This is for HLS mate as in mpegts fragments or fragmented files not mp4. Mp4 is fine on both Safari and IOS. I checked the mimetype in cyberduck and it says the mpegts is correct.

The same stream is working for me in the raw webgl example just not on IOS of course. Your issue seems similar to mine though as in a black frame. No rendering at all.

[dustinkerstein]

@danrossi ok cool. I'll create a new ticket specifically for my issue. Thanks.

[phaseOne]

I think this is the issue: www.krpano.com/ios/bugs/ios10-webgl-video-texture-crash/
I'm currently trying to implement the same workaround for three.js, but not getting very far.

[danrossi]

@phaseOne. It is a safari issue as both a problem on OSX and IOS. Mp4 is working fine for IOS and OSX. I'll tinker with antialias .

Still can't work out the differences between the two. The raw webgl example still won't work on IOS though but getting it working on OSX is a start.

I'm wondering if I can find a working equirectangle shader program that I can drop in.

[phaseOne]

More info here: http://stackoverflow.com/questions/39123109/rendering-a-video-with-webgl-in-ios-10-beta-7-safari-shows-weird-purplish-co.

I realize that this is a browser bug, but implementing a workaround might be necessary given how long these WebGL bugs take to get fixed.

[NishimuraT]

In the case to use the texture through the canvas, the video was displayed in safari.
However, since there are still CORS problems in iOS, it worked only in the Same Origin.
https://bugs.webkit.org/show_bug.cgi?id=135379

I'm not familiar with the webgl, but please refer to the following.
NishimuraT/videojs-panorama@bd99200

[danrossi]

@NishimuraT I think I know what you mean use the canavasrender. I did try that to do reduction method.

I think that was working but it causes severe drop frames and playback performance issues due to cpu. Obviously worse on IOS. Ipad 3 drops frames for webgl already.

[NishimuraT]

@danrossi I had a little misunderstanding. I'm sorry. I had sent a misguided thing.

[danrossi]

I have confirmed the same problem with the elevr project which uses raw webgl and shaders. Will have to hack that one to work my way back. Something in there is causing rendering issues and would affect three.js the same.

[danrossi]

I have confirmed in elevr it is auto clearing binding the texture which is causing it to break ie

webGL.gl.bindTexture(webGL.gl.TEXTURE_2D, null);

If this is commented out it is working.

This was also causing a problem, I'm not sure what it's for.

webGL.gl.pixelStorei(webGL.gl.UNPACK_FLIP_Y_WEBGL, true);

Maybe there is an auto clear property that is doing the same thing in three.js ?

[danrossi]

I see many references to

state.bindTexture(_gl.TEXTURE_2D, null);

But don't know where it could possibly be. Is it required to do that ?

[danrossi]

code is too crazy to track down what is causing the texture drawing issue. Seems to have something with this binding of texture.

[mrdoob]

@danrossi is there any page out there that renders it properly?

[danrossi]

Sorry about the noise mate. I did more AB testing. It is indeed it seems what I mentioned, as I had to comment it out for elevr demo also.

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, texture.flipY );

three.js/src/renderers/webgl/WebGLTextures.js

Line 411 in 6c7f000

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, texture.flipY );

This is causing the problem. However without it the texture is upside down. I'll add another test to show it now working.

[danrossi]

http://dev.electroteque.org/threejs/hlsbroken.html
http://dev.electroteque.org/threejs/hls.html

[danrossi]

Another raw example I found, commented out FLIP_Y. Possible Webkit bug now ?

http://dev.electroteque.org/threejs/webgl3.html

[danrossi]

The only reference to flipY is this but webkit nightly still an issue

https://bugs.webkit.org/show_bug.cgi?id=162491

If I disable flipY which I assume breaks hardware acceleration it is working ?

texture = new THREE.VideoTexture( video );
                texture.minFilter = THREE.LinearFilter;
                texture.format = THREE.RGBFormat;
                texture.magFilter = THREE.LinearFilter;
                texture.flipY = false;

http://dev.electroteque.org/threejs/hlsflipy.html

Trying to see if it's a webkit bug now. How is it possible to rotate the texture around to work around it for now ?

[makc]

How is it possible to rotate the texture around to work around it for now ?

why not just flip geometry uv or do that in the shader

[danrossi]

@makc I have no idea how to go about that.

Just to do heads in further. This FlipY "fix" doesn't help on IOS 9, just tested it.

Something else is a problem there now. It took days just to track this one down.

It may be all webkit's problem not Three.JS from the looks of it. As per usual so sorry about this.

I have filed a report there too https://bugs.webkit.org/show_bug.cgi?id=163866

[makc]

I have no idea how to go about that.

http://jsfiddle.net/hfj7gm6t/2182/

[danrossi]

It calls this. after capturing hls resize events

this.video.width = MathUtils.ceilPowerOfTwo(this.video.videoWidth);
this.video.height = MathUtils.ceilPowerOfTwo(this.video.videoHeight);

then resize

 this.camera.aspect = width / height;
 this.camera.updateProjectionMatrix();
 this.renderer.setSize(width, height);

[makc]

@danrossi so what are you saying is that the video has to be power-of-two or else it goes black? are you sure it is not just bcause you have bad filters / misconfigured mipmapping?

[danrossi]

It needs to be resized on HLS resize as the video width is 0 on metadata. Try some of those things out. resize video, resize renderer Sent with [ProtonMail](https://protonmail.com/) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Saturday, February 13th, 2021 at 2:33 PM, makc ***@***.***> wrote: ***@***.***(https://github.com/danrossi) so what are you saying is that the video has to be power-of-two or else it goes black? are you sure it is not just bcause you have bad filters / misconfigured mipmapping? — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#9754 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAFNNBIE6KBQDXRXVDZ42DDS6XXHVANCNFSM4CQTGPNQ).

[danrossi]

Somebody is now observing my fix works on Ipad which is the only device I have access to test. But Iphone IOS 14, the rendering won't appear. Its not an overall webgl bug like before. It's pathetic regardless though. I don't have access to an Iphone to test it.

My texture is setup like.

        var texture = new Texture(video);
        texture.format = RGBFormat;
        texture.minFilter = LinearFilter;
        texture.magFilter = LinearFilter;
        texture.wrapS = ClampToEdgeWrapping;
        texture.wrapT = ClampToEdgeWrapping;
        texture.generateMipmaps = false;

[danrossi]

Somebody at the bottom here claims it might be "fixed" again in IOS 14.2 beta but on Ipad. But a client says 14.4 on Iphone IOS is still a problem. Webkit don't use HLS textures in their test files. When I first reported I had to modify them with HLS to demonstrate the problem. There may still be a problem with Iphone for who knows what reason.

https://bugs.webkit.org/show_bug.cgi?id=215908

[danrossi]

Fix is in webkit but not in IOS as usual. Might not be released until 14.5. So no point testing fix on Ipad but Iphone.

[arpu]

@danrossi do you found a workaround for iphone ios 14 ? i follow the bugreport https://bugs.webkit.org/show_bug.cgi?id=215908
tested ipone 11 ios 13 works fine ios 14 not ipad 14 works ( tested with Browserstack)

[danrossi]

I'm working on this feature again but to upgrade issues with the webxr polyfill not working with current three, and my virtual controlbar not working with the old vr controller code now.

I will get back to checking HLS in IOS 14.5 but it seems some guy just reported its still not working in beta. I'm not killing myself over Apple antics anymore. It took me 3 months originally in this same thread to find shader work arounds. I don't think this is a shader issue, as its working on IPad 14.4. Its just Webkit still do not use HLS as textures in their webgl tests so they keep breaking things !

Things may change if they finally add WebXR into Safari ! Interest for VR has dropped the past few years, but people still want it, for things like street mapping etc.

I don't use Apple products and have to borrow them for testing. I'll get IOS 14.5 on an Iphone if 14.5 is even released. I think you have to wait for 14.5 to be released to truely see it working again.

[danrossi]

The goal posts have been moved. Iphone 12 fix in IOS 14.6 ! Another 6 months wait. So a full year wait which is standard for IOS. All because they don't test HLS textures. They will keep making these mistakes I think. The original time was enough.

[arpu]

@danrossi where can i find this info for 14.6 fix ?

[danrossi]

In the webkit thread. I thought it was over and done with in IOS 14.5. Now you have to wait another 6 months for 14.6. I don't understand why they can't just auto update Safari ! Unless its an IOS specific problem.

[arpu]

ah thx i do not see the new response in the thread, but still not understand why its not working on iphone 11 and ios 14.5 any idea?

[danrossi]

No point trying to pull up different devices and revisions just wait for IOS 14.6. I might try and get a hold of an Iphone 12 and put 14.5 on it and run some tests. I have no access to actually see the bug yet it works on Ipad. If there is a webgl flag or shader trick I'll report it. But in 14.4 I managed to get it rendering on startup using the video resize event for HLS. It would or may start black. I posted code above. Resize the renderer canvas within resize event.

[radiantmediaplayer]

Just saying hello to this thread to mention that iOS 14.6 was just released and that the issue we have been seeing specifically affecting iPhone 12 devices has been resolved on our side with that release. Hopefully it will stay that way.

[danrossi]

I just asked to really be sure if the fix is in 14.6 as we waited 3 months for it to appear in 14.5 and didn't happen. That bug ticket has been opened since August 2020. And from previous situations dealing with the same problem it can take up to 2 years. That is how much Apple love their own streaming format.

[NehaKushwah993]

Getting same issue for IOS Safari.. any solution found?

[danrossi]

What version are you up to now. The last issue was Iphone not Ipad. It was fixed in 14.6. Have they regressed again ? Webkit dont have HLS textures for their tests.

[SebasScript]

I am having am still getting a security error trying to stream HLS as a video texture on IOS 15.3, also trying to use a normal .mp4 as source, without streaming is not working if the file is no fetched from the same URL that the application. So the only thing that works is bundling an mp4 with the app and loading it from the public folder at the moment. I added Cors headers to the steam but it still breaks.
The Streaming works for default

[danrossi]

Use Inspect to remote debug. Check browser logs. I highly doubt they killed CORS since IOS11. I havent had a complaint yet about texture rendering or CORS. I have to check some wierd problem rendering WebRTC VR streams sometime this week and will confirm. My qemu macOS setup cant get a gpu connected so cant check webgl yet.

[SebasScript]

thanks @danrossi, I did run remote inspect on an iphone 10, and I am getting a three.js security error:

three.webgl state security error the operation is not save

But as mentioned stream to to a video DOM element works fine

[danrossi]

Post screenshot of console ? Sounds like it needs to be under HTTPS. It needs that to activate orientation api with the permission request.

[SebasScript]

[SebasScript]

i am using cloudflares video streaming which uses https

[SebasScript]

I got it to work by reengineering an example from the webkit issue that did track the crossorigin bug

https://bugs.webkit.org/show_bug.cgi?id=135379
example:
https://krpano.com/ios/bugs/ios8-webgl-video-cors/

the issue was the way I tried to set the crossorigin policy on the video element to anonymous. In the Mozilla docs the crossorigin attribute for the html media element is spelled in all small caps, but in the example link it was spelled as
crossOrigin. After setting the attribute in camel case writing the security error disappeared

[danrossi]

lowercase seems to be fine when setting an attribute

 video.setAttribute("crossorigin", "anonymous");

[SebasScript]

interesting if you set it directly on the javascript object with video.crossOrigin, i have to use camel casing otherwise it will break

[mzvast]

Any update on this? similar to this issue

[danrossi]

It doesn't matter how many work arounds I chuck at this Webkit always manage to break HLS textures because they don't use their own format as textures in the Webkit tests. I had to make some to report the bug. It has that first frame bug with HLS again on Iphone. If it gets cached it will appear on a second load. It's had to test for. I just saw it in browserstack. You have to clear cache to keep seeing it. On Ipad and macOS its ok. Iphone is always the problem the whole time.

You can try this for HLS.

video.addEventListener("resize", () => {
//resize video to videoWidth and videoHeight or ceil power of two of the width and height. Is power of 2 still needed ?
//resize canvas
});

If you click the picture in picture icon or fullscreen which is a video rendering fullscreen hack I just implemented and exit it shows. On Iphone picture in picture just goes fullscreen. On Ipad it goes into a pip window.

https://github.com/danrossi/canvas-pip-fullscreen

I am being harassed to fix the xr polyfill nobody wants to touch but people still want it as who knows what they are going to do with it in IOS 17. So that is more important right now. It broke with three end of 2019 it seems.

https://electroteque.org/plugins/videojs/vrvideo/demos/hls/