Video completely distorted and glitchy #3

LeonardRockstar · 2017-09-07T01:50:57Z

When using SceneKitVideoRecorder on an ARSCNView, the video is completely distorted.
This is happening on my iPhone 7 Plus running Beta 10.

okaris · 2017-09-07T07:08:29Z

Wow. Strange yet cool. Will update iOS now to see whats wrong.

okaris · 2017-09-07T08:06:59Z

Tested on iPhone SE iOS 11 Beta 10. Can’t reproduce.

LeonardRockstar · 2017-09-07T11:28:46Z

Weird, I've even created a new AR Kit project and I still see the error. What device were you using?

okaris · 2017-09-07T11:30:08Z

I was on iPhone SE
Can you share your setup options and pod version?

LeonardRockstar · 2017-09-07T11:41:38Z

I was testing on an iPhone 7 Plus with SceneKitRecorder 1.0.3. I created a new Xcode Project with the ARKit template.

LeonardRockstar · 2017-09-07T14:52:33Z

I've just tested the same project on my iPad Pro. The video is not distorted, but it is really choppy.

okaris · 2017-09-08T13:14:25Z

Can you test this issue with 1.2.0 please?

okaris · 2017-09-08T13:45:41Z

As you showed there is a bug in 1.2.0
Can you try this with 1.1.0 until i fix 1.2.0 please

LeonardRockstar · 2017-09-08T14:00:30Z

The issue still persists with 1.1.0

okaris · 2017-09-08T19:47:15Z

Please try with sample configuration. Don't forget to add prepare() on viewDidAppear

LeonardRockstar · 2017-09-09T12:12:35Z

I've tried the sample configuration, but the video still looks like that.

mariedm · 2017-09-10T14:28:45Z

I have the same issue. I'm using an iPhone 6S on iOS 11, I think I was on the 1st beta. I'm currently updating to the last one (beta 10). I'll retry after and tell you.

mariedm · 2017-09-10T20:00:06Z

I updated my iPhone 6S to iOS 11 beta 10. I retested and I'm still having the same issue.

okaris · 2017-09-10T20:43:16Z

I have a strange theory and a really bad way to test it.

@LeonardRockstar @mariedm
Can you paste the following code on line 21 in PixelBufferFactory.swift just under let destinationTexture = currentDrawable.texture

switch destinationTexture.pixelFormat {
        case .invalid: print("invalid"); break;
        case .a8Unorm: print("a8Unorm"); break;
        case .r8Unorm: print("r8Unorm"); break;
        case .r8Unorm_srgb: print("r8Unorm_srgb"); break;
        case .r8Snorm: print("r8Snorm"); break;
        case .r8Uint: print("r8Uint"); break;
        case .r8Sint: print("r8Sint"); break;
        case .r16Snorm: print("r16Snorm"); break;
        case .r16Uint: print("r16Uint"); break;
        case .r16Sint: print("r16Sint"); break;
        case .r16Float: print("r16Float"); break;
        case .rg8Unorm: print("rg8Unorm"); break;
        case .rg8Snorm: print("rg8Snorm"); break;
        case .rg8Uint: print("rg8Uint"); break;
        case .rg8Sint: print("rg8Sint"); break;
        case .a1bgr5Unorm: print("a1bgr5Unorm"); break;
        case .abgr4Unorm: print("abgr4Unorm"); break;
        case .bgr5A1Unorm: print("bgr5A1Unorm"); break;
        case .r32Sint: print("r32Sint"); break;
        case .r32Float: print("r32Float"); break;
        case .rg16Unorm: print("rg16Unorm"); break;
        case .rg16Snorm: print("rg16Snorm"); break;
        case .rg16Uint: print("rg16Uint"); break;
        case .rg16Sint: print("rg16Sint"); break;
        case .rg16Float: print("rg16Float"); break;
        case .rgba8Unorm: print("rgba8Unorm"); break;
        case .rgba8Unorm_srgb: print("rgba8Unorm_srgb"); break;
        case .rgba8Snorm: print("rgba8Snorm"); break;
        case .rgba8Uint: print("rgba8Uint"); break;
        case .rgba8Sint: print("rgba8Sint"); break;
        case .bgra8Unorm: print("bgra8Unorm"); break;
        case .bgra8Unorm_srgb: print("bgra8Unorm_srgb"); break;
        case .rgb10a2Unorm: print("rgb10a2Unorm"); break;
        case .rgb10a2Uint: print("rgb10a2Uint"); break;
        case .rg11b10Float: print("rg11b10Float"); break;
        case .rgb9e5Float: print("rgb9e5Float"); break;
        case .bgr10a2Unorm: print("bgr10a2Unorm"); break;
        case .bgr10_xr: print("bgr10_xr"); break;
        case .bgr10_xr_srgb: print("bgr10_xr_srgb"); break;
        case .rg32Sint: print("rg32Sint"); break;
        case .rg32Float: print("rg32Float"); break;
        case .rgba16Unorm: print("rgba16Unorm"); break;
        case .rgba16Snorm: print("rgba16Snorm"); break;
        case .rgba16Uint: print("rgba16Uint"); break;
        case .rgba16Sint: print("rgba16Sint"); break;
        case .rgba16Float: print("rgba16Float"); break;
        case .BGRA10_XR: print("BGRA10_XR"); break;
        case .bgra10_XR_sRGB: print("bgra10_XR_sRGB"); break;
        case .rgba32Sint: print("rgba32Sint"); break;
        case .rgba32Float: print("rgba32Float"); break;
        case .pvrtc_rgb_2bpp_srgb: print("pvrtc_rgb_2bpp_srgb"); break;
        case .pvrtc_rgb_4bpp: print("pvrtc_rgb_4bpp"); break;
        case .pvrtc_rgb_4bpp_srgb: print("pvrtc_rgb_4bpp_srgb"); break;
        case .pvrtc_rgba_2bpp: print("pvrtc_rgba_2bpp"); break;
        case .pvrtc_rgba_2bpp_srgb: print("pvrtc_rgba_2bpp_srgb"); break;
        case .pvrtc_rgba_4bpp: print("pvrtc_rgba_4bpp"); break;
        case .pvrtc_rgba_4bpp_srgb: print("pvrtc_rgba_4bpp_srgb"); break;
        case .eac_r11Snorm: print("eac_r11Snorm"); break;
        case .eac_rg11Unorm: print("eac_rg11Unorm"); break;
        case .eac_rg11Snorm: print("eac_rg11Snorm"); break;
        case .eac_rgba8: print("eac_rgba8"); break;
        case .eac_rgba8_srgb: print("eac_rgba8_srgb"); break;
        case .etc2_rgb8: print("etc2_rgb8"); break;
        case .etc2_rgb8_srgb: print("etc2_rgb8_srgb"); break;
        case .etc2_rgb8a1: print("etc2_rgb8a1"); break;
        case .etc2_rgb8a1_srgb: print("etc2_rgb8a1_srgb"); break;
        case .astc_5x4_srgb: print("astc_5x4_srgb"); break;
        case .astc_5x5_srgb: print("astc_5x5_srgb"); break;
        case .astc_6x5_srgb: print("astc_6x5_srgb"); break;
        case .astc_6x6_srgb: print("astc_6x6_srgb"); break;
        case .astc_8x5_srgb: print("astc_8x5_srgb"); break;
        case .astc_8x6_srgb: print("astc_8x6_srgb"); break;
        case .astc_8x8_srgb: print("astc_8x8_srgb"); break;
        case .astc_10x5_srgb: print("astc_10x5_srgb"); break;
        case .astc_10x6_srgb: print("astc_10x6_srgb"); break;
        case .astc_10x8_srgb: print("astc_10x8_srgb"); break;
        case .astc_10x10_srgb: print("astc_10x10_srgb"); break;
        case .astc_12x10_srgb: print("astc_12x10_srgb"); break;
        case .astc_12x12_srgb: print("astc_12x12_srgb"); break;
        case .astc_4x4_ldr: print("astc_4x4_ldr"); break;
        case .astc_5x4_ldr: print("astc_5x4_ldr"); break;
        case .astc_5x5_ldr: print("astc_5x5_ldr"); break;
        case .astc_6x5_ldr: print("astc_6x5_ldr"); break;
        case .astc_6x6_ldr: print("astc_6x6_ldr"); break;
        case .astc_8x5_ldr: print("astc_8x5_ldr"); break;
        case .astc_8x6_ldr: print("astc_8x6_ldr"); break;
        case .astc_8x8_ldr: print("astc_8x8_ldr"); break;
        case .astc_10x5_ldr: print("astc_10x5_ldr"); break;
        case .astc_10x6_ldr: print("astc_10x6_ldr"); break;
        case .astc_10x8_ldr: print("astc_10x8_ldr"); break;
        case .astc_10x10_ldr: print("astc_10x10_ldr"); break;
        case .astc_12x10_ldr: print("astc_12x10_ldr"); break;
        case .astc_12x12_ldr: print("astc_12x12_ldr"); break;
        case .r16Unorm: print("r16Unorm"); break;
        case .rg8Unorm_srgb: print("rg8Unorm_srgb"); break;
        case .b5g6r5Unorm: print("b5g6r5Unorm"); break;
        case .r32Uint: print("r32Uint"); break;
        case .rg32Uint: print("rg32Uint"); break;
        case .rgba32Uint: print("rgba32Uint"); break;
        case .bc1_rgba: print("bc1_rgba"); break;
        case .bc1_rgba_srgb: print("bc1_rgba_srgb"); break;
        case .bc2_rgba: print("bc2_rgba"); break;
        case .bc2_rgba_srgb: print("bc2_rgba_srgb"); break;
        case .bc3_rgba: print("bc3_rgba"); break;
        case .bc3_rgba_srgb: print("bc3_rgba_srgb"); break;
        case .bc4_rUnorm: print("bc4_rUnorm"); break;
        case .bc4_rSnorm: print("bc4_rSnorm"); break;
        case .bc5_rgUnorm: print("bc5_rgUnorm"); break;
        case .bc5_rgSnorm: print("bc5_rgSnorm"); break;
        case .bc6H_rgbFloat: print("bc6H_rgbFloat"); break;
        case .bc6H_rgbuFloat: print("bc6H_rgbuFloat"); break;
        case .bc7_rgbaUnorm: print("bc7_rgbaUnorm"); break;
        case .bc7_rgbaUnorm_srgb: print("bc7_rgbaUnorm_srgb"); break;
        case .pvrtc_rgb_2bpp: print("pvrtc_rgb_2bpp"); break;
        case .eac_r11Unorm: print("eac_r11Unorm"); break;
        case .astc_4x4_srgb: print("astc_4x4_srgb"); break;
        case .gbgr422: print("gbgr422"); break;
        case .bgrg422: print("bgrg422"); break;
        case .depth16Unorm: print("depth16Unorm"); break;
        case .depth32Float: print("depth32Float"); break;
        case .stencil8: print("stencil8"); break;
        case .depth24Unorm_stencil8: print("depth24Unorm_stencil8"); break;
        case .depth32Float_stencil8: print("depth32Float_stencil8"); break;
        case .x32_stencil8: print("x32_stencil8"); break;
        case .x24_stencil8: print("x24_stencil8"); break;
      }

This huge switch is because printing the enum or its rawValue didn't work. I am getting bgra8Unorm_srgb and my theory is that somehow yours is different and causing this psychedelic recording.

LeonardRockstar · 2017-09-10T21:12:07Z

Just tried it, it keeps printing "invalid".

okaris · 2017-09-10T21:38:52Z

Interesting. Are you getting a whole video like this, or do you crash or do you have a video with one frame frozen?

LeonardRockstar · 2017-09-10T21:48:50Z

I'm getting the whole video like that.

okaris · 2017-09-10T23:13:16Z

Another idea: can you try with 30fps video please. This is really hard to debug when i cant reproduce the issue

okaris · 2017-09-10T23:13:38Z

@mariedm are you using 6s or 6s plus?

LeonardRockstar · 2017-09-10T23:18:30Z

Still looks the same when using 30fps.

okaris · 2017-09-10T23:31:14Z

@LeonardRockstar can you put the switch to SceneKitVideoRecorder.swift under line 74 ...frameBufferOnly = false

and also changing switch to

switch metalLayer.pixelFormat {

LeonardRockstar · 2017-09-10T23:40:47Z

It prints bgra8Unorm_srgb

okaris · 2017-09-10T23:48:56Z

That looks correct. One more thing to try. Can you pass in a block to updateFrameHandler of SceneKitVideoRecorder and get an image from there to see if its the same with the video.

LeonardRockstar · 2017-09-11T00:25:00Z

I don't know if I understand you correctly. The UIImage returned by PixelBufferFactory.make (and passed into updateFrameHandler) is also distorted, but looks a lot brighter than the video.

GoodbyeCain · 2017-09-11T07:32:35Z

same issue

okaris · 2017-09-11T07:33:41Z

@GoodbyeCain can you please share your device, os version, pod version so we can check if a pattern is forming.

mariedm · 2017-09-11T08:01:33Z

I’m using a regular 6S.
In PixelBufferFactory: the switch statement on metalLayer.pixelFormat prints invalid.
In SceneKitVideoRecorder: the switch statement on metalLayer.pixelFormat after framebufferOnly (line 64 for me) also prints invalid.

LeonardRockstar · 2017-09-14T00:19:30Z

I'm getting the same error as @sanboxapps.
The output is: 1932 1080.0 (1080.0, 1867.0)

mariedm · 2017-09-14T08:28:16Z

I updated the pod, when calling startWriting() I got Error Domain=SceneKitVideoRecorder.SceneKitVideoRecorder.PreparationError Code=1 "(null)".

I readded both switch statements. The one in SceneKitVideoRecorder is now printing bgra8Unorm_srgb(instead of invalid before).

Then in the Options file I changed kCVPixelFormatType_32BGRA to kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and still got the same error, I can't record.

okaris · 2017-09-14T08:30:08Z

@mariedm if you are seeing bgra8Unorm_srgb as pixelFormat I don't know why but it should be fixed for you. Can you run the demo project in the latest pod please.

mariedm · 2017-09-14T09:26:36Z

I didn't manage to run the demo project but I recreated it. I could record and I have the audio (!) but the video is distorded.
The switch statement in SceneKitVideoRecorder prints bgra8Unorm_srgb and the one in PixelBufferFactory prints invalid.

If I change to 30 fps I get the same outputs.

If I change to kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange in Options I get the same outputs + _validateGetBytes:95: failed assertion rowBytes(1156) must be >= (1500) in PixelBufferFactory when it crashed on the line destinationTexture.getBytes(tempBuffer!, bytesPerRow: Int(bytesPerRow), from: region, mipmapLevel: 0).

okaris · 2017-09-15T13:42:01Z

Please try this when the video is colorful like above and share the output. Thanks all!

print(bytesPerRow, currentDrawable.layer.drawableSize.width, currentDrawable.layer.drawableSize) to PixelBufferFactory.swift under line #28 let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)

mariedm · 2017-09-15T14:56:43Z

This is the output: 3008 750.0 (750.0, 1294.0).

okaris · 2017-09-17T14:45:49Z

@warrenm I found your answer about Metal Video Recording on StackOverflow. This issue looks like a corruption you wrote on SO. Can you help us solve it?

ghost · 2017-09-17T18:55:17Z

Sure. The texture whose bytes you're copying probably has a format of MTLPixelFormatRGB10A8_2P_XR10_sRGB or something. As I said very explicitly in my answer on SO, that snippet depends on the source format being MTLPixelFormatBGRA8Unorm. You'll either need to figure out how to set the pixel format of the CAMetalLayer underlying the Scene Kit view to have that format, or else run a conversion step (fragment shader, compute shader, vImage, etc.) that transforms the pixel data into the necessary format. Naively copying from the drawable texture into a pixel buffer isn't going to cut it in this case.

okaris · 2017-09-17T19:18:25Z

@warrenm Thank you very much for your guidance. It provided some perspective to try other solutions.

@mariedm @LeonardRockstar @sanboxapps @GoodbyeCain Can you please run the latest (1.2.7) version of this project and let me know about the results please?

LeonardRockstar · 2017-09-17T19:43:37Z

It crashed at line 20 in PixelBufferFactory: var destinationTexture = currentDrawable.texture.makeTextureView(pixelFormat: .bgra8Unorm)

with the error: validateArgumentsForTextureViewOnDevice, line 1037: error 'source texture pixelFormat (MTLPixelFormatRGB10A8_2P_XR10_sRGB) not compatible with texture view pixelFormat (MTLPixelFormatBGRA8Unorm).' validateArgumentsForTextureViewOnDevice:1037: failed assertion source texture pixelFormat (MTLPixelFormatRGB10A8_2P_XR10_sRGB) not compatible with texture view pixelFormat (MTLPixelFormatBGRA8Unorm).'`

okaris · 2017-09-17T20:05:26Z

@warrenm Can you suggest a source for how to convert these pixelFormats to one another?
Especially I can not seem to find any information or documentation online for RGB10A8_2P_XR10_sRGB pixelFormat. The closest one I got was bgra10_XR_sRGB which is not probably close enough.

Also I don't believe that this pixelFormat is something should be returned by a MTLTexture coming from a SceneKit scene CAMetalLayer. Documentation tells the only following should be returned:

.bgra8Unorm
.bgra8Unorm_srgb
.rgba16Float

with bgra8Unorm_srgb being the default.

mariedm · 2017-09-18T12:47:52Z

I have the same crash & error outputs than @LeonardRockstar.

ghost · 2017-09-18T17:17:10Z

@okaris It's unfortunate that this breaking change occurred, and that neither the change nor the format itself is documented. I don't know if XR biplanar format can be read or sampled by a shader, but if it can, you could bind the framebuffer as a source texture and "re-render" the frame to a bgra8Unorm target (with a full-screen quad pass) in order to get around this limitation. That will measurably impact the performance of the recorder. You might be able to recover part of the framerate hit by not taking a copy and building a UIImage every frame; that's an extremely expensive operation that doesn't seem to be necessary for the primary function of the framework.

okaris · 2017-09-19T04:36:33Z

Thank you @warren. I am still working on the pixel format but dropping UIImage helped for the performance.

@mariedm @LeonardRockstar @sanboxapps @GoodbyeCain
I am discussing this with Apple Developer Technical Support. Can you please confirm that this is reproducable with iOS11 GM and XCode9 GM. And also please include your device model. Thank you.

ghost · 2017-09-19T04:38:10Z

I can reproduce this readily on iPhone 7, iOS 11 GM.

mariedm · 2017-09-19T08:07:29Z

I am actually working on xCode 9 GM with an iPhone 7 on iOS 11 GM.

LeonardRockstar · 2017-09-19T12:19:59Z

I'm using the Xcode 9 GM and iOS 11 GM on an iPhone 7 Plus.

okaris · 2017-09-20T23:53:03Z

Response from Apple:

After some investigation into the pixel format and how it is encountered by your library at run time, we’ve determined that your current approach is not recommended and you should consider alternatives.

In particular, the currentDrawable’s texture vended by SceneKit is not intended to be used this way and the texture format could change at any time. This is partly why this format is undocumented.
I’m unable to disclose more, and don’t want to encourage you further down a non-viable path, so that’s all there is to say about the current approach.

I am planning on changing this core functionality over the weekend.
I am sorry that we could not find a viable solution.

okaris · 2017-09-21T19:06:26Z

Please try the latest version (1.3.1)

LeonardRockstar · 2017-09-21T19:41:38Z

The distortion is gone on my iPhone 7 Plus!!

mariedm · 2017-09-22T08:30:36Z

I can't make the new version work. I'm probably doing something wrong.

I call recorder=try! SceneKitVideoRecorder(withARSCNView: sceneView) in viewDidLoad(). Then recorder.setupMicrophone() and recorder.startWriting() when pushing the record button. But startDisplayLink() is never called. So when finishWriting is called, isRecording is false and it returns without stoping the recording.
(So when I push the record button again I get Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '*** -[AVAssetWriter addInput:] Cannot call method when status is 3').

okaris · 2017-09-22T08:34:18Z

@mariedm I just updated the demo project. Please try that and if that doesn't work please open a new issue.

mariedm · 2017-09-22T08:41:09Z

The only difference is the setupMicrophone() moved to viewWillAppear which I tried and it's still not working. I'll open a new issue.
Congrats for the image distortion btw!

okaris self-assigned this Sep 7, 2017

okaris added Can’t Reproduce Info Needed labels Sep 7, 2017

okaris mentioned this issue Sep 19, 2017

Crash when trying to record ARCL #12

Closed

okaris mentioned this issue Sep 21, 2017

Crash on recording <signal SIGABRT> #13

Closed

okaris closed this as completed Sep 22, 2017

svhawks locked and limited conversation to collaborators Sep 27, 2017

okaris added bug and removed Can’t Reproduce Info Needed labels Sep 27, 2017

Video completely distorted and glitchy #3

Video completely distorted and glitchy #3

Comments

LeonardRockstar commented Sep 7, 2017

okaris commented Sep 7, 2017

okaris commented Sep 7, 2017

LeonardRockstar commented Sep 7, 2017

okaris commented Sep 7, 2017

LeonardRockstar commented Sep 7, 2017

LeonardRockstar commented Sep 7, 2017

okaris commented Sep 8, 2017

okaris commented Sep 8, 2017

LeonardRockstar commented Sep 8, 2017

okaris commented Sep 8, 2017

LeonardRockstar commented Sep 9, 2017

mariedm commented Sep 10, 2017

mariedm commented Sep 10, 2017

okaris commented Sep 10, 2017 • edited Loading

LeonardRockstar commented Sep 10, 2017

okaris commented Sep 10, 2017

LeonardRockstar commented Sep 10, 2017

okaris commented Sep 10, 2017

okaris commented Sep 10, 2017

LeonardRockstar commented Sep 10, 2017

okaris commented Sep 10, 2017

LeonardRockstar commented Sep 10, 2017

okaris commented Sep 10, 2017

LeonardRockstar commented Sep 11, 2017

GoodbyeCain commented Sep 11, 2017

okaris commented Sep 11, 2017

mariedm commented Sep 11, 2017

LeonardRockstar commented Sep 14, 2017

mariedm commented Sep 14, 2017 • edited Loading

okaris commented Sep 14, 2017

mariedm commented Sep 14, 2017 • edited Loading

okaris commented Sep 15, 2017

mariedm commented Sep 15, 2017

okaris commented Sep 17, 2017

ghost commented Sep 17, 2017

okaris commented Sep 17, 2017

LeonardRockstar commented Sep 17, 2017

okaris commented Sep 17, 2017 • edited Loading

mariedm commented Sep 18, 2017

ghost commented Sep 18, 2017

okaris commented Sep 19, 2017

ghost commented Sep 19, 2017

mariedm commented Sep 19, 2017

LeonardRockstar commented Sep 19, 2017

okaris commented Sep 20, 2017 • edited Loading

okaris commented Sep 21, 2017

LeonardRockstar commented Sep 21, 2017

mariedm commented Sep 22, 2017 • edited Loading

okaris commented Sep 22, 2017

mariedm commented Sep 22, 2017

okaris commented Sep 10, 2017 •

edited

Loading

mariedm commented Sep 14, 2017 •

edited

Loading

mariedm commented Sep 14, 2017 •

edited

Loading

okaris commented Sep 17, 2017 •

edited

Loading

okaris commented Sep 20, 2017 •

edited

Loading

mariedm commented Sep 22, 2017 •

edited

Loading