VCE2.0 Firmware bug causes Crash or Artifacting

[razorlikes]

Hey Xaymar, here is a bug I found while playing with different settings. OBS crashes when VBV Buffer Fullness is set to 80. An interesting thing I noticed was, that two out of three times I tried this, OBS crashed after exactly 89,6 MB of video were recorded.

Here are my settings.

Crash 2016-11-11 22-32-15.txt
2016-11-11 22-28-34.txt

Hardware:
AMD FX8350
AMD Radeon R9 390 Nitro
ASUS Crosshair IV Formula
Corsair XMS3 DDR3 12GB
SSD for OS, HDDs for recording and games.

OBS version 0.16.5
Plugin version 1.4.1.5

[Xaymar]

Plugin version 1.4.1.5

Don't lie, the log file will betray you:

22:28:35.409: [AMF Encoder] Version 1.4.1.2-1.3.0.5

Update to 1.4.1.5, reproduce it and upload a new log file, crash log and the raw video file it produces.
If you can, create a memory dump as OBS shows the program crashed window. If you don't know how to do that, open Task Manager to the Details tab, right click on obs64.exe/obs32.exe and click "Create dump file". Having that can help with tracking down what crashed and why.

Please do not upload the video to youtube or twitch, that will not help anyone.

[razorlikes]

I could swear I updated to 1.4.1.5... Maybe OBS overwrote the plugin when I updated it today.

So I updated to 1.4.1.5 now, did a 20 minute recording and wasn't able to reproduce the error for now. I will try more tomorrow but I think its fixed in the update!

[razorlikes]

Okay, I just did some more tests and its most definitely fixed!
Also a VBV Fullness of 0 looks so much better than the 90 or 80 I had before...

[Xaymar]

You should fix your VBV Buffer Strictness and set it lower than 80%. I have mine at 66% so it's still strict but allows for a little bitrate variation.

[razorlikes]

I'm again getting crashes. Uploading the video right now and will update and reopen this issue when everything is uploaded.
Do you need a full dump of OBS or is a minidump okay? I have made both but the full dump has a size of 1 GB.

[Xaymar]

Full dump, minidump does not contain enough information. You can compress the full dump with 7z or winrar and it'll lose about ~20-40% in size

[razorlikes]

Settings
2016-11-12 15-23-26.txt
Crash 2016-11-12 15-35-46.txt
Video
Dump

[Xaymar]

Crash happened inside AMD Driver, will report this to the AMD Team working on the Encoder.

Edit: Please do not delete the files of your Google Drive until it is fixed. Thank you!

[razorlikes]

Another thing to note may be that I left an OC on my VRAM. Its on 1675 right now, which worked fine with Plays.TV and your plugin before. When I also OCed my Core I would get artifacts in Plays' recordings. I also have my Power Limit set to +50% but that never changed anything.

[Xaymar]

Try 16.11.4 and report back.

[razorlikes]

Updated the plugin and my drivers and recorded for ~40 minutes without any errors.
Although I'm getting little artifacts now and then when playing the video in VLC, those were present in other videos recorded with your plugin too but never seemed to show up during streams. Also WMP won't play the video at all. I'm using the MKV container.
Is this just some kind of error within VLC or is it something else?

Another thing would be B-Picture Delta QP values. I'm using 0 for both Reference and normal B-Picture Delta QP right now. Are those setting acceptable or is there some recommended value for those?

[Xaymar]

Only MPC-HC, ffmpeg and VLC seem to be able to play back the files. I don't know why other players have such a hard time.

As for delta QP, 0/0 is acceptable, the default for them is +4/+2. You can also use -2/-4 if you want to get better quality in B-Frames. I'd recommend not using B-Frames for now though, there are some quality issues with them that only go away at higher bitrates.

[razorlikes]

Okay, I'll probably turn them off for now then.
But just so you get an idea what artifacts I'm talking about, I found an example where they are pretty prominent: http://i.imgur.com/zlqn1VM.jpg
It just looks like small parts of the image get placed at the wrong position.
I also noticed that those artifacts mostly happen in the beginning of the recording and my bitrate fluctuates extremely in the beginning of the recording: http://i.imgur.com/3K6Yv2I.jpg

[Xaymar]

That looks like decoding failed for those macroblocks, I've never seen that happen. I have seen the bitrate fluctuation, but that usually fixes itself after 15-17 seconds.

[razorlikes]

I'm doing a little more testing right now and it seems like it mostly happens when I have no or just very little movement on my screen.

[razorlikes]

While testing I just got another crash.

Crash 2016-11-18 21-05-44.txt
2016-11-18 21-03-32.txt

Video: http://5.230.156.11/nextcloud/index.php/s/yZwUeK31yFHScMN

Unfortunately I forgot to take a dump.
In the video you can see that it starts lagging really hard when I just look at the wall at ~1:00, things like this usually happen in most of my test recordings when loading screens come up. The "tearing" at the end of the video is also often happening when there is no movement.

[razorlikes]

Here is another video where it is very noticeable at around 2:05. This screen-tear + lag thing happens almost every time the movement stops, for example in elevators/loading screens.

http://5.230.156.11/nextcloud/index.php/s/Iub9IB2wGQA4gm5
And this should be the corresponding log: 2016-11-18 20-46-31.txt

And to make it all complete, here are the settings I used when recording both videos: http://i.imgur.com/L73P4r9.jpg

[Xaymar]

That's the section where you stare at the wall, it looks like for whatever reason it stopped encoding. The log file at that point also matches the output, with the average size being less than 500 bytes per packet. This isn't exactly something that should happen.

Could you set the VBV Buffer Strictness to 0 and Enforce HRD Restrictions set to Enabled?

[razorlikes]

Did that, uploading the video right now but it happened again...

[razorlikes]

Here you go.
Log: 2016-11-18 21-30-50.txt
Video: http://5.230.156.11/nextcloud/index.php/s/c7eAMUuGmHlm6Vv

At 1:00 I again looked at a wall and at 3:05 there is a loading screen.

[razorlikes]

Any news on this issue or any idea what may be causing this?

[Xaymar]

If you mean the crash, it still happens with 16.11.4. If you mean the encoding failure, I have yet to be able to reproduce it.

[razorlikes]

Got another crash today, this time I took a dump of the process although I think this is the same crash as usual.
Also I gave OBS a complete reinstall and deleted my streamEncoder.json to ensure I get fresh settings I then tested with Min QP set to 1 and VBV Strictness set to 0 this time but to no avail.

Anyways here are the links to all files.
Video: http://5.230.156.11/nextcloud/index.php/s/oaLazdRGHTKIy2o
Dump: http://5.230.156.11/nextcloud/index.php/s/lUKkzO7d4eDhDvG
Log: 2016-11-21 20-34-02.txt
Crashlog: Crash 2016-11-21 20-45-17.txt
Settings: http://i.imgur.com/nNqQoPR.jpg

[Xaymar]

Merged #144, #145 and #151 to this issue, as they all are caused by the same bug and will be fixed by the same driver update in the future.

[Xaymar]

16.11.5 does not contain a VCE firmware update, but it does contain changed DLLs. I however don't expect it to fix this driver bug yet.

[CKannas]

The new driver update makes OBS use less CPU when encoding with the AMD VCE, but it still crashes in that dll. Also CBR seems to not work as expected, which I guess is linked to the expected fix.

[Xaymar]

With 16.12.1 on the horizon (scheduled release 8th December), I hope that this outstanding bug will be fixed. Tagging all the people affected: @R4BC, @sneaky4oe, @k0zyrev; once the driver is out, please try using the encoder again.

[BobbyFuzzy]

Will do. Many thanks for the update, @Xaymar.

…

On 6 December 2016 at 23:21, Michael Fabian Dirks ***@***.***> wrote: With 16.12.1 on the horizon (scheduled release 8th December), I hope that this outstanding bug will be fixed. Tagging all the people affected: @R4BC <https://github.com/r4bc>, @sneaky4oe <https://github.com/sneaky4oe>, @k0zyrev <https://github.com/k0zyrev>; once the driver is out, please try using the encoder again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#143 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE--Grld3Ws1L5sZQ--gkNzqtXu-D98Rks5rFe3dgaJpZM4KwK2H> .

[k0zyrev]

please try using the encoder again

With the new drivers I can't record/stream at all no matter the settings

log: https://gist.github.com/anonymous/1353f0515c559795d1cb6eb30c849cc4

[Xaymar]

23:14:06.368: [AMF Encoder] <Plugin::AMD::VCEEncoder::SetFullColorRangeEnabled> Setting to Disabled failed with error AMF_INVALID_ARG (code 4).
23:14:06.368: [AMF Encoder] <Plugin::AMD::VCEEncoder::SetFullColorRangeEnabled> Setting to Disabled failed with error AMF_INVALID_ARG (code 4).
23:14:06.457: [AMF Encoder] [23:14:06.368][AMFEncoderVCE]  Warning: AVEQueryEncodeCapabilitiesEx bad status : 0x 80000002

Looks like a driver bug, possibly the one @MikhailAMD mentioned.

[razorlikes]

Same issue on my end!

2016-12-08 20-53-31.txt

[MikhailAMD]

SetFullColorRangeEnabled is undocumented property. The driver has changes in preparation for AMF update. Please don't use it till API published.

[Xaymar]

Wasn't my intention to even have it crash the encoding thread, I merely forgot to wrap that part in a try-catch block. Fixed by commit 1e37497.

[Xaymar]

1.4.3.3 fixes #175, so those affected by the old bug (@razorlikes, @k0zyrev, @R4BC, @sneaky4oe) should now be able to test if the crash/corruption still happens or if it was indeed fixed.

[k0zyrev]

I can say for sure that I am still getting frame corruption when frame skipping is enabled. Everything else needs further testing.

[Xaymar]

I am aware of one type of corruption with Frame Skipping: GPUOpen-LibrariesAndSDKs/AMF#44
If you mean that, then this isn't the same issue as before but another one.

[k0zyrev]

This one #151
I think it was labeled as another manifestation of the bug that caused crashes
https://youtu.be/INvfPE3bXQY?t=145

[Xaymar]

That one is tracked here: GPUOpen-LibrariesAndSDKs/AMF#46

[razorlikes]

Tested for a bit today and haven't had a crash. But it seems like the old VCE 2 B-Frame bug is back...

[razorlikes]

I have multiple hours of test recordings now, without any crashes or anything, so I think its safe to assume this issue is fixed.

[Xaymar]

Alright, if it comes back report it again.