vaEndPicture returns Invalid Parameter status on Ubuntu 17.04

[kingargyle]

[VAAPI encoder]: "vaEndPicture(q->display, q->context)": invalid parameter

It seems that with obs-studio and the linux-vaapi plugin, if a user is using a newer driver than 1.7.1 they get the above as soon as they try to record. The plugin and driver work fine, with Ubuntu 16.10 or any system really using drivers before 1.7.3.

I'm trying to debug the source code for the plugin for obs-studio but unforately the error information I get back doesn't give me enough information on what Parameter is invalid or what is actually wrong.

Some additional information:

device_type : X11
device: Intel i965 driver for Intel(R) Haswell Mobile - 1.8.0
device_path: (null)
fps_num: 30
fps_den: 1
width: 1280
height: 720
profile: main
keyint: 2 (s)
rc_type: CBR
bitrate: 2500 (kbit)
use_custom_cpb: no

Not sure what needs to be changed, the only difference here is upgrading systems.

[xhaihao]

Could you attach your LIBVA_TRACE log? You should set LIBVA_TRACE variable in your environment.

E.g.
$> cd
$> export LIBVA_TRACE=libva

You will get a log file (libva.xxx.xxx) after running your application.

[kingargyle]

@xhaihao Here are the log files.

https://gist.github.com/kingargyle/b00feaf79aef20277c4092b89397ffd9

[kingargyle]

An additional note, I installed a dual boot of Ubuntu 16.04, which has the 1.7.0-1 version of the intel driver, and this version works correctly. We have had reports as well that 1.7.3 does not work, and anything higher does not work. Hopefully the log files help.

[xhaihao]

intra_period is set to 60 however ip_period is to 0 in your case, however 0 is invalid for ip_period when using CBR mode. The new driver adds some checks for intra_period/ip_period. Please set a valid ip_period, sorry for the inconvenience.

[xhaihao]

You may refer to https://github.com/01org/libva-utils/blob/master/encode/h264encode.c for how to set ip_period in your application:

[kingargyle]

@xhaihao thanks, I'll add the link to the issue for the code that is having the problem. I'm going to go ahead and close this out and will re-open if necessary.