Missing VIDIOC_STREAMON may cause loss of V4L2_CAP_VIDEO_OUTPUT

[BenBE]

Step 1: Read this

Done.

Step 2: Describe your environment

• v4l2loopback version: 0.12.5 (dkms from distro: 0.12.5-1ubuntu3, also present on main in git [untested])
• kernel version: 5.17.1, also reports for 5.13.0-28 & 5.13.0-30
• Distribution (+version): Lubuntu 22.04 LTS, also reported for Ubuntu 21.10 and Ubuntu 20.04
• See reference bug report: restart of backscrub not possible on ubuntu 21.10 floe/backscrub#133

Step 3: Describe the problem:

The v4l2loopback dev will loose its V4L2_CAP_VIDEO_OUTPUT when failing to do an ioctl with operation VIDIOC_STREAMON before writing any frames.

Steps to reproduce:

1. int fdwr = open(device, O_RDWR|O_CLOEXEC);
2. ret_code = ioctl(fdwr, VIDIOC_QUERYCAP, &vid_caps);
3. Set video format:

    	memset(&vid_format, 0, sizeof(vid_format));
	vid_format.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
	vid_format.fmt.pix.width = w;
	vid_format.fmt.pix.height = h;
	vid_format.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;
	vid_format.fmt.pix.sizeimage = framesize;
	vid_format.fmt.pix.field = V4L2_FIELD_NONE;
	vid_format.fmt.pix.bytesperline = linewidth;
	vid_format.fmt.pix.colorspace = V4L2_COLORSPACE_SRGB;
	ret_code = ioctl(fdwr, VIDIOC_S_FMT, &vid_format);

4. Write frame data to fdwr
5. close(fdwr);

Observed Results:

• v4l2-ctl -D is now missing V4L2_CAP_VIDEO_OUTPUT
• Subsequent tries to open the device for output fail with EINVAL

Expected Results:

• Closing the device should restore the V4L2_CAP_VIDEO_OUTPUT if it was removed during operation of the device (which it is inside the v4l2_loopback_write handler AFAIC).
• Trying to output frame data on a device not in streaming mode should fail with an error, e.g. ENOTTY to force proper state transitions.

Relevant Code:

Cf. https://github.com/floe/backscrub/blob/395ac39776e1c54f33d1b5d565834590caadd707/videoio/loopback.cc#L29-L82

Runs standalone if -Dstandalone is provided for compilation of that single file.

Problem remarks:

Looking through the code I noticed that the driver leaves quite a bit of leeway for performing operations without proper pre-condition checks, like allowing to write frame data while not streaming, reading frames on an opener of type WRITER, writing on an opener of type READER, or starting streaming as WRITER, but closing things of as an READER. All those lax checks should be made much more strict to enforce arguments to be consistent with internal state and enforcing proper protocol (start streaming before writing actual frame data or at least updating the opener to be an WRITER when dropping the ready_for_output state as a consequence of the write operation).