--camera-{v,h}flip throw an error now when they used to work with the same camera

[foosel]

Hey there, as already mentioned in this comment on the OctoPrint project I'm seeing a regression with the current code with regards to flipping.

I just built a new binary of v0.1-16-gcdb62ef (cdb62ef) (current master) against the current RPi libcamera0 package, 0.0.4 (debian packaged version: 0~git20230302+923f5d70-1): https://github.com/OctoPrint/camera-streamer-archive/releases/tag/0.1-16-gcdb62ef-1

This runs great on my test image with RPiCam v3 and USB Camera, however I can no longer use --camera-{v,h}flip with the RPiCam like I could before (the USB camera says it doesn't support that in the first place). If I try, camera configuration fails:

octo@octopib:~ $ /usr/bin/camera-streamer --http-port=8080 --camera-type=libcamera --camera-path=/base/soc/i2c0mux/i2c@1/imx708@1a --camera-format=YUYV --camera-vflip=1 --camera-hflip=1
/usr/bin/camera-streamer Version: v0.1-16-gcdb62ef (cdb62ef)
[0:12:52.107127607] [1308]  INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70
[0:12:52.260933243] [1319]  INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media2 and ISP device /dev/media0
device/libcamera/device.cc: CAMERA: Device path=/base/soc/i2c0mux/i2c@1/imx708@1a opened
[0:12:52.261933303] [1308] ERROR Camera camera.cpp:1016 Can't configure camera with invalid configuration
device/libcamera/buffer_list.cc: CAMERA:capture: Failed to configure camera

I am absolutely sure this used to work with an earlier version, because back then I tested this and documented it on the OctoPrint community forums on February 22nd. The first user report of this no longer working reached me on March 6th with a (develop branch) build from February 28th.

I've created an strace for you just in case this might help: flip-strace.txt. I'm happy to test anything or provide any further info you need.

From my understanding, based on the command output, this is failing inside libcamera, so either the interface changed, or there's a regression upstream?

[ayufan]

Hmm. I tried on my arm64 build with the same libcamera0 version and it seems to work fine. I will check using Octoprint image, maybe the armhf behaves differently.

[ayufan]

Oh. It fails on armhf build.

[rlekey]

Any way to get this working with an old build?

[ayufan]

@rlekey It is not build specific. It is system-specific, or libcamera specific. Hard to say yet. I'm thinking on completely moving away from this and rather inject that into jpeg and h264 stream.

[foosel]

Am I understanding this correctly in that that would then have the benefit of also working with any camera, regardless of hardware?

[ayufan]

Yes, that is the idea. It would require the client library to properly support this. Which is true for most part.

It would also work with USB cams, currently this only works with libcamera backend.

[mat100]

Unfortunately I have the same problem. I use your amazing camera streamer in Yocto kirkstone with the latest version of libcamera compiled from the raspberrypi repository. I'm using an ov5647 camera for which this setup has worked so far.

[ayufan]

@mat100 @foosel @rlekey I believe this is fixed. Please validate running develop branch: https://github.com/ayufan/camera-streamer/tree/develop.

Here is how to run it: #53 (comment)

Of course, this only works with libcamera, and does not work for USB cameras.

[foosel]

@ayufan Will look into this ASAP, might be a few days though. In any case, thank you so much for your work!

[rlekey]

@ayufan

Looks like this is working on the develop branch, at least for a direct camera run using tools/libcamera_camera.sh --camera-vflip=1

It still does not seem to work with RESCALLER:STREAM options set in a url ("192.xxx.xxx.xxx/option?verticalflip=1")

I do get the confirmation that RESCALLER:STREAM: The 'verticalflip' was set to '1'. But it doesn't seem to effect the stream or video.

This no longer effects my use, as I can flip the pixels down the road, but figured you'd want to know about the bug. Thanks!

[ayufan]

It still does not seem to work with RESCALLER:STREAM options set in a url

("192.xxx.xxx.xxx/option?verticalflip=1") It will not work to change it dynamically.

…

On Sat, Apr 22, 2023 at 1:12 AM rlekey ***@***.***> wrote: @ayufan <https://github.com/ayufan> Looks like this is working on the develop branch, at least for a direct camera run using tools/libcamera_camera.sh --camera-vflip=1 It still does not seem to work with RESCALLER:STREAM options set in a url ("192.xxx.xxx.xxx/option?verticalflip=1") I do get the confirmation that RESCALLER:STREAM: The 'verticalflip' was set to '1'. But it doesn't seem to effect the stream or video. This no longer effects my use, as I can flip the pixels down the road, but figured you'd want to know about the bug. Thanks! — Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASOSQM32FRAFYYEOOVD7DTXCMH5NANCNFSM6AAAAAAWBLMR74> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[foosel]

Sorry for the delay in getting back to you. I can now confirm that yes, it is indeed fixed in a build of develop that's reporting as v0.1-23-ge0e4c25 (e0e4c25):

pi@octopib:~ $ ~/camera-streamer --http-port=8080 --camera-type=libcamera --camera-path=/base/soc/i2c0mux/i2c@1/imx708@1a --camera-format=YUYV --camera-vflip=1 --camera-hflip=1
/home/pi/camera-streamer Version: v0.1-23-ge0e4c25 (e0e4c25)
[0:03:25.695185562] [1158]  INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70
[0:03:25.843604867] [1169]  INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media3 and ISP device /dev/media0
device/libcamera/device.cc: CAMERA: Device path=/base/soc/i2c0mux/i2c@1/imx708@1a opened
[0:03:25.845042784] [1158]  INFO Camera camera.cpp:1028 configuring streams: (0) 1920x1080-YUYV
[0:03:25.845637836] [1169]  INFO RPI raspberrypi.cpp:851 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SRGGB10_1X10 - Selected unicam format: 2304x1296-pRAA
[0:03:25.849380023] [1158]  INFO Camera camera.cpp:1028 configuring streams: (0) 1920x1080-YUYV (1) 2304x1296-SRGGB10_CSI2P
[0:03:25.849916272] [1169]  INFO RPI raspberrypi.cpp:851 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SRGGB10_1X10 - Selected unicam format: 2304x1296-pRAA
device/buffer_list.c: CAMERA:capture: Using: 1920x1080/YUYV, buffers=3, bytesperline=3840, sizeimage=0.0MiB
device/buffer_list.c: CAMERA:capture: Opened 3 buffers. Memory used: 11.9 MiB
device/buffer_list.c: CAMERA:capture:1: Using: 2304x1296/RG10, buffers=3, bytesperline=2880, sizeimage=0.0MiB
device/buffer_list.c: CAMERA:capture:1: Opened 3 buffers. Memory used: 10.7 MiB
device/v4l2/device.c: SNAPSHOT: Device path=/dev/video31 fd=41 opened
device/v4l2/buffer_list.c: SNAPSHOT:output:mplane: Requested resolution=1920x1080 is unavailable. Got 1920x1088.
device/buffer_list.c: SNAPSHOT:output:mplane: Using: 1920x1056/YUYV, buffers=3, bytesperline=3840, sizeimage=3.9MiB
device/buffer_list.c: SNAPSHOT:output:mplane: Opened 3 buffers. Memory used: 0.0 MiB
device/buffer_list.c: SNAPSHOT:capture:mplane: Using: 1920x1056/JPEG, buffers=3, bytesperline=0, sizeimage=4.0MiB
device/buffer_list.c: SNAPSHOT:capture:mplane: Opened 3 buffers. Memory used: 12.0 MiB
device/v4l2/device.c: VIDEO: Device path=/dev/video11 fd=45 opened
device/buffer_list.c: VIDEO:output:mplane: Using: 1920x1080/YUYV, buffers=3, bytesperline=3840, sizeimage=4.0MiB
device/buffer_list.c: VIDEO:output:mplane: Opened 3 buffers. Memory used: 0.0 MiB
device/buffer_list.c: VIDEO:capture:mplane: Using: 1920x1080/H264, buffers=3, bytesperline=0, sizeimage=0.8MiB
device/buffer_list.c: VIDEO:capture:mplane: Opened 3 buffers. Memory used: 2.2 MiB
device/device.c: CAMERA: Setting frame interval_us=0 for FPS=30
device/libcamera/device.cc: CAMERA: Configuring option aftrigger (00000021, type=3) = 1
device/v4l2/device_options.c: SNAPSHOT: Configuring option compressionquality (009d0903) = 80
device/v4l2/device_options.c: VIDEO: Configuring option repeatsequenceheader (009909e2) = 1
device/v4l2/device_options.c: VIDEO: Configuring option videobitratemode (009909ce) = 0
device/v4l2/device_options.c: VIDEO: Configuring option videobitrate (009909cf) = 2000000
device/v4l2/device_options.c: VIDEO: Configuring option repeatsequenceheader (009909e2) = 5000000
device/v4l2/device_options.c: VIDEO: Configuring option h264iframeperiod (00990a66) = 30
device/v4l2/device_options.c: VIDEO: Configuring option h264level (00990a67) = 11
device/v4l2/device_options.c: VIDEO: Configuring option h264profile (00990a6b) = 4
device/v4l2/device_options.c: VIDEO: Configuring option h264minimumqpvalue (00990a61) = 16
device/v4l2/device_options.c: VIDEO: Configuring option h264maximumqpvalue (00990a62) = 32
device/links.c: ?: Link 0: CAMERA:capture[1920x1080/YUYV/3] => [SNAPSHOT:output:mplane[1920x1056/YUYV/3], VIDEO:output:mplane[1920x1080/YUYV/3]]
device/links.c: ?: Link 1: SNAPSHOT:capture:mplane[1920x1056/JPEG/3] => [SNAPSHOT-CAPTURE, STREAM-CAPTURE]
device/links.c: ?: Link 2: VIDEO:capture:mplane[1920x1080/H264/3] => [VIDEO-CAPTURE]
[0:03:25.894752152] [1172]  WARN IPARPI raspberrypi.cpp:1183 Could not set AF_TRIGGER - no AF algorithm or not Auto
device/buffer_list.c: CAMERA:capture: Streaming started... Was 0 of 3 enqueud
device/buffer_list.c: SNAPSHOT:output:mplane: Streaming started... Was 0 of 3 enqueud
device/buffer_list.c: VIDEO:output:mplane: Streaming started... Was 0 of 3 enqueud
device/buffer_list.c: SNAPSHOT:capture:mplane: Streaming started... Was 0 of 3 enqueud
device/buffer_list.c: VIDEO:capture:mplane: Streaming started... Was 0 of 3 enqueud

Great work! 🚀

Shall I close this, or do you want to keep it open until you've merged develop?

[ayufan]

Lets wait till I merge this. It should happen over weekend.

[foosel]

Roger that. Are you still also thinking about switching the implementation over to doing this in software instead as mentioned in #51 (comment), for the sake of USB camera owners? Or is that now off the table again? No pressure, just curious!

[ayufan]

@foosel

I tried, exif worked, but was unable to achieve the same for h264 using display sei. So, no, this is no-go until I figure out the display sei.

I will try to re-implement this via rescaller, but it will come with performance penalty for USB cams since JPEG will have to be decoded, rescalled and encoded instead of just pass through.

Additionally, Olof from Bondtech (when discussing with @meteyou) mentioned that the easier trick (and more reliable) might be just doing cropping and rotation via css on frontend...

Ref.: https://github.com/ayufan/camera-streamer/commits/exif-support

[foosel]

Yeah, rotating via frontend is something I already offer in OctoPrint as well since hour 1, but when I announced a potential (and now pretty much decided on) switch to camera-streamer as default camera stack on future versions of the image, I immediately got the feedback that rotation/flipping options in the backend are very much needed for certain workflows (my guess is alternative frontends without rotation built in).

Hence my curiousity. Personally, I never have had to use rotation or flipping myself, I simply mount the camera in the right orientation to begin with, but who am I to judge how people are forced to set up their stuff 😅

Thanks for clarifying! 😊

[ayufan]

Hence my curiousity. Personally, I never have had to use rotation or flipping myself, I simply mount the camera in the right orientation to begin with, but who am I to judge how people are forced to set up their stuff 😅

I guess, sometimes it is hard. Overall 90o rotation is hard on memory bandwidth, but the horizontal/vertical flip is OK. Unsure, maybe the solution is a mix: use EXIF for JPEG (snapshot, to provide zero-copy solution), but use rescaller method for H264 since we already have to decode JPEG there so the additional rescaling step can be basically free, unsure yet.

So, don't expect 90o rotation, but flip might be plausible to support everywhere.

The develop crop function requires re-encoding already.

[ayufan]

@foosel I merged the camera-vflip/hflip fix into master along some other smaller bits.

The develop only contains crop function on top of it.

I will close it, but if you could re-test it I would be grateful.

[foosel]

@ayufan Just built a package from master and things are still looking good here, --camera-{h,v}flip are functional against an RPiCam3! 👍

[progSides]

Just in case anyone (or me in the future) needs to know how to flip or rotate the camera image horizontally, vertically or both on the new v3 camera module... add this ( --camera-hflip=1 --camera-vflip=1 ) to the end of the default file below. Notice: this is not --camera-options, it is camera-hflip, etc but it is on the same line.

/boot/camera-streamer/libcamera.conf

# The port on which the webcam server for the camera should listen on. If you have only
# one camera, leave at 8080. If you have more, change to 8081, 8082, etc. The primary
# camera will be considered the one with 8080.
PORT=8080

# The resolution to request on the camera sensor. Defaults to 1280x720.
WIDTH=1920
HEIGHT=1080

# The height to use for the video stream. Defaults to 720.
VIDEO_HEIGHT=720

# The height to use for the snapshots. Defaults to 1080.
SNAPSHOT_HEIGHT=1080

# The framerate to set on the camera. Defaults to 15fps.
FRAMERATE=15

# Additional options. By default enables continuous auto focus (if possible).
OPTIONS='--camera-options="AfMode=2" --camera-options="AfRange=2" --camera-hflip=1 --camera-vflip=1'

[stuomas]

Just in case anyone (or me in the future) needs to know how to flip or rotate the camera image horizontally, vertically or both on the new v3 camera module... add this ( --camera-hflip=1 --camera-vflip=1 ) to the end of the default file below. Notice: this is not --camera-options, it is camera-hflip, etc but it is on the same line.

/boot/camera-streamer/libcamera.conf

# The port on which the webcam server for the camera should listen on. If you have only
# one camera, leave at 8080. If you have more, change to 8081, 8082, etc. The primary
# camera will be considered the one with 8080.
PORT=8080

# The resolution to request on the camera sensor. Defaults to 1280x720.
WIDTH=1920
HEIGHT=1080

# The height to use for the video stream. Defaults to 720.
VIDEO_HEIGHT=720

# The height to use for the snapshots. Defaults to 1080.
SNAPSHOT_HEIGHT=1080

# The framerate to set on the camera. Defaults to 15fps.
FRAMERATE=15

# Additional options. By default enables continuous auto focus (if possible).
OPTIONS='--camera-options="AfMode=2" --camera-options="AfRange=2" --camera-hflip=1 --camera-vflip=1'

Hi, such file does not exist by default, should it be created? Where is this file documented?