[BUG] Cannot Allocate memory Bookworm x64 and Arducam 64mp

[quitmeyer]

Describe the bug
I got one of those (notorious) arducam 64mp cameras (the semi-independent drivers which i am sure are driving all you folks nuts)
but back in march I actually wrote up a full walk through for how to get picamera 2 to take full res 9152 x 6944 photos and it worked great
https://forum.arducam.com/t/full-walkthrough-setup-rpi4-take-64mp-photos-and-control-focus/4653

To Reproduce
you can follow my whole walkthrough here for a pi4 and bullseye
https://forum.arducam.com/t/full-walkthrough-setup-rpi4-take-64mp-photos-and-control-focus/4653

or follow Henri's guide for bookworm:
https://forum.arducam.com/t/raspberry-pi-5-and-arducams/5773/4?u=hikinghack

Expected behaviour
take a photo and save it
but it crashes just from trying to configure it.

Console Output, Screenshots
I even make sure i have plenty of CMA
it says
CmaTotal: 524288 kB
CmaFree: 501984 kB

though when the program crashes, look at how little CMAfree there is

Hardware :
pi4 8gb
arducam 64

fresh install of bookworm 64 with fresh installed and updated picamera2 (as of dec 11 2023)

Thanks for your help!

[davidplowman]

You're asking for one full-res (64MP) RGB output buffer, so from my understanding the code will allocate:

1 x 64MP RGB buffer = 192MB
1 x 64MP 10-bit raw buffer = 80MB
Then the code also has to allocate 2 extra raw buffers so that it can drop frames while the camera starts (this is a bit technical, but is something we're trying to change). Anyway, that's an extra 160MB

So altogether I can account for 192 + 80 + 160 = 432MB, before anything else tries to use CMA memory.

We can probably expect things always to be a bit marginal in this situation, I would think.

Another idea might be to run a preview mode first. Supposing you have a a raw mode that is 4x4 binned (so about 2300x1700). That would be 5MB per raw buffer. Then:

Start by allocating a preview configuration with buffer_count=2. That would allocate:

2 x raw buffers = 10MB.
2 x output buffers, but you could make these really small, e.g. VGA at about 1MB each, so that's another 2MB.
Then you get the annoying extra 2 raw buffers because of the frame drops, but now that's only another 10MB.

Next you'll have the capture mode. This time it will use 1 x raw buffer = 80MB, plus 1 x output buffer = 192MB.

Adding all that up I have: 10 + 2 + 10 + 80 + 192 = 294MB.

So that sounds like it should be more likely to work.

As I said, there are those 2 annoying extra raw buffers that we currently have to allocate, but we are trying to get rid of them - still work in progress, though.

Another idea if you're desperate would be to capture a YUV420 buffer instead of RGB. You can't feed that straight to most Python JPEG encoders, but you could convert it to RGB with OpenCV and then save it. That's only 96MB of CMA instead of 192, so much easier. The big 192MB RGB buffer only lives in regular memory, so you can have as much as you want (within reason).

[quitmeyer]

Thanks for your help! I realize a big photo like this is qutie demanding!
and oh catching any other type of buffer like a YUV would be fine! Our main problem is that we can take a photo with libcamera, but it takes 8 seconds. and we are deploying a camera to last in the field as long as possible. and during the time the photo is being taken we have very large, energy expensive lights. So i am hoping to take photos with picamera where the lights can stay off except for the actual time the photo is being taken (off for the starting up of the camera process)

so if i can take a different type of buffer, grab the photo, and do post-processing with a light off, that's fine! Ill try it now and report back if it works!

[davidplowman]

Synchronising with things like lights coming on can be tricky because there's a lot of pipelining inside Picamera2 and libcamera. And in fact rolling shutter sensors make this even worse. Just because an image comes out after you turn the lights on, it doesn't mean the image didn't start exposing earlier as these are all rolling shutter sensors. So the top part could have the lights off and the bottom half could have them on.

The latest version of Picamera2 has a flush parameter for capture_request. If you pass flush=True it will discard all the images where any pixels were exposed before the moment of the capture request. So you could turn on the lights, call request = picam2.capture_request(flush=True), turn off the light, then copy the image (array = request.make_array('main')) and convert/save it.

If you use the preview mode method that I described, you're guaranteed the nothing is exposed before you switch to the capture mode.

[quitmeyer]

oh wow! @davidplowman this is SO useful. I am trying right now!

Thanks to you i already got it to not crash with a 64MP image!

capture_config = picam2.create_still_configuration(main={"size": (9152, 6944), "format": "YUV420"}, buffer_count=1)
picam2.configure(capture_config)

picam2.start()
print("cam started");

lightson()
picam2.capture_array("main")
lightsoff()

still figuring out right now how to turn that into a file, but excited it didn't crash!
but my dismay was the flash still stayed on for like 10 seconds (which was actually longer than just running a CMD libcamera-still command

but the flush technique looks exactly what im looking for!

[davidplowman]

One problem you've got is that you need to give your camera a while to get going. The first "several" frames get dropped while all the AGC/AWB is sorting itself out, so you'd get a better result if you had a time.sleep(1) before lightson().

[quitmeyer]

ok i now have

from picamera2 import Picamera2, Preview
picam2 = Picamera2()
capture_config = picam2.create_still_configuration(main={"size": (9152, 6944), "format": "YUV420"}, buffer_count=1)
#capture_config = picam2.create_still_configuration(main={"format": 'RGB888',"size": (9152, 6944)})
picam2.configure(capture_config)

picam2.start()
print("cam started");
time.sleep(10)
start = time.time()
flashOn()
request = picam2.capture_request(flush=True)
flashOff()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))
array = request.make_array('main')

and unfortunately, it still takes 8.3 seconds to take a photo
(which i think wouldn't be the case because Arducam published their example showing that the max full res photo speed is 1-2 seconds)

https://github.com/ArduCAM/picamera2_examples/blob/main/64mpGetDataSpeed.py

it also seems odd to me that it takes about as much time as doing a full commandline operation with libcamera (which includes starting the camera and saving the image, right?)

The following code takes 9 seconds

cmd= "libcamera-still --lens-position 7.5 -n --width 9152 --height 6944 --awb cloudy --metering average --ev .5 -o "+filename

#start timer
start = time.time()
print("Lights on")
#Control the Channel 1
GPIO.output(Relay_Ch2,GPIO.LOW)
GPIO.output(Relay_Ch3,GPIO.LOW)

subprocess.call(cmd, shell = True)

print("Lights off")
GPIO.output(Relay_Ch2,GPIO.HIGH)
flashtime=time.time()-start
print("picture take time: "+str(flashtime))

[quitmeyer]

a side confusing thing more pertinent to the original question is that back in march i could run this exact command fine

capture_config = picam2.create_still_configuration(main={"format": 'RGB888', "size": (9152, 6944)})

and it would not give me memory errors (with the same setup)
but now it does!? Im guessing something new in picamera changed something?

[quitmeyer]

"and unfortunately, it still takes 8.3 seconds to take a photo"

hey! i think i solved that problem! I just set my wait after
picam2.start()
print("cam started");
time.sleep(10)

from 10 seconds to 25 seconds (I guess that camera is just slow to get moving!)

and now

start = time.time()
flashOn()
request = picam2.capture_request(flush=True)
flashOff()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))

this takes only 3 seconds!

so that's great! @davidplowman you just helped us reduce our light energy use by 2/3! Thanks so much

of course Ideally it seems like the photo isn't actually being exposed for 3 full seconds, so maybe there's a way to reduce it further, but this is a great start!

[davidplowman]

A few things to unpack here.

As to the original question, my observation was that I thought the numbers looked a bit marginal so yes, changes that happen in a new OS revision could easily cause it to work/fail.

One problem you have is that by running the camera all the time in the max resolution, it's using just 1 buffer. This halves your framerate because every other camera frame gets dropped. Using buffer_count=2 would fix this, but then that's twice the memory.

It might actually work better if you took the approach of a different (faster, lower memory) preview mode, and then switch mode for the capture. In that scenario once you switch mode there's no need to wait, you can just grab the first frame so the single buffer is fine.

[quitmeyer]

"It might actually work better if you took the approach of a different (faster, lower memory) preview mode, and then switch mode for the capture. In that scenario once you switch mode there's no need to wait, you can just grab the first frame so the single buffer is fine."

oh that sounds good to me, but i don't really know anything about the preview modes? How do i change preview modes?

(Or do you mean just take a lower resolution photo? because that's my other challenge, for this project they need the full res 64Mp photos)

[quitmeyer]

It might actually work better if you took the approach of a different

also i don't need to preview the image at all! no human needs to actually see the images until a month after this thing is deployed in the field :)

[quitmeyer]

ohhh! maybe this is what you were trying to tell me @davidplowman

i just switched the photo call to
picam2.capture_array("raw")

and my total lights on time was just 0.6 seconds!

if i do
raw_np_array=picam2.capture_array("raw")
though i get 1.8 seconds (which is still so much better than my original 9 seconds)
but i wonder if i don't have the
raw_np_array=
part, if i can copy that array to a variable after i turn off the flash?

[davidplowman]

Well, many ways you could try it but this ought to be close:

from picamera2 import Picamera2
import time

picam2 = Picamera2()
capture_main = {"size": (9152, 6944), "format": "YUV420"}
capture_config = picam2.create_still_configuration(main=capture_main)
preview_main = {"format": 'YUV420',"size": (640, 480)}
preview_raw = {'size': (2312, 1736)}
preview_config = picam2.create_preview_configuration(main=preview_main, raw=preview_raw, buffer_count=2)
picam2.configure(preview_config)

picam2.start()
print("cam started");
time.sleep(10)
picam2.stop()
picam2.configure(capture_config)
start = time.time()
flashOn()
picam2.start()
request = picam2.capture_request()
flashOff()
picam2.stop()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))
array = request.make_array('main')

Completely untested, of course! You might find that you could put your capture format back to 'RGB888' as I think the memory configuration there is smaller.

[quitmeyer]

Well, many ways you could try it but this ought to be close:

just tested your awesome code! it works and has a flash time of just 2 seconds!

if i add flush=True it comes in at just 2.1 seconds still, so that's cool!
If i rearrange it so that

picam2.start()
start = time.time()
flashOn()
request = picam2.capture_request(flush=True)

i can get the flash time down to just 1.9 seconds!

why do you start the cam and then stop it and then start it again?

also i wonder if it's possible to have the camera do whatever it was doing here to go so fast? (or maybe that wasn't going to give me usable info anyway?)

picam2.capture_array("raw")

and my total lights on time was just 0.6 seconds!

[davidplowman]

The way I wrote the code, the flush=True isn't necessary, because the camera isn't running when the flash is turned on, so there's no race hazard.

I think your rearrangement is probably good because starting the camera is usually a bit slow (takes a little while extra for the first frame), and you get to skip that. But adding flush=True should make it safe because there's the risk that the first frame could start to be exposed before the flash comes on, but the flush will discard the frame if that seems to have happened. But as always, check carefully!!

The reason for starting the camera in preview mode, then stopping and restarting in capture mode are:

• In preview mode it runs at a much faster frame rate, so all the algorithms (like AWB, AGC etc.) all converge way faster.
• This is particularly helpful for all those 7 or 8 frames that you never see. They probably all go past at ~30fps, whereas in the capture mode, with just 1 buffer, you're running at much less than 1fps.
• The configuration with the preview uses less memory. As I explained earlier, we have these two annoying buffers that we have to allocate in whatever mode the camera first starts in. If that's the full res mode, then it's really bad (as you discovered). In the preview mode, we hardly notice it.
• Because of the slow full res framerate, waiting for a full res frame to come out is actually no better than starting the camera from scratch (which is not usually true with other cameras, e.g. the HQ cam runs at 10fps in the full res mode) - so long as we've done the preview and don't have to wait for the algorithms to converge.

I wasn't quite sure what you were doing with picam2.capture_array('raw'). I think the danger is that you're capturing an earlier frame, so it might not all be exposed with the flash on everywhere. There's no way that capture_array is quicker than capture_request, because capture_array does a capture_request and then copies the big numpy array. But only capture_request has the flush parameter (and generally, it's the function I recommend to people).

[quitmeyer]

I realized i should close this thread since you solved the original problem i had a while ago, and i made a different "How to" thread.

You are amazing @davidplowman !

[quitmeyer]

I wasn't quite sure what you were doing with picam2.capture_array('raw').

Ok, so if i have my older code setup, i can get the flash time down to .6-1 seconds (but maybe it doesn't produce anything viable?)

like this

capture_config = picam2.create_still_configuration(main={"size": (9152, 6944), "format": "YUV420"}, buffer_count=1)
picam2.configure(capture_config)

picam2.start()
print("cam started");
time.sleep(25)

start = time.time()
flashOn()
numpy_array=picam2.capture_array("raw")

flashOff()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))

vs if i have this same code but use a capture request instead it takes 3.9 seconds

capture_config = picam2.create_still_configuration(main={"size": (9152, 6944), "format": "YUV420"}, buffer_count=1)
picam2.configure(capture_config)

picam2.start()
print("cam started");
time.sleep(25)

start = time.time()
flashOn()
#numpy_array=picam2.capture_array("raw")
request=picam2.capture_request(flush=True)
flashOff()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))

[quitmeyer]

You might find that you could put your capture format back to 'RGB888'

I checked and no it still won't let me change the config back to RGB888 still not enough memory :)

[davidplowman]

I wasn't quite sure what you were doing with picam2.capture_array('raw').

Ok, so if i have my older code setup, i can get the flash time down to .6-1 seconds (but maybe it doesn't produce anything viable?)

like this

capture_config = picam2.create_still_configuration(main={"size": (9152, 6944), "format": "YUV420"}, buffer_count=1)
picam2.configure(capture_config)

picam2.start()
print("cam started");
time.sleep(25)

start = time.time()
flashOn()
numpy_array=picam2.capture_array("raw")

flashOff()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))

Yes, so here I don't think you're guaranteed that all (or indeed any) of the pixels in the captured image will have the flash on. You need to use capture_request with flush for that. Just as a note - I'm not clear whether you want the raw Bayer data from the sensor (the 'raw' stream), or a processed YUV or RGB image from the 'main' stream. There should be no difference in the capture times as they all come out together.

vs if i have this same code but use a capture request instead it takes 3.9 seconds

capture_config = picam2.create_still_configuration(main={"size": (9152, 6944), "format": "YUV420"}, buffer_count=1)
picam2.configure(capture_config)

picam2.start()
print("cam started");
time.sleep(25)

start = time.time()
flashOn()
#numpy_array=picam2.capture_array("raw")
request=picam2.capture_request(flush=True)
flashOff()
flashtime=time.time()-start
print("picture take time: "+str(flashtime))

I think this is guaranteeing that everything, even the first pixel, is entirely exposed after the flash comes on. But I think it's finding that it actually has to drop a couple of frames for that to be true, and the really slow framerate means that costs you several seconds.

Again, just as an aside, I wonder why you want 64MP images (other than "more is better"). You could easily run a 12MP sensor using several buffers and would get 10fps. A capture in these circumstances would cost you, I guess, 200 to 300 milliseconds.

[quitmeyer]

wonder why you want 64MP images (other than "more is better").

Hahaaha I've been making these arguments for a while too with the project im working on, but basically they qre trying to identify moths from a standard size target sheet down to the species level, and they want every pixel they can get. But other approaches might be better (multiple cheaper cameras with lower resolution, etc...) , but we already have these cameras and are doing what we can in this nice, scrappy little project, and it's going quite well!

And you just save this a lot of battery life that would normally just be pointlessly wasted turning our photography lights on but not exposing anything! So thanks a bunch!

Yes, so here I don't think you're guaranteed that all (or indeed any) of the pixels in the captured image will have the flash on.

Cool, yeah,. I suspected it was getting that speedy performance by taking some shortcuts that we couldn't really use. I suspect at least for now, that the approach you shared with me that got me down to like 1.9 seconds with good safe pixels is probably the best we will get for now.

Someone on the arducam forums had a suggestion of telling picam2 to do something like just take a region of interest that's half the size of the full frame and split it into two buffers to make it quicker (basically something like an image of each half of the sensor, and then we would stitch the two halves back together post-processing)? But I'm not sure if that will work or if the logic behind that is sound. What do you think?

[davidplowman]

Someone on the arducam forums had a suggestion of telling picam2 to do something like just take a region of interest that's half the size of the full frame and split it into two buffers to make it quicker (basically something like an image of each half of the sensor, and then we would stitch the two halves back together post-processing)? But I'm not sure if that will work or if the logic behind that is sound. What do you think?

Hmm, I'm not convinced. The bottom line is that you want to read 64MP out of the sensor and that's the hard limit, whether you read it in one go or in bits!

[quitmeyer]

want to read 64MP out of the sensor and that's the hard limit, whether you read it in one go or in bits!

Exactly what i was thinking, like why not do 64 million buffers one pixel each, hahaha ;)