Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bayer data not located in stream with Python 3 #133

Closed
LabyrinthTechnology opened this issue Jul 28, 2014 · 13 comments

Comments

@LabyrinthTechnology
Copy link

commented Jul 28, 2014

Using example mentioned in documentation how to extract raw data from jpeg+bayer stream (short version using PiBayerArray) I received error Unable to locate Bayer data at end of buffer.

Using longer commented code I tested 'BRCM' string exists in the stream, but data[:4] are treated as b'BRCM', not ASCII 'BRCM'.

Any guess where the problem can be? I am Python + Raspberry beginner, but in my opinion the problem is not in the assert / comparison, but it seems the stream is created in a different format than is expected.

Regards,
Jan

@waveform80

This comment has been minimized.

Copy link
Owner

commented Jul 29, 2014

Hi Jan,

The example is working fine for me with picamera 1.6 and firmware revision 698. Could you provide the exact code you're using, and possibly the firmware revision as well? I don't think it's a firmware issue; as far as I know bayer-capture has been available in firmware for ages now, but I just want to make sure it's not a truly ancient one (just in case). As for the code, I need to make sure you're using bayer=True and not using anything like use_video_port=True (bayer captures only work from the still port).

Thanks,

Dave.

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 29, 2014

Hi Dave,

thanks for your fast response.

Raspberry used: Model A (not Ethernet), Raspberry Linux 3.12.25+.
Interesting is vcgencmd version command reports a version number in a form of GUID with a date July 23 2014 and it notes (clean) (release), but not a "normal number" is available. If rpi-update is used, it reports the firmware is up-to-date.

The code used is just a copy-paste from a documentation:

import sys
import picamera
import picamera.array
import time

with picamera.PiCamera() as camera:
    with picamera.array.PiBayerArray(camera) as stream:
        camera.capture(stream, 'jpeg', bayer=True)
        output = (stream.demosaic() >> 2).astype(np.uint8)
        with open('/home/pi/picamera/bayer.jpg', 'wb') as f:
            output.tofile(f)

Today I will do the same test with another Raspberry (I obtained a couple of them from my customer to work on the project), because during the testing and development different small update, removal and configuration changes were done with the current piece and others are more or less standard scratch installations.

Regards,

Jan

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 29, 2014

One more note: If I use raspistill to capture an image and raspiraw / raspi_dng to extract raw data, it works fine.

Jan

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 29, 2014

Hmmmm ... tested with another Raspberry, but the same result:

pi@raspberrypi ~/CameraControl $ sudo python3 simplePicture.py
Traceback (most recent call last):
  File "simplePicture.py", line 17, in <module>
    camera.capture(stream, 'jpeg', bayer=True)
  File "/usr/lib/python3/dist-packages/picamera/camera.py", line 1291, in capture
    if not encoder.wait(self.CAPTURE_TIMEOUT):
  File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 688, in wait
    self.stop()
  File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 719, in stop
    self._close_output()
  File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 633, in _close_output
    output.flush()
  File "/usr/lib/python3/dist-packages/picamera/array.py", line 425, in flush
    raise PiCameraValueError('Unable to locate Bayer data at end of buffer')
picamera.exc.PiCameraValueError: Unable to locate Bayer data at end of buffer

What I do wrong?

With regards,

Jan

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 29, 2014

I did another test: If I create jpg file with bayer data using

camera = picamera.PiCamera()
camera.iso = 100
camera.awb_mode = "off"
time.sleep(2)
camera.capture('/home/pi/workFile-bayer.jpg', 'jpeg', bayer=True)

and than I extract data with raspi_dng, it works fine, so the file itself seems to be correct.

Jan

@waveform80

This comment has been minimized.

Copy link
Owner

commented Jul 29, 2014

Ah, my apologies - I didn't make clear how to determine the firmware revision number. If you run uname -a at the command line you'll see some output like:

Linux kermit 3.12.25+ #698 PREEMPT Wed Jul 23 21:00:22 BST 2014 armv6l GNU/Linux

This indicates the firmware revision is 698 (the value before PREEMPT). The value itself doesn't mean much but it gives me a rough idea if you're on a newer, older, or very old firmware. Still, if you've run rpi-update then you'll be on a recent one so I think we can rule that out as the issue. The fact you're on a model A is potentially interesting - at the moment I can't test on a model A as my wife's nicked mine for some tests at work! I don't think it should matter that it's an A (there should still be plenty of memory, even for big stuff like Bayer data in numpy arrays), but I'll try and rule it out as soon as I get mine back.

That said, there's a few discrepancies we could clear up as causes first; the stack trace you pasted in a couple of messages back begins with File "simplePicture.py", line 17, in <module> but there aren't 17 lines in the script. Could you paste the actual script that produced the traceback?

One other thing to mention is the line with open('/home/pi/picamera/bayer.jpg', 'wb') as f in the script. The output of numpy's tofile method isn't a JPEG - it's just the binary data from the array (similar to an unencoded RGB capture). This won't cause an error as such but if you're expecting the resulting file to be a JPEG, you'll be disappointed!

Finally, I have just noticed an issue with the demosaic method (a bug in the padding code) which causes a broadcast error; I'll get that fixed for 1.7 (it's filed as #134).

Dave.

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 29, 2014

Hi Dave,

thanks a lot for detail comment. I will try to provide you more detail information:

Firmware revision: Linux raspberrypi 3.12.22+ #691 PREEMPT Wed Jun 18 18:29:58 BST 2014 armv6l GNU/Linux

I am unsure a bit how the firmware update works, because after the first rpi-update this was reported: Linux raspberrypi 3.12.25+ #700 PREEMPT Thu Jul 24 17:51:46 BST 2014 armv6l GNU/Linux, but I realized I am using a new Raspberry board, so I used apt-get update and apt-get upgrade to upgrade everything and after it the version 691 is listed and rpi-update tells "the firmware is up-to-date". But anyway, it's not very old firmware I guess.

A bit more information about Raspberry HW model:

root@raspberrypi:/home/pi# cat /proc/cpuinfo
processor   : 0
model name  : ARMv6-compatible processor rev 7 (v6l)
Features    : swp half thumb fastmult vfp edsp java tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part    : 0xb76
CPU revision    : 7

Hardware    : BCM2708
Revision    : 0008
Serial      : 00000000e6a6e5f3

Sorry for the mess in reported line numbers, I removed some comments forgetting it will cause problems with exception report. So, the code

import sys
import picamera
import picamera.array
import time

with picamera.PiCamera() as camera:
    with picamera.array.PiBayerArray(camera) as stream:
        camera.capture(stream, 'jpeg', bayer=True)
        output = (stream.demosaic() >> 2).astype(np.uint8)
        with open('/home/pi/picamera/bayer.jpg', 'wb') as f:
            output.tofile(f)

causes this exception:

root@raspberrypi:/home/pi/CameraControl# python3 simplePicture.py
Traceback (most recent call last):
  File "simplePicture.py", line 8, in <module>
    camera.capture(stream, 'jpeg', bayer=True)
  File "/usr/lib/python3/dist-packages/picamera/camera.py", line 1291, in capture
    if not encoder.wait(self.CAPTURE_TIMEOUT):
  File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 688, in wait
    self.stop()
  File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 719, in stop
    self._close_output()
  File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 633, in _close_output
    output.flush()
  File "/usr/lib/python3/dist-packages/picamera/array.py", line 425, in flush
    raise PiCameraValueError('Unable to locate Bayer data at end of buffer')
picamera.exc.PiCameraValueError: Unable to locate Bayer data at end of buffer

I am aware the output file is not jpeg but binary data. What I'd like to have is the same raw data as I can receive from raspi_dng tool. I have to mention from yesterday "blocking issue" state the issue is now "nice to solve it" for me, because I tested I can use camera.capture() in my Python code and to use raspi_dng as an external tool to export raw data. Not elegant, but it works ;-)

Your note about memory is quite correct, a couple of times when I run the code from GUI, I ended with memory or resources allocation exception, but it works perfect if started directly from terminal and it's how it will be used at the end, so the limited memory should not be a problem.

Regards,

Jan

@waveform80

This comment has been minimized.

Copy link
Owner

commented Jul 30, 2014

Hi Jan,

I've just had a go with my model A and I can't seem to reproduce the error here, even when running X (it locates the Bayer data quite happily - it goes on to produce the error mentioned in #134, but that's not what we're concerned with here). I'm wondering if you've got an odd GPU memory split set or something like that? For example, on my model A I've got:

pi@raspberrypi ~ $ grep mem /boot/config.txt 
gpu_mem=128

Could you check your boot/config.txt for the memory split?

Thanks,

Dave.

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 31, 2014

Hi Dave,

thanks a lot for your effort. I checked memory split and I have the same as you. :-(

Personally (with a limited knowledge about Raspberry and internal Python structure) I don't think it's the memory issue. I tried a script mentioned in Documentation > Advanced Recipes > 5.11. Raw Bayer data captures. The only modification I did I don't use from __future__ import statement at the beginning, because it doesn't work.

The script failed very soon at assert data[:4] == 'BRCM', so I checked what is at data[:4] using print(data[:4]) and the result is b'BRCM' ... and because b'BRCM' != 'BRCM', the assert failed. And the same test (in slightly different form) if data[:4] != 'BRCM' is used in array.py.

Can from some reason the stream / data object be in not expected format, so the data are not processed correctly? Or this idea is wrong?

With regards,

Jan

@waveform80

This comment has been minimized.

Copy link
Owner

commented Jul 31, 2014

Doh! I'm an idiot for not picking up on this sooner as you did mention b'BRCM' earlier in the thread and for some reason it just didn't register! The issue is that you're using Python 3, or more precisely that I was too lazy to test PiBayerArray with Python 3 and that check fails because Python 3 won't allow a bytes object to be equated with a unicode object, while Python 2 will. I'll get that fixed up for 1.7 (the fix is trivial in case you want to patch it yourself in the meantime: just change 'BRCM' to b'BRCM' in /usr/lib/python3/dist-packages/picamera/array.py)

@waveform80 waveform80 added the bug label Jul 31, 2014

@waveform80 waveform80 added this to the 1.7 milestone Jul 31, 2014

@waveform80 waveform80 self-assigned this Jul 31, 2014

@waveform80 waveform80 changed the title Bayer data not located in stream. Bayer data not located in stream with Python 3 Jul 31, 2014

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Jul 31, 2014

Hi Dave,

The issue is that you're using Python 3

Is there any reason to use Python 2? ;-)

the fix is trivial in case you want to patch it yourself in the meantime: just change 'BRCM' to b'BRCM'

Yes, it is, I did it ... but it leads to already known broadcast error. But at least, it's the movement closer to solution!

Crossing my fingers you will be able to solve issue #134 soon and to release the fixes in version 1.7.

Regards,

Jan

@waveform80 waveform80 closed this in 28054c6 Aug 2, 2014

@waveform80

This comment has been minimized.

Copy link
Owner

commented Aug 2, 2014

Hi Jan,

Just pushed fixes for this and #134 - unfortunately I cocked up the commit for #134's fix and accidentally included it in something else I was working on (raw video capture). If you just want the bits for #134 just check out the picamera/array.py changes in the commit mention in that ticket (they're pretty minimal changes).

On the subject of Python 2 - the main reason I'm still using it at the moment is that several libraries I use aren't ported yet (python-daemon is probably the most important one). There's also an argument to made for performance (Python 3.3 and 3.4 are equivalent or faster than Python 2, but of course on Raspbian we're stuck on 3.2). Still, I really should set up something to run the test suite on both 2 and 3 (I hear tox is good for this sort of thing).

Dave.

@LabyrinthTechnology

This comment has been minimized.

Copy link
Author

commented Aug 4, 2014

Hi Dave,

thanks for the info, I will try the updated version of array.py.

Regards,

Jan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.