-
Notifications
You must be signed in to change notification settings - Fork 30
Software Technical Overview
This is an overview of the rpi-film-capture software architecture. The current version is written in python 2.7
The GUI was designed in Qt 4 Designer and translated to python via the command pyuic4 fcClientgui.ui>fcClientgui.py
See the software setup section for details on the required libraries.
###High-level overview This software operates as two python programs: A server running on a headless raspberry pi, and a client on a more powerful desktop computer. Together, they capture image files representing individual film frames, which can later be combined into a single film.
The server needs to handle a lot of different roles simultaneously: It listens for commands from the client, operates the stepper motors to drive the projector, adjusts the camera, and takes and streams those images to the client. Fortunately, none of these tasks are particularly processor-intensive and a Pi 2 or 3 can handle them easily. To help speed things along, we use multiple threads so as not to block on I/O, and when driving the stepper motor, we use multiple processes so that stepping is handled by a separate core on the pi. (Adding an arduino into the design would probably help.)
The client runs the GUI, sends commands to the server, processes captured images, and displays the resulting image.
Together, they operate in two 'modes':
- Preview mode allows the user to adjust the camera's field of view and color/exposure settings, and to run the projector forwards/backwards to position it on the proper frame. The user can also save their settings to a file so they don't need to be reloaded every time. In this mode, the server streams single images to the client as quickly as possible to simulate a 'live' video feed.
- Capture mode lets the user control the parameters behind the actual capture process: Image size, shots per frame, setting the initial exposure, and specifiying numbering/location of captured images. In capture mode, the user can take test images using the same parameters to be used during capture, and can of course start/stop the actual capture process. Captured images are displayed in a separate window once processed.
##Key Design Factors The motivation behind this design is to maximize capture speed while sacrificing as little quality as possible, while giving the user some control over the tradeoff between speed/quality.
In my tests, image capture was fairly fast, but saving the images to USB or CF card was much slower. Any processing on the images slowed down the capture more. I found that the pi could write images to a network stream faster than I could save them. By streaming the images over a fast network, we can offload processing and keep the actual capture operation as fast as possible.
The lossiness and low bit depth of jpeg makes it an unlikely candidate for film capture, but Pi's low-end camera mean that other formats are both considerably slower to use and provide only small quality improvements. Fortunately, some other techniques help us to deal well for jpeg's drawbacks while still benefitting from it's size and speed advantages. Read on...
Film generally supports a much higher dynamic range (difference between the lightest and darkest colors) than is supported by most digital sensors, particularly inexpensive ones like that on the PiCamera; and 8-bit JPEG images are not capable of representing that higher range anyway.
For films with with well, evenly lit shots isn't bothersome; the camera's auto-exposure will often capture an excellent image. But many home movies are shot in less-than-ideal conditions: Improperly shots, deep shadows, and glaring sunlight are common. The PiCamera often loses important detail in these shots. And even worse, 'washed-out' areas in digital photographs don't just lose detail; they are distracting, more than dark areas.
Several techniques exist for combining multiple exposures, but most are quite tedious and require calibration data about the camera and the exposure. However, one algorithm (accessible through merge_mertens works well with any set of multiple images, is reasonably fast, and is supported through version 3 of OpenCV. We use this bracketing/merging technique to capture detail in high-dynamic range images.
(These algorithms are also used by the enfuse command available in the enblend package, for those who want to try blending images via the command line without compiling OpenCV 3.
rpi-film-capture allows the user to capture each frame using an arbitrary number of bracketed photos at an arbitrary range of (effective) f/stops between the brightest and darkest. We do this by varying the camera's shutter speed for each shot from the 'default' one initially suggested by the camera at the start of the capture. So, for example, if the shutter speed was originally set by auto-exposure at 12ms, and the user asks for 3 bracketed shots per frame at a spacing of 2 stops, the server would take exposures at 6ms, 12ms, and 24ms. Determining what level of bracketing is needed for quality is up to the user, but 3 shots/2 stops often works well for me.
Obviously multiple shots per image seems like it should involves a significant performance hit, but it needn't necessarily, because of what else it allows us to do: Read on.
In my tests, image quality for SINGLE images is noticeably better when using the stills port: It uses an internal denoising algorithm to provide noticeably cleaner images. However, capture from the stills port is considerably slower, on the order of 300ms per image. For 3 shots per frame on a 50-foot reel, this could add up to another hour of capture time, about doubling it.
However, blending multiple shots of a frame together also seems to effectively suppress noise quite well; to my eye, blended shots from the video port look about as good as blended ones from the still port. This technique also helps to somewhat mitigate both the bit-depth and lossiness of jpegs. Capturing and combining 3 images is actually faster capturing a single PNG or TIFF, and seems (to my untrained eye) to result in consistently better quality.
##Client-server communication: Messages from the client take the form of a single ASCII character code, sometimes followed by a short string representing a value, e.g.: 'v': Flip image vertically 't': Capture a test image and send it to the client 'y': Adjust '>': Wind the projector forward (At this time, the server does not send acknowledgement of commands; we just assume that they have arrived.) In addition to the command protocol, the server also sends data back to a thread on the client, primarily but not exclusively in the form of images. All server->client data is in the form (image type character)(image size)(image data) or
FINISH THIS!