This is part of a project developing a fully autonomous ecosystem monitoring unit. The full details of the device is described in an academic paper, easy to follow step by step instructions of setting one up from scratch can be found on our website, and this page focuses more on the details of the software running on the device, targeted at more technical users.
setup.py script is used to configure the required sensor to be used for data capture and creates a JSON config fle. Once a sensor configuration has been created, the recorder is started up using
recorder_startup_script.sh, which runs the
record() function from
python_record.py. This then does the following:
- Sets up error logging.
- Logs the id of the Pi device running the code and the current git version of the recorder script.
- Loads the config file.
- Sets the reboot time.
- Checks the working and upload directories for data files and copies previous logs into the upload directory.
- Runs the
configure_sensorfunction to instantiate a sensor class object.
- Attaches the function
exit_handlerto run if a SIGINT signal is detected either from reboot or user interrupt.
- Creates a thread instance that executes the FTP synchronisation at a server sync interval defined by the sensor config using the
- Creates a thread instance that runs the
continuous_recording()function. This function is just a wrapper that repeats the
sensor_recordfunction while the thread is running.
sensor_recordfunction itself executes the sensor methods: a)
sensor.capture_data()to record whatever it is the sensor records; b)
sensor.postprocess()is run in a separate thread to avoid locking up the
sensor_recordloop; and then c)
sensor.sleep()to pause until the next sample is due.
- When a SIGINT occurs then
exit_handlerintercepts SIGINT and raises a
StopMonitoringexception to exit the recording. The exception handling sets a threading event instance that has been passed to the two threads running
continuous_recording(), and signals that the functions running in these thread should finish their current loop and exit. The
record()function then exits.
- As long as
recorder_startup_script.shis setup to run on boot, then the process repeats from the first step.
Setup from our pre-prepared SD card image
To setup the monitoring unit from our pre-prepared SD card image follow these steps:
- Boot the Raspberry Pi with our prepared SD card inserted. Let the startup script run until it exits with the message "Config file not found!". If you would like to change an existing configuration, press
Ctrl+Cwhen you see "Start of ecosystem monitoring startup script"
python setup.pyand follow the prompts. This will create a
config.jsonfile which contains the sensor type, its configuration and the FTP server details. The config file can be created manually, or imported from external storage without running
- Make sure the timezone is set correctly. Check by typing
sudo dpkg-reconfigure tzdataand following the prompts
- If your SD card is larger than the size of our pre-prepared image (4GB) run
sudo raspi-configand choose: Advanced Options -> Expand Filesystem. Press
Escwhen this is complete
sudo haltto shut down the Pi
- Take the microSD card from the Pi, and make a copy of it onto your computer (How?). Now you can clone as many of these SD cards as you need for your monitoring devices with no extra setup required
Setup from a stock Raspbian image
If you would rather start using a stock Raspbian image, there's an extra couple of steps before you start the above process. The below steps assume you have downloaded and installed the Raspbian Stretch Lite image.
You will need the Pi to be connected to the internet for the below process.
- Login when prompted with user 'pi' and password 'raspberry'
sudo raspi-configand configure the Pi to boot to a command line, without login required: option Boot Options -> Desktop / CLI -> Console Autologin. Press
Escwhen this is complete
- Install git:
sudo apt-get install git
- Clone this repository in the home directory of the Raspberry pi:
git clone -b lts https://github.com/sarabsethi/rpi-eco-monitoring.git(see below regarding branches)
- Make sure all the scripts in the repository are executable:
chmod +x ~/rpi-eco-monitoring/*
- Configure the Pi to run
recorder_startup_script.shon boot by adding
sudo -u pi ~/rpi-eco-monitoring/recorder_startup_script.sh;to the last line of the file
- Install the required packages:
sudo apt-get -y install fswebcam lftp libav-tools usb-modeswitch ntpdate libvpx4 zip
- Then follow the instructions above to complete the setup
N.B. This clones the long-term support branch, which will have software that has been extensively field-tested, whilst the
dev branch will have the latest development code which may inherently be more unstable. For long remote deployments we recommend only using the LTS branch, and this is the branch used in our pre-prepared SD card images. If you plan on implementing a new sensor, fork the codebase and make your changes, but be sure to submit a pull request back to this repo when you're done!
Implementing new sensors
To implement a new sensor type simply create a class in the
sensors directory that extends the SensorBase class. The SensorBase class contains default implementations of the required class methods, which can be overridden in derived sensor classes. The required methods are:
__init__- This method is loads the sensor options from the JSON configuration file, falling back to the default options (see the
optionsstatic method below) where an option isn't included in the config. The
__init.py__file in the
sensorsmodule provides the shared function
set_optionsto help with this.
options- This static method defines the config options and defaults for the sensor class
setup- This method should be used to check that the system resources required to run the sensor are available: required Debian packages, correctly installed devices.
capture_data- This method is used to capture data from the sensor input. The data will normally be stored to a working directory, set in the config file, in case further processing is needed before data is uploaded. If no further processing is needed, the data could be written directly to the upload directory.
postprocess- This method performs any postprocessing that needs to be done to the raw data (e.g. compressing it) before upload. If no post processing is needed, you don't need to provide the method, as the default SensorBase implementation contains a simple stub to handle calls to
sleep- This method is a simple wrapper to pause between data captures - the pause length is implemented as a variable in the JSON config, so you're unlikely to need to override the base method.
Note that threads are used to run the
postprocess methods so that they operate independently.
For worked examples see classes made for monitoring audio from a USB audio card (
USBSoundcardMic.py) and for capturing time-lapse images from a USB camera (
TimelapseCamera.py). For a really simple example, see the UnixDevice sensor (
UnixDevice.py): this just demonstrates the use of the class methods to read data from one of the basic system devices.
from sensors.YourNewSensor import YourNewSensor to
This is a cross disciplinary research project based at Imperial College London, across the Faculties of Engineering, Natural Sciences and Life Sciences.
Sarab Sethi, Rob Ewers, Nick Jones, David Orme, Lorenzo Picinali
Feel free to drop me an email with any questions, and contributions to this codebase are always welcome.