The Smart(er) Machine-Learned IoT Cat Feeder for Lola the Cat

tl;dr – I modified my cat feeder to only feed one of my two cats using a Raspberry Pi webcam and image analysis with TensorFlow.

Meet Lola

My cat for seven years who is healthy and fuzzy and an all-around good cat. However, she is a very picky eater. She takes small bites of food sporadically throughout the day from a food dish just for her on top of a tall dresser.

Meet Madison (Maddie for short)

Obese cat that was recently put on a strict diet of canned food twice a day (no carbs!). Due to her obesity, she’s been unable to leap onto the dresser to eat Lola’s food. Until now. Perhaps she’s regaining some athletic ability with the new diet, or her appetite has overwhelmed her fear, but a couple weeks ago she finally made the leap to the bounty of Lola’s food dish. Not good.

The Problem

Lola likes her dry food (a mix of Science Diet and Orijen), and only eats a tiny bit at a time whenever she wants. I’ve made several attempts at getting her on a schedule, but this results in incessant yowling at all hours along with a stubborn refusal to eat during the allotted mealtime. All was well when Lola was the only cat with the leaping ability to reach the skyward food dish. Now there’s no place for cat food to hide that Maddie won’t find.

I needed a way to keep Maddie on her diet, but allow Lola to peck at her food when she decides it’s mealtime. If tiny portions of food could be presented to Lola, she would eat all of it leaving none for Maddie. But how to get Lola (and not Maddie) tiny portions of food when I’m away at work or sleeping?

The Feeder

I bought an automated cat feeder a few years ago to dispense food to the cats whilst on vacation. It’s sturdy, but very primitive compared to today’s IoT designs. It works by plugging into a wall timer, and dispensing food for a short duration when AC current is supplied, and then resets when the power is cut. The amount of food dispensed is determined by twisting a small potentiometer with a tiny screwdriver (seriously!).

If the Feeder Only Had A Brain

I have a cursory fascination with machine learning, and sometimes read about what’s what in the ML world. TensorFlow has been a hot topic of late, and they have a few dead simple tutorials for lay people like myself. TensorFlow For Poets shows how to retrain an image labelling model to recognize objects of your choosing… like perhaps your own pets.

I thought maybe, just maybe, I could fiddle with the TensorFlow tutorial’s code and cobble together a system that allows the feeder to ‘recognize’ Lola and dispense a tiny meal for her, but refuse service to Maddie. Given the simple nature of the feeder (plug in = food, unplug = reset), some sort of smart-plug for the feeder would work just fine.

The Bright Idea

I have a couple Raspberry Pis lying around (what self-respecting tech nerd doesn’t?), and decided to use one for the guts of the operation. The full solution looked something like this…

Set up the RPi + camera to watch for cats mulling about near the feeder. Periodically snap a photo, and analyze said photo with TensorFlow to find Lola. If Lola is identified in the image, flip the switch on a smart plug or relay connected to feeder with the RPi. Then shut off the power to the feeder and sleep for an hour or two before allowing another feeding. We don’t want it constantly spitting out food while Lola is eating.

The Working Contraption

After a few days of tinkering, trial and error, ordering bits and bobs from the internet, and convincing my wife that I’m not crazy, I finally got a working version! As I suspected from the outset, it’s very much cobbled together and has a few quirks, but it does what I set out for it to do – feed one cat, but not the other.

(Quick reminder, I do still feed Maddie, but only twice a day with incredibly expensive cat food that I swear is better than what I eat some days)

The parts list for the working prototype are as follows…

• The SuperFeeder brand cat feeder
• A Raspberry Pi 3
• 32GB MicroSD card for RPi (with latest Raspbian)
• 5v power adapter for RPi
• Raspberry Pi Camera v1 (5mp)
• RF Controlled Outlets ( ETekCity )
• RF transmitter and reciever
• Thin copper wire (for an antenna)
• A box and/or stand for the RPi

The software involved includes…

• Raspbian OS
• TensorFlow
• Python (and Flask)
• Node.js
• ffmpeg
• IFTTT/Twitter/Twilio account (optional)

The “How it Works”

See a video in action here.

The RPi is set up near the feeder in a small box. The camera is pointing to the area in front of the feeder. After turning on the feeder, a python script bootstraps TensorFlow which loads the trained model into memory and gets ready to analyze an image. It waits until it is triggered by a GET request set up with Flask. So TensorFlow twiddles its thumbs, waiting patiently for some work to do.

Concurrently, A Node script fires up to do the ‘detection and feeder’ cycle. The script goes a little something like this…

A picture is taken with the camera. After the photo is saved to disk, the Node app calls the python service which analyzes the image and returns some JSON data. The data includes the probabilities of Lola and Maddie being in the image. If the “Lola probability” is greater than 99.3%, the Node script sends an RF signal to turn on the outlet with the connected feeder plug. The feeder gets power and dispenses a tiny bit of food. After a few seconds, the Node script sends an RF signal to turn off the feeder, then sends a message to IFTTT to alert me that Lola got food. It also copies the last image taken into another folder and timestamps the filename. The cycle then times out for 90 minutes, and then resumes taking pictures.

It all works surprisingly well, except at night when the camera sees only blackness. Luckily, the RF controlled outlets come in packs of five, so I hooked up a lamp to another one and have the Node script also turn the light on for a couple hours in the evening, and early in the morning (Lola really likes to eat at 5:30am, and she lets us know it).

The IFTTT notification and copied image are nice-to-haves that let me keep an eye on things until I get a better interface put together. It also lets me check out any false positives that might occur, and give me better ideas for retraining the TensorFlow model.

EXTRA DETAILS

Below are some more in-depth implementation details.

Raspberry Pi 3

A Raspberry Pi 3 has just enough horsepower to run pre-trained TensorFlow apps, and a bunch of code for getting up and running on the Pi. There’s also a burgeoning IoT and hacking community, which makes is a great hub for controlling internet connected stuff.

A basic Pi setup needs a bit of hardware:

• A SD Card. I recommend 32GB – 64GB to have plenty of room for TensorFlow models, pictures, and video.
• 5V Power. There’s no power cord, so I got a 10ft. micro USB cable and 5V adapter.
• A RPi case. Plenty of these out there. I got a cheap plastic one for a couple bucks on eBay.

And the basic software setup:

• Raspbian & Pixel Desktop. I went with the new desktop GUI rather than headless to try out Pixel, and to do development and testing right on the Pi itself without another computer.

• Add a hidden network to the Wifi list:

  sudo iwlist wlan0 scan essid [yourSSID]

• Less cruft. Raspbian comes with too much stuff. Easy to remove, though.

• Node.js 7

• VNC. In case I did want to log in remotely, I had to fiddle with the video settings in /boot/config.txt (Using hdmi_group=2 and hdmi_mode=27 did the trick).

The Raspberry Pi Camera

I read a few places that the version 1 (5MP) camera module is better with auto focus than the v2 (8MP) module, so I got a v1 from eBay for about $15. I’m not dissappointed. I ended up using 600×600 pictures in the project anyway, so extra megapixels would have been wasted, and all the features (focus, white balance, filters, rotation, etc…) work great.

There are some clones of the camera module from China for a few dollars less, but why risk it? I did, however, order a protective camera covering from China for $1. I’m still waiting on it as of this writing, though.

Remote Control Sockets and RF Transmitter

Here’s where the real fun begins. To turn the feeder (and accompanying lamp) on and off, I used remote controlled outlets, but spoofed the remote control frequencies with the Rapsberry Pi and a radio frequency (RF) transmitter attachment. In order to get the RF codes to activate the outlets, a RF receiver attachment is necessary as well. Luckily, the receiver and transmitter are sold in pairs, and are really cheap.

The basic steps for the setup are as follows:

1. Install WiringPi and RFSniffer
2. Plug in the reciever to the Raspberry Pi.
3. Start RFSniffer.
4. Push all the buttons on the remote control, and write down all the codes. It will look something like

  Received 21811
  Received pulse 192
  Received 21813
  Received pulse 192

1. Plug in the transmitter to the Raspberry Pi.
2. Test the codes with codesend. Be sure to substitute in your own code and pulse values read from RFSniffer. ./codesend 21811 -l 192 -p 0

When buying outlets, the popular choice is ETekCity. Amazon was sold out of these when I was looking, so I got another brand. One of the outlets broke a day later, so I’d recommend ETekCity.

Also, the first RF Transmitter I got was busted. I noticed that the circuitboard was slightly different than those pictured in the guides I used. Instead of ‘data’ printed on the transmitter, it said ‘ADAT’. I returned the set, and ordered a receiver/transmitter with ‘DATA’ and everything worked fine.

Training TensorFlow

How does a computer know the difference between two cats? An image classifier created from a trained neural network, of course! Seriously though, I’ve always figured the realm of machine learning is out of my reach, but some of the tools are becoming more accessible to lay-people. Luckily YouTube had plenty of resources.

Learning About Machine Learning

With a confident attitude I went out searching for some ideas on how to create my own classifier. Luckily I didn’t have to look far. Google’s most basic introductory tutorial happened to fit my problem like a glove. Tensorflow for Poets takes you from nothing to a custom image classifier in less than an hour. It also leaves you with all the tools you need to keep using it with your own images… which is what I did.

Cat Training

The basics of creating the classifier went like this…

I needed pictures of my cats. Lots of them. Like hundreds. So I fired up the picam on the Raspberry Pi to record for a couple minutes while ‘nudging’ each cat in front of the camera in various positions and with different lighting. For this I used the raspivid program that comes with Raspbian. The following command records ten seconds of video:

raspivid -o video.h264 -t 10000

For image extraction, I used ffmpeg and the following command to output an image for every second of video:

ffmpeg -i video.h264 -vf fps=1 out%d.png

The ffmpeg docs have lots of useful snippets. Oh, and ffmpeg can be installed as a command line utility via Homebrew on Mac (brew install ffmpeg).

The images were all in PNG format, and I needed .jpeg for Tensorflow. Luckily, OS X has a built in batch conversion tool right in the Preview app. See OSXDaily for details. After the conversion, the images were sorted into two folders – one for Lola, the other for Maddie.

For the training, I simply repeated step 4 in the Tensorflow for Poets tutorial, but used ‘lola’ and ‘maddie’ pictures instead of roses and daisies. I used 4000 iterations for extra thorough training. The basic command is below. Of course, instead of flower photos, I obviously had cat photos.

# In Docker
python tensorflow/examples/image_retraining/retrain.py \
--bottleneck_dir=/tf_files/bottlenecks \
--model_dir=/tf_files/inception \
--output_graph=/tf_files/retrained_graph.pb \
--output_labels=/tf_files/retrained_labels.txt \
--image_dir /tf_files/flower_photos

The main output of this training is the model file (retrained_graph.pb), and the accompanying labels file (retrained_labels.txt). These are the files used in the program that will examine new pictures of my cats and determine which one is which (if any cats are present at all).

Finally, I could take the label_image.py script from Step 5 of the tutorial, and use that to analyze new photos of my cats and tell me the confidence of Lola or Maddie being present in the image. All this runs from inside the original Docker image like so:

curl -L https://goo.gl/tx3dqg > $HOME/tf_files/label_image.py
docker run -it -v $HOME/tf_files:/tf_files  gcr.io/tensorflow/tensorflow:latest-devel 
python /tf_files/label_image.py /tf_files/new_lola_pic.jpg

Notice in the last line I used a picture of lola instead of a daisy. The output was something like:

lola (score = 0.99138)
maddie (score = 0.35342)

Even though Maddie was not in the picture, the confidence was pretty high. I’m guessing because Maddie is a nearly solid black cat, any dark mass in the photo may appear to the model as Maddie. Luckily, if Maddie actually is in the photo, the confidence is much higher. Here’s a score with Maddie, but no Lola:

lola (score = 0.01483)
maddie (score = 0.99683)

Good Job, computer!

Tying it Together With Code

Now that I have a Raspberry Pi that can take pictures and turn electrical sockets on and off, as well as a trained image classifier that knows my cats, it was time to stitch everything together.

The Lola Detector

The following steps were taken to construct a small program that would scan an image on demand and identify Lola (or not):

1. Install TensorFlow on the Raspberry Pi as a Python library.
2. Refactor the example image label Python script to start TensorFlow, load the model, and wait.
3. Use Flask to create a URL that would kick off the image analysis function, and return a JSON object with the Lola/Maddie label probabilities.
4. Keep this program running.

Please note, this is the first Python script I’ve ever written. Feedback (maybe) appreciated.

The Lola Feeder

Rather than use Python for the whole app (like a sane person), I opted for Node to do the rest of the stuff. A big, convoluted Node script does the following:

1. Takes a picture with raspistill and saves it as rpicam.jpg.
2. Sends an http request to the Lola Detector service, and waits for a response.
3. Checks the response for a high Lola probability.
4. If Lola is NOT found, go back to step 1.
5. If Lola is found, send a signal via the FM transmitter to power on the cat feeder.
6. Send a tweet with the recently taken picture to a hidden Twitter account.
7. Send a message to IFTTT which sends me a push notification.
8. Wait 60 seconds and send another signal to power off the feeder.
9. Wait 90 minutes and go back to step 1.

Additionally, I have a small lamp attached to another RF power socket, and have a “cron job” that turns the lamp on for a few hours in the evening, and a few hours in the early morning.

Here’s the code for the RF thing. I also have a white noise machine, another lamp, and some Christmas lights hooked up to other RF outlets. Unfortunately, the Xmas lights outlet stopped working :(

To run and monitor both scripts, I use PM2. It’s awesome. You should use it, too.

On the TODO List

• Definitely a sturdier box and camera mount. Lola has already threatened to knock the whole thing off the desk (as cats do), and barely a tap will knock the camera out of place.
• A PIR motion detector to trigger the camera instead of running the camera non-stop. Not sure how useful it would be, but I have a PIR sensor, so why not?
• A database to record data. Collect feeding times and pictures to get a better idea of Lola’s eating habits.
• A web interface. It would be nice to adjust the timeout between feedings, manually trigger a feeding, control the lamp, and generally see what’s going on via the web.
• A Maddie scarecrow. Sometimes Lola does’t quite eat all the food, and Maddie jumps in behind her to finish the job. Might be worthwhile to play a loud noise or something if Maddie is detected (but not Lola) soon after a feeding occurs.
• 100% Python. I used Node because I know Node, but there’s no reason this couldn’t be one Python script instead of two separate programs. Someday I’ll fulfill my dream of learning Python, and may refactor this project accordingly.