Home
Welcome to the SetPlayer wiki!
We aim to make a program that will play the card game Set for you, using nothing but a webcam, your computer, and... well, your hands to pick out the cards too. No thinking required!
In a bit more detail, we envision you holding a webcam over a table with a spread of set cards. Our software would then point out which cards visible to the webcam form a "set" (by the rules of the Set game). The purpose of this project is partially to create something fun (albeit possibly a bit useless... since your friends probably won't let you cheat), but also to develop skills in OpenCV, the computer vision library.
Just run "python settester.py" from your terminal and follow the instructions. If you don't use a webcam, it will prompt you for a path to some image instead (e.g. "../Pictures/solid_set.png"). Any sets would be printed to the terminal as well as displayed in an image. The image "solid_set.png" in our repository makes a pretty nice example.
Here's how our Set player currently works:
- Take an image from the webcam.
- Get Hough Lines from the image.
- Remove / merge extraneous Hough lines (i.e. ones that should be a single line).
- Find pairs of roughly perpendicular lines.
- Use the pairs to find |_|-shaped triplets (i.e. lines that roughly form 3 sides of a rectangle).
- Extract the corners of these quadrilaterals.
- Filter out unlikely cards (i.e. wrong aspect ratio)
- Extract the images bound by these quads and transform them into a standard card-shaped size.
- Filter out unlikely cards again (i.e. don't have consistent white borders).
- Get the card's color (count & compare pixels in red, green, and blue channels).
- Match cards against our database of templates (by directly comparing the differences between card and template).
- Count the number of pixels per shape to test for striped-ness.
- Find sets from these identified cards and display them.
Here's a screenshot of the code in action:
Right now, our program actually functions from start to finish. In other words, if you run it while holding a webcam over some set cards, it'll actually pick out and show you which cards it believes is a set. Granted.... sometimes it'll pick out sets that aren't, and not pick out sets that are...
We've worked on a few things since the last update. For one, we tried a few ideas to pick out striped vs open/filled cards:
- counting pixels using the saturation channel
- counting using just a grey-scaled image
- dithering the image instead
- masking out irrelevant parts with the template it matches against
- basing threshold values for striped vs non-striped on shape (i.e. ovals tend to have more pixels than squiggles, so taking that into account)
Picking out stripedness of cards turns out to be harder than we expected though... sometimes we get situations where filled cards have fewer pixels per shape than striped cards, for example =\ One idea we haven't tried yet, though, is to take a sample patch from the center of a shape, and measure using just that.
We also depend on reasonable lighting. If there's specular reflections that block out shapes with highlights, we also can't pick out what's what. (To be fair, a human looking through the webcam couldn't either).
We have made many changes and fixes! Here are the highlights:
- We merge similar hough lines
- There is now basic template matching, so the program can identify cards
- We eliminated a lot of cases for incorrect cards, fixing the "too many possibilities on a grid" problem mentioned last time
- Duplicate card outlines get removed
- We connected the app to our webcam so it can operate in real time
Overall, the system is able to identify cards with high reliability. It gets confused about striped versus solid or empty cards, but we intend to fix that soon. The segmentation routines are very good at ruling out non-cards, but they might be too aggressive; we often miss obvious cards as well.
Next steps:
- Identify striped cards correctly
- Find matching sets and display that information
- Try some ideas for improved segmentation
We've made quite a bit of progress without logging anything here (woops! we sure are on top of things!) Our algorithm currently extracts (some) of the cards, and correctly identifies (most of) them!
Here's how it currently works:
- Find the edges of the cards (using Hough Lines, Canny Edge Detection, and the value from HSV)
- Combine edges to get possible card outlines (checking for right angles)
- Filter out desired outlines (eliminating duplicates and outlines that don't have all-white borders, which cards should)
- Extract card images from these outlines (and transform them to their "canonical form" :D)
- Match shapes and numbers against our training-card images (XOR thresholded versions of the cards together and count the remaining pixels)
- Get the colors (by examining RBG channels)
tada!
Current issues to resolve / improve on:
- We don't recognize striped cards yet - we just have open and filled. However, we have most of the code implemented for striped recognition (we're just counting how many pixels the shapes take up, and categorizing them accordingly :) hooray simple stuff)
- If we line the cards up in a neat grid (like you would in any 'ol round of Set), we recognize wayyy too many rectangles as potential cards and significantly slow the process down. We're currently fast only if the cards are scattered every-which-way... which often isn't the case for games in the real world.
- Some card edges get split up into two lines instead of one, making us loose the ability to recognize it as a card (the issue is apparent with the double-squiggle card in black_bg_3.png at the moment). A naive solution would make the previous one worse, I think.
- We don't notice all the cards if they go off the screen too much. This might actually be reasonable.
In addition to some messing around, coding, storing test images, and whatnot, we started thinking of some ideas.
We need to extract the specific Set shapes before we can try recognizing them. Just how we do that, though, miiight have several possibilities. Maybe only one of these possibilities works. We'll see.
- just extract contours and figure out which ones make sense
- eliminate everything in the picture except for the shapes, so we only have the contours we need
- block out which parts are the cards, using their rectangularish shape and white background, and take it from there
- use color channels and/or hsv channels to extract which parts matter
We successfully installed OpenCV and wrote a HelloWebcam to demonstrate that our basic inner workings are in ship-shape. This little program demonstrates that OpenCV is indeed working on our hardware, and that we have access to the webcams.
Additionally, we procured access to a deck of Set cards, and got GitHub (for sharing and versioning control) up and running.
All of the above, plus some brainstorming and setting up this page, took about 6 hours.
Tasks for this project include:
- create a GUI for playing
- create a GUI for testing / debugging
- collect data for testing / debugging (i.e. pictures of Set cards through our webcams)
- develop the image recognition <- this will probably be the hard part
- develop the game AI
- connect it all together!
BATTLE PLAN:
- Get started with image recognition and a simple GUI for testing / debugging. Have the testing / debugging tools more or less complete by the end of 2 weeks.
- Work on whatever part we're in the mood for afterward
- By the end of 4 weeks (2/28), have some level of image recognition functioning (i.e. recognize Set for a subset of Set sets)
- By the end of 6 weeks (3/9), have a complete program that can play Set as described above.