When we get ready in the morning, we spend so much time in front of the mirror. While it is important to make sure that we look presentable, we wanted to improve the functionality of the mirror. Mirrors have existed since 6000 BC according to wikipedia, but mirrors have not changed much since then. We wanted the mirror to show a reflection and display information. In the morning, we might want to know what the weather is going to be throughout the day, or we might want to know what is on our calendar. It is easy to get bored while brushing teeth, so we wanted a youtube video for entertainment with a timer. With these ideas in mind, we decided to build the smart mirror. The goal of the smart mirror was to provide the aforementioned capabilities onto a visually appealing mirror. We wanted to make a product that we would enjoy using, so we spent a lot of time working on the details. This focus on all aspects of the mirror helped us create a viable commercial product.
Mirror with a background displaying time, date, and current temperature with multiple apps for weather forecast for the day, a short Youtube video from a selected channel, and the user’s Google calendar events for the day. The system includes six LED segments representing a 2-minute timer, two photoresistors controlling the brightness of the LEDs, three buttons and voice recognition for user control. The Smart Mirror supports multiple users with their own personal calendar. The users alongside their Youtube channel and calendar and wifi are preprogrammed into the Raspberry Pi and could be reprogrammed using a mouse and a keyboard. The user has the following control over the Smart Mirror: ● Left click: password input or moving between apps ● Left long press: shutdown (schedules shutdown for 30 sec in the future) ● Right click: password input or moving between apps ● Right long press: log in as guest or logout user ● Center click: password input ● Center long press: reboot ● Voice recognition ○ Calendar: show calendar ○ Youtube: show Youtube video ○ Weather: show weather forecast ○ Logout: log the user out ○ Sleep: put the Smart Mirror to sleep ○ Wake up: wake up the Smart Mirror
The figure above presents a high-level flowchart of the Smart Mirror system design. The overall circuit was designed and tested on a breadboard, and then designed on a PCB using altium summer 09. The PCB layout can be seen in the figure below, and the schematics are available in the appendix. The PCB was mounted on top of the Raspberry Pi using a 40-pin header. The PCB had all the necessary connections between components and the Pi built-in and was wired directly to the different components (LEDs, buttons, photoresistors) through 22ga wires.
Three momentary on/off waterproof pushbuttons were connected to three GPIO pins on the Raspberry Pi using the built-in pull-down resistor circuit. The signal from the push buttons remains low and turns high when the push button is pressed.
Six RGB LED segments (12 LEDs each) were connected to six Raspberry Pi GPIO pins through a power MOSFET circuit (figure 3.4) to control when each segment is on. The RGB pins were all connected together to three PWM outputs controlling R, G, and B through another MOSFET circuit (figure 3.5) to control the brightness and color of the LED segments (all segments have the same color and brightness).
Two photoresistors were mounted to the top of the frame to read the ambient lighting in the room. If the room was bright, we made sure that the LEDs shined brightly. If the room was dark, we dimmed the LEDs. Unfortunately, the Raspberry Pi does not have analog inputs, so we could not directly wire the photoresistor output to the Raspberry Pi. Instead, we used an ADC to convert the photoresistor output to 8-bit digital outputs. We decided to use only the 5 most significant bits from the ADC to measure the room brightness because the 3 least significant bits were not influential on the photoresistor reading and in order to minimize the number of Raspberry Pi pins used.
Voice was captured using a Kinobo USB 2.0 Mini Microphone “Makio” and used to recognize certain key words. At first, the recordings were sent to an online service, which returned the word(s) in the recording. While this method worked, it took a long time for the server to respond with the word(s). Instead, Snowboy Hotword Detection was used. This open source technology is used on the Amazon Alexa to wake up the device when a user says “Alexa.” Initially, a similar approach was taken to wake up the device (maybe when the user says “Miroslav”). However, since only 6 hotwords were needed (weather, youtube, calendar, Miroslav, log out, wake up), Snowboy was programmed to detect multiple words. When the user says “weather,” the display changes to the weather screen. Using Snowboy on the Raspberry Pi required many days of debugging, configuration, and changing device settings. The Microphone is always listening, and continuously compares voice input with available sound files that were personalized by the developers. Unfortunately, in order to get accurate keyword detection, the word needs at least 500 recordings. The only available option for this project was to record a personal keyword with 3 recordings only, so the recognition not as accurate as it could be.
Behind the two-way mirror, a 20 inch Samsung TV was mounted. When the screen was black, only the reflective portion of the mirror was visible. When the TV displayed output, the content was clearly visible through the screen as well as the mirror reflection. The TV did not have speakers or any controls besides switching between HDMI and VGA screens. The Raspberry Pi HDMI port was connected to an HDMI to VGA and AUX adapter, in which the VGA was connected to the TV and the AUX was connected to an external speaker.
In order to control the Raspberry Pi OS, a mouse and a keyboard are needed. The Pi is mounted on the back of the Smart Mirror with easy access to the USB ports. Neither a mouse nor a keyboard is required to use the Smart Mirror. They are only needed to preprogram user details and to debug and change software.
To access the internet, either an ethernet cable or a wifi connection would work.
All our devices used power from a standard 120V outlet. The Pi, TV, speaker, and LEDs all connected to a power outlet through a surge protector power distributor. The LEDs were powered with 12V through an adapter, which was connected to the PCB to provide power to the LEDs through the MOSFET circuits. The Raspberry Pi was powered with 5V through another adapter. The TV was connected directly to the 120V supply.
The software for this project was broken into two portions. We had a web app for the display and c++ code for interacting with hardware. To communicate between our website and the c++ code, we used system calls that simulate keyboard presses. The web app listens for keyboard presses and reacts accordingly.
The web app was split into an html file and a js file. The html included the display for the entire application (including weather, youtube, calendar, and log in). The html always shows the current temperature in Durham, a title for the screen, the date, and the time. In the js code, we made sure to hide the divisions that were not being used (i.e. when we were on the weather screen we made sure to hide all the other screens). The js code handled all the functionality for the web app - most importantly it listened to the button presses and acted accordingly. We allowed to press left/right/center to switch between pages or also allowed direct switching to pages with voice commands. We have 3 different users and a guest programmed to the device, so the js ensures that the correct information is displayed for each person. We loaded the weather info from openweathermap.org API and loaded the youtube video/played and paused it from the youtube API. For the calendar, we used Google’s calendar API. In order to maintain access to a connection to a user’s calendar, we used access and refresh tokens. An access token is an expiring key that is sent in the get request to authorize the request and retrieve the info for that user. A refresh token does not expire and can be used to request a new access token when the access token expires. This allowed us to get a user’s calendar events without signing in every time. The js handles logging in (with correct/incorrect logon parameters), switching users, and other important web app features. Maintaining concurrency between the web app and the hardware was one of the trickiest parts of this project, but we believe the web app worked well. The webapp consists of an html file and a js file. Both get served on localhost:8000 during start up of the C++ code. The entire app is built within one webpage, which allows for better responsiveness since the app can just show/hide divs by responding to key presses instead of loading an entire new webpage. The app has a few states: loggedOut, loggedIn, and asleep. When a user is loggedOut, the app displays a welcome page and waits for a signal to log in or display helpful messages to the user, such as password input, password success, or password fail. When a user is loggedIn, the app displays the user name and can switch between calendar, youtube, and weather. When a guest is loggedIn, the app does not display the calendar, since a guest has no google calendar account. When the app is asleep, it does not respond to any user input except for the wake command. The following is a list of events that the web app responds to. All are key presses, that can be emulated by the c++ code by using xdotool. Global events (app always responds to them) ● z: Sleep (turn display off) ● o: Wake (turn display on) LoggedIn events (app only responds to them when user is logged in) ● Left arrow: switch to app to the left ● Right arrow: switch to app to the right ● c (show calendar if not guest) ● w (show weather) ● y (show youtube) ● m (show miroslav) ● l (logout) LoggedOut events (app only responds to them when user is logged out) ● p (add star for password input) ● s (show password success message) ● f (clear stars and show password fail message) ● 0 (login as guest) ● 1 (login as user 1) ● 2 (login as user 2) ● 3 (login as user 3) Calendar The app loads a user’s calendar by submitting a get request to the google calendar api for all events within the next 24 hours. It extracts the data from these events and displays up to the first 3 of them in a custom UI. The app stores a refresh token for each user (retrieved beforehand through another get request), uses the refresh token to retrieve an access token, and authenticates the user using the access token. Youtube The app loads a random video from a specific youtube channel. When the user switches to youtube, the video automatically plays. When the user switches out of youtube, the video automatically stops playing. Weather The app loads weather data from the openweathermap.org API, extracts the appropriate data and displays it in a custom UI.
We have one C++ file “smartMirror.cc” that handled all threads. The C++ was split into six threads: left button thread, center button thread, right button thread, photoresistor thread, LED thread, and the main thread.
- Left Button Thread Polls for left button press every 5 ms. Minimum press registered is 30 ms. a button press (less than two seconds) handles moving to the previous app (move left) and password input. A long button press (longer than 2 seconds) shuts down the Raspberry Pi for a safe power down.
- Center Button Thread Polls for center button press every 5 ms. Minimum press registered is 30 ms. a button press (less than two seconds) handles password input. A long button press (longer than 2 seconds) reboots the Raspberry Pi.
- Right Button Thread Polls for right button press every 5 ms. Minimum press registered is 30 ms. a button press (less than two seconds) handles moving to the next app (move right) and password input. A long button press (longer than 2 seconds) logs in as guest, and logs out any user.
- Photoresistor Thread Reads the voltage across each photoresistor every 100 ms. These two values are averaged and included into a running average. This thread controls the PWM for the brightness of RGB pins on the LEDs. As the environment gets brighter, the photoresistor values decrease, and the PWM value increase. Higher PWM values leads to higher light intensity.
- LED Thread This thread is called on the Youtube app. The thread starts a counter for the LEDs and turns on an LED segment every 15 seconds starting with the bottom left LED. Once all LEDs are on, they start blinking for another 15 seconds before turning off. The code checks every 500 ms for a change in state (changed Youtube app or went to sleep mode) to turns off the LEDs and finish running the thread.
- Main Thread Initializes the program and runs an infinite while loop that handles when to start the LED thread. The LED thread is created everytime the user goes to the Youtube app, and terminates only when the user changes the Youtube app or puts the Smart Mirror to sleep.
- Communication between threads The threads communicate with each other via global variable sharing. All three button threads operate independently of the others, waiting for their button signal to change. When one button is pressed, it needs to send a message to the other button threads to ignore their button presses until its button press is released. It does this by changing the value of variable ledThreadRunning. When a button press is registered that makes the app go to the Youtube page, the currentApp variable changes value. This causes the isYoutube() method to return true, which causes another ledThread to be created by the main thread. Similarly, when the sleep command is registered while youtube is playing, the boolean isAsleep gets set to yes, causing the ledThread to quit and youtube to stop playing.
- Communication between c++ and web app The c++ code interacts with the web app by making system calls that emulate key presses. The web app is designed to respond to various key presses (listed in section 3.2.1). The c++ code emulates these key presses using a command line tool called xdotool. We wrote a script called key.sh that takes in a command line argument that represents the name of a key press, and the script emulates that key press using xdotool. To send a key press signal to the web app, the c++ code calls system(“sh key.sh key”), where key is the name of the key to emulate.
- Synchronization Issues. Our code contains a couple possible synchronization issues, but since they happen so infrequently (they actually have not occurred at all during our testing of the mirror), we decided to prioritize the rest of the smart mirror features. A few things we could have done are added 1) add locks to all the shared global variables, to prevent miscommunication, and 2) Create a locked global system command queue, in which we store all system calls. We then add a thread that periodically flushes the queue by executing the code. This would allow us to execute our system commands “asynchronously” in the sense that our other threads can always be responsive to user input, while maintaining correctness since all commands would execute in order.
A flexible, thin two-way mirror was bought and used. When the mirror covered the TV, both the reflection in the mirror and the content on the TV could be seen.
Four corner brackets were 3D modeled on SolidWorks and 3D printed at the Duke Colab to hold the TV in place. Additionally, a simple mount for the Raspberry Pi was designed, 3D printed and attached to the back of the frame.
Eight pieces of pine wood (four 2x4 and four 2x2) were bought from Home Depot. The wood was stained with a dark finish for visual appeal. The 2x2 wood pieces were then cut at 45 degree angles and milled using the CNC machine at the colab to make 8 inch valleys in the 2x2. These valleys were used to house the LED strips. To build the frame, the 2x4 pieces and the 3D printed corner brackets were glued together using wood glue. The frame was clamped in place overnight to assure a strong and accurate frame. Then, in order to fit the mirror, the 2x4s were cut to a 2x2 at the machine shop using the table saw. Then, the wood was screwed together for extra strength. This was the back of the frame. Next, the 3D brackets were filed down to fit the TV and the TV was placed in the frame. The mirror was then taped on top of the 2x4 frame and on top of the TV. Finally, the 2x2 pieces were attached on top of the mirror with wood glue. After solidifying the glue for a day, holes were drilled for the button, LED, and photoresistor wires. Then all wires from the PCB to the buttons/LEDs/photoresistors were soldered. Precise wire lengths were used so that there were not any loose wires on the back of the frame. Overall, the mirror and frame look great.
We achieved all of our goals with the smart mirror project. The mirror supports weather, youtube, and calendar screens with logging in/logging out capabilities, which were our main requirements and goals for the project. We have implemented the hardware we promised: buttons for user control, LEDs for a timer, and photoresistors for brightness control. Getting the display to work with the hardware was tricky because they were controlled/implemented in different environments (html/js and C++), but once we synchronized the buttons and LEDs with the website we were able to work on improving the design and the finishing touches. The LED timer for brushing teeth started as single LEDs, but we eventually switched to more luminous LED strips. The LED strips improved the design, but made the hardware much trickier due to the LED strip power requirements. We went above the initial requirements by adding word recognition to switch between pages. During the presentations, we liked the idea of controlling the mirror with our voice (instead of the buttons - we wouldn’t want to touch the buttons if our hands were occupied). Controlling the audio input proved to be quite difficult because of configuration errors with the raspberry pi, but the result was awesome. The smart mirror provides lots of functionality and looks great. For this project we didn’t have any set power or timing guidelines, but our goal was to make the product responsive and low power. Because we have so many components in our design it was tough to keep the device low power. However, we programmed the device to use less power when possible (LEDs off most of the time, sleep screen displays nothing). We also put the display on one webpage which sped up switching between pages. The website is very responsive which makes this truly a consumer product. We were happy that the mirror had many different features while still maintaining speed. This project was not time critical, but all of our functionalities fell within our desired latency (e.g. less than 1 second delay for LEDs switching on/off, button response accurate to 1 ms, etc).
The smart mirror is a fully functional consumer product. While there are small bugs occasionally, we are very pleased with the results. In a semester we improved the mirror, which has provided the same functions for over 8000 years. Showing the weather, a video and timer, and the calendar improves the mirror, simplifies our lives, and keeps us connected with daily weather forecasts and personal events and entertained with Youtube videos. The world is moving more and more towards connected devices. We want all our information accessible from anywhere. Instead of checking our phone in the morning, we can use the smart mirror to show us information. We want to use this mirror. So if we like using it, we believe there is a market for this product. We wanted to improve the capabilities of the mirror while maintaining the visual appeal of a mirror. The website design, LED regress, hidden photoresistors, and stained frame make the mirror look good, maintain its original reflective capabilities, and improve functionality. Powering through many development issues, we finally produced a working smart mirror and we are very happy with the final result. Now, we have to decide who gets to keep it.