Shadow Remix

Live demo at: apps.wonderfulidea.co/shadow-remix/app/

• Overview
  • Shadow Remix App
  • Online Gallery
  • Gallery Admin Interface
  • Arduino
  • Repo Structure
• Use
• License
• Hotkeys
• Limitations
• Parts List
• Schematics and PCB
• Backend Setup
• Serial Communication
• Code
  • Install nodejs and npm
  • Compiling UI Library
  • Compiling Desktop App
  • Adding Screensaver Animations
  • Compiling Gallery

Overview

This project consists of a few pieces of code that interact with each other:

Shadow Remix App

The Shadow Remix drawing interface is available both as javascript code that runs in a web browser or as a compiled desktop app for Windows and Mac. A live demo of the code is hosted at apps.wonderfulidea.co/shadow-remix/app/.

The browser-based drawing interface does not have the ability to upload photos to our online gallery due to security issues (we don't want to let anyone on the internet upload photos to the gallery), but it does let users save images to their computers.

You will need to compile your own copy of the desktop Shadow Remix drawing interface with credentials for editing your database.

Online Gallery

The gallery is a separate web app that displays approved photos drawn by visitors at the museum exhibition. It has a feature where any new (unapproved) drawing made in the museum is shown for a brief period and then hidden until it is approved by an admin.

The gallery is currently hosted at apps.wonderfulidea.co/shadow-remix/gallery/.

Gallery Admin Interface

The gallery admin interface is a private web app that displays saved photos drawn by visitors at the museum exhibition and allows an admin to approve them for public viewing. You will need to compile and host your own copy of the admin interface with credentials for editing your database.

Arduino

The museum exhibit features a motorized shadow-making setup and a panel of button controls for easier interaction. All the electronics for this exhibit are controlled by an Arduino which communicates via USB to the desktop app.

Repo Structure

This repo consists of several subdirectories:

• app folder contains both a standalone browser-based application for shadow remix and the UI library code for connecting to a webcam, drawing to the screen, and saving images.
• electron folder has code for compiling the desktop app with Electron, it depends on the UI codebase located within the app folder.
• gallery folder has code for a publicly accessible gallery webpage showing images uploaded by users.
• gallery-admin folder has code for a private admin interface for approving images.
• arduino folder contains arduino code for connecting hardware buttons to the app.

Use

You need to compile your own Windows (64 and 32 bit) and Mac installers following the instructions for compiling the desktop app. Once built, you should be able to install the app on any desktop setup. These app is not signed and may require some additional steps to open:

• On Mac, hold down the control key and right click on the application icon, then select Open. Confirm that you would like to open the application by an unidentified developer. You should only need to do this the first time you run the application. More info here.

The first time you open the app, it will ask permission to access the webcam in a pop-up dialog.

License

This work is distributed under an MIT license with exception to the "Lunch" font by Anthem Type. See Lunch Font for more details about licensing.

Hotkeys

This app is primarily intended for use in a museum environment via a touchscreen and peripheral button controls. If you have access to a keyboard or MakeyMakey, you can also use the following hotkeys:

• ctrl/command + z OR d to undo
• ctrl/command + s OR a to save image
• backspace/delete OR g to clear
• c brings up a dialog for selecting the webcam device
• p brings up a dialog for selecting the serial port (desktop app only)
• v saves your current drawing as an animated video (webm format). Webm videos are large and difficult to play, but very fast to save – which is why we use them. You can also use video editing software like Adobe Premiere to convert the video format. Once you have the mp4s, you can add them to the desktop app.
• h puts the app into screen saver mode (desktop app only).

Limitations

• Currently, this app expects to receive webcam feeds in landscape orientation and renders out at the resolution specified in app/src/constants.ts. I've noticed that when mobile devices are in portrait mode, the returned width and height from getUserMedia() look like the stream is still in landscape. I have not found a consistent way to correct this yet.

Parts List

See docs/Parts_List.md for a list of hardware we used for our exhibit.

Schematics and PCB

See docs/PCB.md for schematics and info about making circuit boards for the exhibit.

Backend Setup

This app uses an AWS S3 bucket to store saved images in the cloud. The saved images can be viewed through a webpage at apps.wonderfulidea.co/shadow-remix/gallery/. Info about configuring the AWS Backend can be found at docs/Backend_Setup.md.

Serial Communication

The serial communication protocol between the desktop app and Arduino is described in docs/Serial_Communication.md. All arduino code can be found in the arduino folder.

Code

Install nodejs and npm

Before you compile any part of the code, you will need to install nodejs and npm on your machine. All compilation steps are done through the command line.

Compiling UI Library

This repo comes with a pre-built version of the UI library, located at app/dist/ui.js. This library should be fine as is for most cases. If you do not plan to modify the UI code, skip to the next section.

If you want to change the available colors, line thicknesses, and other aspects of the UI, you can make those edits within app/src/constants.ts.

If you want to change the hotkeys, you can make those edits within app/src/hotkeys.ts.

After make changes to the UI code you need to compile the app/src/ folder to app/dist/ using the following commands:

First, install nodejs and npm.

In the terminal, navigate to the app directory:

cd app

Install all dependencies:

npm install

Compile with Webpack:

npm run build

You can also start a development server at localhost:8080/ using the following command:

npm run start

Compiling Desktop App

The javascript code in this repo is compiled with Electron so that it can be run as a desktop app (rather than in a web browser).

You will need to compile your own desktop app after following the steps to set up an AWS bucket and permissions and create an .env file with your access keys. If make changes to the UI library in the app directory, you will need to compile it first before compiling a new copy of the desktop app.

If you want to change things about the app's functionality (e.g. wait time before showing the screensaver, or which buttons map to which functions) you can make those edits within electron/src/constants.ts.

To compile for Windows and Mac:

First, install nodejs and npm.

In the terminal, navigate to the electron directory:

cd electron

Install all dependencies:

npm install

To compile for Windows and Mac:

npm run build

Once complete, builds are located in the electron/build folder. The shadow-remix Setup #.#.#.exe installer should work on 32bit and 64 bit Windows machines, and the shadow-remix-#.#.#.dmg or shadow-remix-#.#.#-mac.zip files contain builds for Mac.

If you would like to compile for Linux, edit the following line in electron/package.json:

"build": "npm run compile && electron-builder -mw",

change to:

"build": "npm run compile && electron-builder -mwl",

Then run:

npm run build

Linux build has not been tested and may require additional configs, see the electron-builder docs for more information.

If you want to make changes to the app's code, you can quickly test them by running a local development server:

npm run start

This will open an app window and begin running the new code automatically.

Adding Screensaver Animations

The desktop app features a screensaver mode where you can show animations of previously drawn shadow-remixes if the station has remained inactive for a period of time. To create the videos, draw a shadow remix and then hit the v key when you are done. This will step through your entire drawing sequence and save out a video (it may be a little slow). Using a video editor, convert the webm videos that are saved to your computer into mp4.

Next, you need to create a folder at electron/animations/ and add several videos to this folder (with format mp4). After making these changes you will need to recompile the app. Alternatively, if you would like to change these videos (or add new ones) without recompiling the app, you can edit the contents of the Resources/animations/ folder in the app's build and restart the app (on Mac, right click on the app's icon and select "Show Package Contents", on Windows right click on the app's icon and select "More > Open file location"). To be safe, use filenames without spaces for any animations you add to the app.

Compiling Gallery

This repo comes with a pre-built version of our public gallery, located at gallery/dist/ and currently hosted at XXXX.

In order to view your own images, you will need to recompile the gallery after setting up your AWS bucket and permissions and creating a .env file with your access keys.

You can also edit some parameters of the gallery (e.g. how long images stay up, how often to refresh) at the top of gallery/src/index.ts.

Once complete, compile the gallery/src/ folder to gallery/dist/ using the following commands:

First, install nodejs and npm.

In the terminal, navigate to the gallery directory:

cd gallery

Install all dependencies:

npm install

Compile with Webpack:

npm run build

You can also start a development server at localhost:8080/ using the following command:

npm run start

Note that you will also need to add http://localhost:8080 to you list of Allowed Origins in the AWS backend setup.

Follow these same instructions to compile the gallery admin interface (located in the gallery-admin folder). Once compiled, the files inside gallery-admin/dist/ can be run locally or hosted behind a password protected server to allow admins to approve new image that are saved from the app. Do not post the compiled gallery-admin interface publicly because it contains your AWS secrete keys!! By default, all files in gallery-admin/dist (with the exception of index.html) should be ignored from git.

You can edit some parameters of the gallery admin interface (e.g. how often to refresh) at the top of gallery-admin/src/index.ts.