Workshop: Reading and Writing Inhabited Space
A workshop and demo exploring artistic applications of smart home technologies. Topics include wireless sensing, computer vision, and machine listening to narrate inhabited space. We consider the creative possibilities and personal consequences of life with ubiquitous sensing, perceiving machines.
- Instructor: Robert Twomey (website)
- Date: Friday Nov 4, 2017
- Talk: 10-11am
- Workshop: 11:30-2pm
- Location: CMU STUDIO for Creative Inquiry
- Audionode demo:
- Machine Listener, "voice" triggered recording with webrtcVAD
- vadnode.py on pi0.local
- Videonode demo:
- Machine observer, motion triggered recording with pikrellcam
- pikrellcam on studio.local
- Physical Sensors
- particle photon
- Audio analysis:
- feature analysis (SCMIR or librosa)
- clustering (k-Means, HDBSCAN)
- Google TTS, Kaldi, CMU Sphinx housemachine/data/speech/google
- Video analysis:
- Background subtraction (KNN, MOG, etc) with OpenCV housemachine/data/cv/clips
- inhabitance maps housemachine/data/cv/json
- ROIs (future directions, f.ex. with moose)
- Sensor analysis
- cleaning data housemachine/data/time/sensor_times_clean.png
- meaningful filtering, clustering, etc (future directions)
- Representational strategies
- audio synthesis housemachine/supercollider/audio_sandbox/tsne_scmir_GUI_previewer.scd
- video synthesis housemachine/data/one_day/
- sensor narrative
- speech transcripts
- robotic re-embodiment
- Technologies used
- Compiling and uploading to the photon
- Communicating with the photon
- Concept and applications
- Going further
- Physical Sensing: Particle Photon with various sensors (PIR, IR rangefinder, beam break, reed switch, DHT, piezo, force sensor)
- Machine Listeners: Raspberry Pi with MEMS microphone, (alternative from adafruit) Python and webrtcvad.
- Machine Watchers: Raspberry Pi with Fisheye Camera and pikrellcam
- Server software in Python, Supercollider, p5, Processing
- Sensor software in C++/Wiring/Particle HAL
- Data analysis in Python with OpenCV and librosa, Supercollider with SCMIR.
- Server: port 9999, address 192.168.1.20, broadcast addr 192.168.1.255
- broadcast addresses are used to announce the nodes
- Sensor nodes: port 8888, addresses vary on the 192.168.1.xxx
- Local WiFi network: ssid housemachine, pwd 2029973952
- Messaging: OSC
Install the following software if you do not already have it:
- SuperCollider: http://supercollider.github.io/
- Particle Photon CLI: https://docs.particle.io/guide/getting-started/connect/core/
- A text editor like Sublime or Atom. I will use sublime for this workshop.
Download the source code for the Particle Photon
In terminal, clone the particle software:
git clone https://github.com/spark/firmware
NOTE: all further paths are relative to wherever you downloaded the particle source.
Compile and Upload to the Photon
In order to update the firmware and upload programs, you need to place the particle in Device Firmware Upgrade (DFU) mode. Follow these setups:
- Hold down BOTH buttons
- Release only the RESET button, while holding down the SETUP button.
- Wait for the LED to start flashing yellow (it will flash magenta first)
- Release the SETUP button
The indicator ight should be flashing yellow. This means it is ready to upload new programs.
Compile and upload firmware
- Change directory to the particle firmware modules:
Place the particle in DFU mode (see above)
Build the firmware:
make all PLATFORM=photon -s program-dfu
When done building the firmware, it will upload to the particle. It will do two separate uploads. You will see it cycle back to blinking yellow DFU mode twice.
Compile and upload your program:
Write your program or use an existing one (f.ex. particle/helloosc)
Link the folder with your program to the firmware/user/applications/ directory:
ln -s path/to/housemachine-workshop/particle/helloosc firmware/user/applications/
- From the particle firmware directory, change to main:
- Place the particle in DFU mode (see above)
- Compile and upload your application (f.ex., to compile helloosc):
make PLATFORM=photon APP=helloosc program-dfu
Communicating with the Photon
Connect to the local WiFi network
Select the housemachine network:
- ssid: 'housemachine'
- pwd: 2029973952
Run the server program
The server program in either supercollider, p5, processing or python will listen for broadcast messages on the local network. When a particle turns on it will announce it's presence.
We will start with the supercollider program, supercollider/photon_server.scd
Detect the Photon when it goes live
Power on the photon by plugging in an AC adapter. It will go through it's startup routine. When it finally connects to the network, it should show a slow green pulse.
Did your server program see the announce message? If so, what is the IP address and name for the photon?f
Renaming your photon
Set a new name for your photon. Either modify the code of helloosc.cpp, or switch to the housenode program and configure with messages from supercollider.
Read Sensor Data from Photon
Data logging Processing data
Send Commands to Photon
Actuators, physical outputs. F.ex. control a relay.
Physically staging the photon
How do we attach our sensors to things?
Concept and Applications
How can we use these wireless sensors?
Brainstorm on uses of wireless sensor technology
- What would you monitor? What physical gestures / operations?
- What would you control? What actuators?
- What modalities? (voice, vision, sensors)