A workshop exploring artistic applications of smart home technologies
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
p5/browser-speechrec
particle
python
supercollider
.gitignore
README.html
README.md

README.md

Workshop: Reading and Writing Inhabited Space

A workshop and demo exploring artistic applications of smart home technologies. Topics include wireless sensing, computer vision, and machine listening to narrate inhabited space. We consider the creative possibilities and personal consequences of life with ubiquitous sensing, perceiving machines.

Details:

Demos

  1. Audionode demo:
    • Machine Listener, "voice" triggered recording with webrtcVAD
    • vadnode.py on pi0.local
  2. Videonode demo:
    • Machine observer, motion triggered recording with pikrellcam
    • pikrellcam on studio.local
  3. Physical Sensors
    • particle photon
  4. Audio analysis:
    • feature analysis (SCMIR or librosa)
    • clustering (k-Means, HDBSCAN)
    • visualization
    • housemachine/data/audioanalysis/hours
    • Google TTS, Kaldi, CMU Sphinx housemachine/data/speech/google
  5. Video analysis:
    • Background subtraction (KNN, MOG, etc) with OpenCV housemachine/data/cv/clips
    • inhabitance maps housemachine/data/cv/json
    • ROIs (future directions, f.ex. with moose)
  6. Sensor analysis
    • cleaning data housemachine/data/time/sensor_times_clean.png
    • meaningful filtering, clustering, etc (future directions)
  7. Representational strategies
    • audio synthesis housemachine/supercollider/audio_sandbox/tsne_scmir_GUI_previewer.scd
    • video synthesis housemachine/data/one_day/
    • sensor narrative
    • speech transcripts
    • drawings
    • sculpture/installation
    • robotic re-embodiment

Workshop

  1. Technologies used
  2. Setup
  3. Compiling and uploading to the photon
  4. Communicating with the photon
  5. Concept and applications
  6. Going further

Technologies used

Hardware

Software

  • Server software in Python, Supercollider, p5, Processing
  • Sensor software in C++/Wiring/Particle HAL
  • Data analysis in Python with OpenCV and librosa, Supercollider with SCMIR.

Communication

  • Server: port 9999, address 192.168.1.20, broadcast addr 192.168.1.255
    • broadcast addresses are used to announce the nodes
  • Sensor nodes: port 8888, addresses vary on the 192.168.1.xxx
  • Local WiFi network: ssid housemachine, pwd 2029973952
  • Messaging: OSC

Setup

Software

Install the following software if you do not already have it:

Download the source code for the Particle Photon

In terminal, clone the particle software:

git clone https://github.com/spark/firmware

NOTE: all further paths are relative to wherever you downloaded the particle source.

Compile and Upload to the Photon

DFU mode

In order to update the firmware and upload programs, you need to place the particle in Device Firmware Upgrade (DFU) mode. Follow these setups:

  1. Hold down BOTH buttons
  2. Release only the RESET button, while holding down the SETUP button.
  3. Wait for the LED to start flashing yellow (it will flash magenta first)
  4. Release the SETUP button

The indicator ight should be flashing yellow. This means it is ready to upload new programs.

Compile and upload firmware

  1. Change directory to the particle firmware modules:

cd firmware/modules

  1. Place the particle in DFU mode (see above)

  2. Build the firmware:

make all PLATFORM=photon -s program-dfu

When done building the firmware, it will upload to the particle. It will do two separate uploads. You will see it cycle back to blinking yellow DFU mode twice.

Compile and upload your program:

  1. Write your program or use an existing one (f.ex. particle/helloosc)

  2. Link the folder with your program to the firmware/user/applications/ directory:

ln -s path/to/housemachine-workshop/particle/helloosc firmware/user/applications/

  1. From the particle firmware directory, change to main:

cd firmware/main

  1. Place the particle in DFU mode (see above)
  2. Compile and upload your application (f.ex., to compile helloosc):

make PLATFORM=photon APP=helloosc program-dfu

Communicating with the Photon

Connect to the local WiFi network

Select the housemachine network:

  • ssid: 'housemachine'
  • pwd: 2029973952

Run the server program

The server program in either supercollider, p5, processing or python will listen for broadcast messages on the local network. When a particle turns on it will announce it's presence.

We will start with the supercollider program, supercollider/photon_server.scd

Detect the Photon when it goes live

  1. Power on the photon by plugging in an AC adapter. It will go through it's startup routine. When it finally connects to the network, it should show a slow green pulse.

  2. Did your server program see the announce message? If so, what is the IP address and name for the photon?f

Renaming your photon

Set a new name for your photon. Either modify the code of helloosc.cpp, or switch to the housenode program and configure with messages from supercollider.

Read Sensor Data from Photon

Data logging Processing data

Send Commands to Photon

Actuators, physical outputs. F.ex. control a relay.

Physically staging the photon

How do we attach our sensors to things?

Concept and Applications

How can we use these wireless sensors?

Brainstorm on uses of wireless sensor technology

  • What would you monitor? What physical gestures / operations?
  • What would you control? What actuators?
  • What modalities? (voice, vision, sensors)

Try it

Going Further