READING FAÇADES: INTEGRATING HUMAN AND COMPUTER VISION
ARCH 5110: Architecture as Catalyst | March 9-13, 2015 | U. of Minnesota
Guest Instructor: Jentery Sayers, Assistant Professor, English and Social, Cultural, and Political Thought; Director, Maker Lab in the Humanities, University of Victoria
Faculty Instructor: Andrea J. Johnson, AIA, LEED BD+C, Assistant Professor, UMN School of Architecture
This Catalyst workshop explores the intersections of human and computer vision in the construction of three-dimensional space. How does the emergence of computer vision, or machine phenomenology, inform our interpretations of the built environment? How can the face or exterior of a building be detected, organized, and understood? Instead of approaching human and computer vision in a binary fashion, how might they be blended to ask questions about society, technology, and design?
In this workshop, we will combine image capture, computer programming, and physical computing techniques with object-detection frameworks in order to not only expand existing perceptions of built environments but also consider the relevance of computer vision to building facade design, archiving, and analysis. Here, the affordances of computer vision to systematically, superficially, and rapidly detect, stitch, and model 3D objects will prove informative. These affordances will be combined with critical studies of algorithms and computational culture. Students will participate in hands-on, introductory workshops on Python, Git, photogrammetry, and image processing. No previous experience in these areas will be assumed.
Foundational lectures to introduce topics; workshops for skill-building; studio sessions with project critiques; seminar discussion
- Approach computer vision as a technical and cultural matter, through a combination of theory and practice.
- Build 3D models with repositories of 2D images.
- Construct, describe, archive, and share image repositories using distributed version control.
- Consider the relevance of computer programming (e.g., in Python) to the representation and expression of 3D space.
- Experiment with computer vision across a spectrum of realist representation and speculative expression.
- Computer Vision: methods for acquiring, processing, analyzing, and understanding images
- Object Recognition: task of finding and identifying objects in an image or video sequence
- Photogrammetry: taking measurements from photos to determine locations of surface points
- Physical Computing: interactive physical systems using software and hardware that can sense and respond to the analog world
- Python Programming: an open-source, high-level, easy to learn programming language
- Git: a distributed revision control system for archiving and sharing data
- Software Studies: the examination of how software and algorithms are embedded in society and culture, often with an emphasis on the material composition of media
In addition to short readings, students will complete short exercises anchored in computer vision, programming, and 3D modeling. Class process and work will be posted by students throughout the week via GitHub. Each student will develop, create, and document a final project.
The readings, exercises, and seminar discussions will stress how work in computer vision operates on a spectrum, from realist representation (e.g., depicting the built environment as accurately as possible) to speculative expression (e.g., using computation to create things that do not exist in the world). Throughout the week, students will be encouraged to explore and test this spectrum. What does photogrammetry allow us to see that we may not otherwise? How can it help us model lived, social reality? How can it help us stitch together alternative realities, make curious media, and prototype counterfactuals? How can it be performed collaboratively or creatively?
During exhibition, students will be expected to share work that responds to these questions and more, through digital or tactile media. Through this work they will also be expected to assume a position (if you will) on the spectrum of representation and speculation.
Complete documentation of process and final project is required. All final files must be uploaded to GitHub by Monday, March 23. Minimum requirements:
- Summary of work according to provided template
- Images that fully document your final project (min. 10 images)
- Images that document the workshop and your process (min. 10 images)
- Working files used to create your project
Note: Save images as 72 ppi jpeg, min. 3600 pixels in one dimension, maximum quality 10 or above
Each student should have access to a computer. If possible (but not required), students should bring the following to meetings:
- A laptop (Windows, OSX, or Linux)
- A camera (a DSLR with an SD card, if possible)
Students are also encouraged to install the following on their machines:
- Agisoft PhotoScan 1.1.3, http://www.agisoft.com/downloads/installer/
- Git, http://git-scm.com/downloads
They should also create an account with https://github.com/, if they have not already.
During the week, they may be asked to work with additional software and languages, such as Rhino, SketchUp, and Python, in which case they will be given additional instruction.
- Monday 9:30-10:30: Introductions; Discussion of Syllabus and Schedule
- Monday, 10:30-11:30: Computer Vision as Culture + Technique
- Monday, 2-6: Gentle Introduction to Git, GitHub, and Markdown
- Tuesday, 9-12: Introduction to Photoscan
- Tuesday, 2-6: Experimenting with Photoscan (may include some Python programming)
- Wednesday, 9-12: Fieldwork with Various Cameras
- Wednesday, 2-6: Processing Image Sets
- Thursday, 9-12: Sharing and Discussing Results
- Thursday, 2-6: Preparing the Final Show
- Friday, 9-2: Prepping and Setting Up the Final Show
- George, https://github.com/KilluaHxH/Catalyst
- Chris, https://github.com/Christopheralph/Catalyst15
- Erica, https://github.com/schw1309/Catalyst
- Xian, https://github.com/huxxx832/
- Sam, https://github.com/daley045/Catalyst-2015
- Dustin, https://github.com/schip062/Catalyst
- Jeremy, https://github.com/jbernardy42/JBCatalyst2015
- Arnall, Timo. 2012. Robot Readable World. https://vimeo.com/36239715.
- Backes, Martin. 2010. “New Artwork: Pixelhead.” Martin Backes - Official Website. http://www.martinbackes.com/new-artwork-pixelhead/.
- Bagnall, Kate, and Tim Sherratt. 2011. “Invisible Australians: Living under the White Australia Policy.” http://invisibleaustralians.org/.
- Berry, David. 2012. “What Is the ‘New Aesthetic’?” Stunlaw. http://stunlaw.blogspot.ca/2012/04/what-is-new-aesthetic.html.
- Bridle, James. 2011. “The New Aesthetic: Waving at the Machines.” Booktwo. http://booktwo.org/notebook/waving-at-machines/.
- Browne, Simone. 2010. “Digital Epidermalization: Race, Identity and Biometrics.” Critical Sociology 36 (1): 131–50. doi:10.1177/0896920509347144.
- Calderara, Simone, Andrea Prati, and Rita Cucchiara. 2009. “Video Surveillance and Multimedia Forensics: An Application to Trajectory Analysis.” In Proceedings of the First ACM Workshop on Multimedia in Forensics, 13–18. MiFor ’09. New York, NY, USA: ACM. doi:10.1145/1631081.1631085.
- Chun, Wendy Hui Kyong. 2011. Programmed Visions: Software and Memory. MIT Press.
- Columbia University. 2010. “Pubfig: Public Figures Face Database.” http://www.cs.columbia.edu/CAVE/databases/pubfig/.
- Drucker, Johanna, and Bethany Nowviskie. 2004. “Speculative Computing.” In Companion to Digital Humanities (Blackwell Companions to Literature and Culture), edited by Susan Schreibman, Ray Siemens, and John Unsworth, Hardcover. Blackwell Companions to Literature and Culture. Oxford: Blackwell Publishing Professional. http://www.digitalhumanities.org/companion/.
- Eden, Terence. 2014. “Tate Hack.” GitHub. https://github.com/edent/Tate-Hack.
- Elliott, Devon, Robert MacDougall, and William J. Turkel. 2012. “New Old Things: Fabrication, Physical Computing, and Experiment in Historical Practice.” Canadian Journal of Communication 37 (1). http://www.cjc-online.ca/index.php/journal/article/view/2506.
- Gibson, William. 2010. "Google's Earth." New York Times. http://www.nytimes.com/2010/09/01/opinion/01gibson.html?_r=0.
- Harvey, Adam. 2013. “Stealth Wear.” AH Projects. http://ahprojects.com/projects/stealth-wear/.
- Jones, Matt. 2011. “Sensor-Vernacular.” BERG. http://berglondon.com/blog/2011/05/13/sensor-vernacular/.
- Manovich, Lev. 2009. “Cultural Analytics.” Software Studies. http://lab.softwarestudies.com/p/cultural-analytics.html.
- Minsky, Marvin. 1974. “A Framework for Representing Knowledge.” https://web.media.mit.edu/~minsky/papers/Frames/frames.html.
- Papert, Seymour. 1966. “The Summer Vision Project,” July. http://dspace.mit.edu/handle/1721.1/6125.
- Resig, John. 2015. “Using Computer Vision to Increase the Research Potential of Photo Archives.” John Resig. http://ejohn.org/research/computer-vision-photo-archives/.
- Sayers, Jentery. 2014. “The Relevance of Remaking.” The Maker Lab in the Humanities. http://maker.uvic.ca/remaking/.
- Sterling, Bruce. 2005. Shaping Things. MIT Press.
- ———. 2009. “Design Fiction.” Interactions 16 (3): 20–24.doi:10.1145/1516016.1516021.
- ———. 2012. “An Essay on the New Aesthetic.” Wired. April 2. http://www.wired.com/2012/04/an-essay-on-the-new-aesthetic/.
- Szeliski, Richard. 2010. Computer Vision: Algorithms and Applications. Springer Science & Business Media.