PIG: "Allow Me Into Your Dream" — from Designing a 'Secret Handshake' Gestural Protocol for Sharing Permission-less Mixed Realities, to Speculating Experiential Network States
- Botao Amber Hu from Reality Design Lab
- You?
Note inviting collaborators
As a spatial computing researcher and mixed reality designer with a computer science background, I'm looking for a second team member. Ideally, this member would be interested in speculative design, skilled in essay writing, and familiar with digital humanities to contribute to this project. If you are interested, please send me email with links to your cv, portfolio and writing samples.
Email me: future@reality.design
Apple's release of Vision Pro indicates the beginning of the spatial computing era. The increasing prominence of Mixed Reality (MR) head-mounted displays (HMDs) suggests a future where MR HMDs are as ubiquitous as smartphones. In this protocol research works, we propose three interconnected studies in three chapters. These will explore how human society, spatial computing, and programmable cryptography intertwine to shape our future reality.
Hyper-Reality (2016) by Keiichi Matsuda
In this chapter, we aim to predict the near-term future of the spatial computing era and develop a media theory called "Permissionless Realities" that reflects the inherent nature of mixed reality media. This concept aligns with the permissionless characteristic of blockchain. We envision a future where reality becomes asymmetric, layered with numerous entangled mixed realities, all without needing anyone's permission. Supporters of the blockchain world believe that "protocol is permissionless". From this perspective, based on our literature review, mixed reality media has not yet been theorized from such a protocolized, permissionless angle. This theory comes from the observations during my previous CHI work, MOFA, which is a magic dueling AR game in the public street. So in this under-researched theory, we plan to explore further and summarize our findings in an academic paper to share our new media theory released in Digital Art, or SIGGRAPH community.
Multiplayer Omnipresent Fighting Arena, i.e. MOFA.ar (2023) by Botao Amber Hu
Chapter 1's media theory posits that "reality is asymmetric". Typically, users of Mixed Reality (MR) Head-Mounted Displays (HMDs) use Augmented Reality (AR) apps privately to perceive their own version of reality without requiring others' approval. But what happens when two HMD users meet in person? How do they share their mixed reality? Inspired by the "Secret Handshake" from gangster culture, we propose an embodied protocol called "FingerSync". This protocol allows two users to synchronize and share their mixed realities simply by touching fingertips, akin to the famous Sistine Touch, thus creating layered mixed realities (for example, saying "Wizarding World"). Yes, I know. It's truly like unbelievable magic, but we've indeed created a prototype with two Vision Pros. The final deliverable of this chapter will be an academic paper released in SIGCHI community.
The famous "Sistine Touch" is part of "The Creation of Adam" (1512) by Michelangelo.
FingerSync is inspired by the secret handshake culture.
How can layered mixed realities persist without permission, even when there are no HMD wearers to perceive them? Can they evolve and operate autonomously? This is much like nature, which, for example, does not need permission for a tree or an animal to grow.
With the rise of Programmable Cryptography (ProgCrypto) technologies like zero-knowledge Machine Learning (zkML) and Decentralized Physical Infrastructure Networks (DePIN), we foresee blockchain evolving into a powerful "planetary-scale computation megastructure." that term coined by philosopher Benjamin H. Bratton. This structure can permissionlessly and autonomously execute complex AI tasks, such as Digital Physics, as discussed in Dante’s talk at the AW Assembly. Following this vision, we propose a meta-protocol called "Autonomous Realities" for persisting the mixed reality layer. This conceptual protocol could underpin many protocols for future mixed realities. Firstly, it relies on (1) “permissionless autonomy,” using the existing Autonomous Worlds infrastructure to persist the rules, digital physics, and virtual objects of mixed reality layers. (2) it implements a kind of "permissionless permission," building on the work in Chapter 2, "sharing mixed reality by permitting others." This protocol integrates with Zupass, a zero-knowledge-proof-based digital passport for network state citizens. Thus, only those who have access to this Autonomous Reality layer can see its virtual objects, like buildings, pets, and items within the realm of the network state. This protocol could potentially evolve into what we've termed an "Experiential Network State".
This chapter is derived from my hackathon work “Zuzaland” in Montenegro, where I built the first monument ZuzuluEmblem as the first public virtual building/object for Zuzalu, a first-of-its-kind pop-up network state experiment. The final deliverable of this chapter will be an essay paper circulated within the Programmable Crypto or, more specifically, the Autonomous Worlds Network community.
https://github.com/realitydeslab/SoP2024/assets/2534431/8cb760a1-df62-4e40-befb-912c3d9f4f54 Zuzaland (2023) by Botao Amber Hu. ZuzuluEmblem is a virtual monument located in the pop-up city. In reality, it's located in a hotel bay situated in front of the residences of all co-living citizens in Montenegro. The monument symbolizes key topics of Zuzulu, revolving around its logo in text. Topics like "longevity", "network state", and "zero-knowledge proof" are all represented, as these are subjects that citizens care about.
https://github.com/realitydeslab/SoP2024/assets/2534431/e8fc18ea-ae1c-48e4-aa60-97127da97b8c Drifting Whispers (2023) by Botao Amber Hu and Fangting. The virtual monument is situated at the co-living space of ZuConnect, Istanbul during DevConnect conference. Every flying star symbolizes a participant in this co-living event.
We envision a future in the era of spatial computing where MR HMD becomes more prevalent. This project involves protocol research and design work, based on four existing protocols.
-
Secret Handshake or 'Gang' Handshake
A "Secret Handshake" is a private, prearranged gesture or series of gestures used by individuals to identify membership in a group or to grant mutual recognition.
-
Session-Sharing Protocol and Technology for Collocated Mixed Reality
- Local networking - Apple MultipeerConnectivity:
- Coordinate Registration - Collaborative simultaneous localization and mapping (ColSLAM)
- Game Object Synchronization - Netcode for GameObjects
-
Zupass, a Digital Passport based on Zero-knowledge proof
Zuzalu's digital passport, similar to Apple's Wallet, houses Zuzalu Passport Cards to verify the holder's identity. This system is specifically designed for the storage and management of Proof-Carrying Data, which is data whose authenticity and structure can be cryptographically verified.
-
Autonomous Worlds
It's a meta-protocol that treats the blockchain as a autonomous substrate, similar to nature. This substrate runs its own digital physics with composability.
We remix these protocols to innovate theory, interaction, and protocol, speculating on a future vision of how human society, spatial computing, and programmable cryptography might interweave. The philosophical thinking behind this can be encapsulated by the famous quote:
“A dream you dream alone is only a dream. A dream you dream together is reality.” ― Yoko Ono
Chapter 1. Permissionless Realities
We introduce a new media theory, "Permissionless Realities," which offers fresh perspectives on understanding the nature of mixed reality in the era of spatial computing.
Chapter 2. FingerSync
In envisioning a future where MR HMDs are prevalent, we're curious in moment how two wearers can share their mixed realities. We've invented a new embodied protocol, "FingerSync," to seamlessly and almost magically establish sessions for sharing collocated mixed reality.
Chapter 3. Autonomous Realities.
We have proposed a new meta-protocol, "Autonomous Realities", which is based on the concept of "Autonomous Worlds". This envisions permissionless realities independent of users, thereby creating layers of new 'nature'.
Chapter 1. Permissionless Realities
- Literature Review.
- Write an peer-reviewed paper.
Chapter 2. FingerSync
- Research Through Design with N=30 user studies.
- Write an peer-reviewed paper.
Chapter 3. Autonomous Realities
- Formative Studies with Expert Interviews
- Compose an essay to receive feedback from the AW community.
In what form will you prototype your improvement idea? Eg: Code, reference design implementation, draft proposal shared with experts for feedback, A/B test of ideas with a test audience, prototype hardware, etc.
Chapter 1. Permissionless Realities
Write the theory paper.
Chapter 2. FingerSync
We plan to prototype the protocol on multiple Vision Pros and make the code open source.
Chapter 3. Autonomous Realities
We then create a prototype based on Zupass and the MUD engine.
How will you field-test your improvement idea? Eg: run a restricted pilot at an event, simulation, workshop, etc.
Chapter 1. Permissionless Realities
Plan to host an academic workshop that explores the permissionless aspect of mixed reality media. This event will engage expert communities from the fields of spatial computing, human-computer interaction, and media theory. Possible venues for this workshop could be academic conferences like CSCW, CHI, and SIGGRAPH.
Chapter 2. FingerSync
We use a Research Through Design approach to explore various augmented "secret handshakes" as our protocol for spontaneous sharing of mixed reality. This involves iterating and testing across 20-30 user groups, along with conducting interviews and tests with mixed reality experience designers.
Chapter 3. Autonomous Realities
We host an academic workshop titled "Autonomous Realities" within the Autonomous Worlds Community, using Formative Studies.
Chapter 1. Permissionless Realities
We are experts from the media art and media theory community. Our intention is to compile our discoveries into an academic paper. This paper will describe our new media theory. We plan to submit it for peer review at relevant academic conferences, such as SIGGRAPH art papers.
Chapter 2. FingerSync
The experts from the Spatial Computing and HCI community. We plan to submit the paper to be peer-reviewed in related academic conferences such as SIGCHI and ISMAR.
Chapter 3. Autonomous Realities
Experts from the Autonomous Worlds community, or zkML community.
Chapter 1. Permissionless Realities
- Publish a paper at the SIGGRAPH conference and in media art related events.
Chapter 2. FingerSync
- The open-source code related to our paper is attached on GitHub.
- Publish a paper at a conference related to SIGCHI.
- Organize a workshop at universities for students with a background in interaction design. The aim is to encourage them to use our open-source code as a base for creating multiplayer mixed reality vision pro apps.
Chapter 3. Autonomous Realities
- Publish an essay in the community of Autonomous Worlds.
- Publish an essay in the community of Network State commnuity.
Chapter 1. Permissionless Realities
- Successfully accepted by peer-reviewed publications.
- The theory has the potential to impact the paradigm of future mixed reality.
Chapter 2. FingerSync
- The open-source code associated with our paper can be utilized by many other mixed reality applications on Vision Pro.
- Successfully accepted by peer-reviewed publications.
- The theory has the potential to impact the paradigm of future mixed reality.
Chapter 3. Autonomous Realities
- Publish an essay in the community of Autonomous Worlds.
- Publish an essay in the community of Network State commnuity.