AI Crew Control: Super Heavy Catch Simulation What is this? This is an interactive learning simulation that demonstrates how SpaceX’s Mechzilla system is able to catch Falcon Super Heavy rockets - aka “catching a rocket with chopsticks.”

The user is able to talk to AI crew members to understand and overcome the logical and physical interactions, as well as react to real-time emergencies. The goal is to learn the complex physics and decision-making behind the catch through a natural, voice-first conversation.

Demo link: https://youtu.be/euDeW2U1pnA?si=ZPZw9Ct-o31g5lgH

How we used Gemini and Pipecat

• Gemini: Acted as the reasoning engine for our AI crew. Gemini interprets the user's urgent, natural language commands during emergencies (e.g., "Fire the cold gas thrusters!"). It understands the intent behind the commands and provides the AI crew with relevant, context-aware responses, turning a simple script into a dynamic problem-solving session.
• Pipecat: Served as the core of our real-time voice interface. We used it for low-latency Speech-to-Text (STT) and Text-to-Speech (TTS), enabling a fluid, natural conversation between the Commander and the AI crew without noticeable delay.

Other tools we used

• we used Gemini code, 2.5 pro, and Pipecat

What we did new during the hackathon This project was conceived and built entirely from scratch during the hackathon. All components are new:

• We built an entire physics engine for simulating a rocket takeoff, landing, and controls from scratch in 8 hours.
• Pathfinding machine learning algorithm for the rocket to land and be caught by the Mechzilla system.
• The development of the AI crew personalities and their conversational logic using Gemini.
• The integration of Pipecat for real-time voice command and response.
• The simple simulation UI and the underlying state machine.

Feedback on tools

• Pipecat: The low latency for STT/TTS was fantastic for building a voice-first experience feel natural and responsive. T
• Gemini: Gemini is simply an outstanding all-purpose model. We used Gemini code and Gemini build for some of the The model's ability to quickly grasp the context of a high-stakes, technical situation and understand urgent, informal commands is impressive. It allowed us to move beyond simple "A/B/C" choices into more dynamic, natural language interaction