A closed-loop robotic drawing installation where a machine watches you scribble and responds with quiet, mechanical order.
This is Phase 1 — a one-directional response system where the robot normalizes human gesture. The next phase will transform it into a co-creative companion: an agent that reads your latest doodle, interprets it, and suggests the next mark — turning the encounter from correction into shared visual storytelling. See What's Next below.
Doodle Mind is a closed-loop doodling installation, but it begins as a game. You step in without a clear instruction sheet, and the "rules" only become legible once you start drawing. You scribble in red — quick, unplanned, almost pre-verbal — and the robot responds in black. The color split makes the encounter readable on the page: red as human drift and impulse, black as machine order.
At first you don't really know what it's doing; you test it, vary your marks, try to figure out the logic. Slowly, a pattern emerges: the system isn't trying to understand your doodle or complete it "correctly." It normalizes it. For each gesture, the robot places a single short straight "stamp" near your mark — smooth, calm, and slightly indifferent in placement — compressing messy, multi-directional expression into approved geometry: a small set of valid angles, one line, one decision.
It's not erasure and not an argument; it's gentle discipline — almost polite — quietly replacing complexity with order.
In an era where AI increasingly mirrors us and finishes our marks for us, Doodle Mind holds a different moment: watching normalization itself, and noticing how it reshapes the gesture — and maybe the person making it.
Left: the physical setup from another angle. Right: the OpenCV interface detecting red scribbles and showing stamp overlays.
Presenting Doodle Mind at Bezalel Academy of Arts and Design, M.Des Industrial Design.
┌──────────┐ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌───────────┐
│ Camera │───▶│ HSV red-ink │───▶│ Centroid + │───▶│ Quantized │───▶│ Robot │
│ (live) │ │ detection │ │ direction │ │ angle stamp │ │ draws │
└──────────┘ └─────────────┘ └──────────────┘ └──────────────┘ └───────────┘
- A camera watches the paper from above in real time
- OpenCV detects red ink using HSV color filtering and morphological cleanup
- For each scribble: compute centroid, dominant direction via PCA, and snap to a quantized angle
- The arm travels smoothly to the scribble location, lowers its pen, draws one straight stamp, lifts, and parks
- Press SPACE for the next scribble — the robot skips areas it has already addressed
Phase 1 establishes the hardware platform, the camera-to-arm pipeline, and the core interaction loop. But the robot's response — a rigid, normalizing stamp — is deliberately limited. The next phase shifts the system from correction to co-creation:
- The robot as a drawing companion — instead of stamping a straight line, the agent interprets the latest doodle and proposes its own next mark. It plays with you, not against you.
- LLM-driven visual reasoning — using a vision-language model to read the evolving drawing, suggest compositional moves, and drive the arm toward unexpected aesthetic territory.
- Bidirectional dialogue — the drawing becomes a conversation: human gesture → machine interpretation → human response → machine counter-move.
A closely related direction was explored by artist Patrick Tresset in his Companion project (EACVA, 2025), which integrates a drawing robot with LLMs for co-creative sketching through speech and drawing. Doodle Mind's next phase shares this ambition — transforming the robot from a passive executor into a playful co-creative partner — while grounding it in the specific language of gesture, normalization, and the material encounter between pen and paper.
| Component | Details |
|---|---|
| Computer | MacBook Pro (Apple Silicon) running macOS |
| Camera | USB camera (tested with Intel RealSense F200 RGB) |
| Controller | ESP32 Dev Module (via USB serial at 115200 baud) |
| Servo Driver | PCA9685 16-channel PWM board (I²C address 0x40) |
| Servos | 3× hobby servos — CH0: base rotation, CH1: elbow, CH2: pen lift |
| Arm | BrachioGraph-style 2-link arm, 80 mm per segment |
| Drawing Surface | A4 paper — red pen (human), black pen (robot) |
ESP32 GPIO 21 (SDA) ──┐
ESP32 GPIO 22 (SCL) ──┤── PCA9685 I²C
│
PCA9685 CH0 ──────────── Base servo
PCA9685 CH1 ──────────── Elbow servo
PCA9685 CH2 ──────────── Pen lift servo
PCA9685 V+ ──────────── 5–6 V power supply (separate from ESP32)
src/
├── robot_align.py # Main app — camera feed, detection loop, stamping
├── robot_calibrate.py # Interactive servo calibration tool
├── processing.py # ScribbleProcessor — HSV filtering, skeletonization, contours
├── analysis.py # ScribbleAnalyzer — centroid, PCA direction, metrics
├── calibration.py # PaperCalibration — 4-corner perspective transform
├── camera.py # Camera class — device listing, selection, capture
├── alignment/
│ └── calibrator.py # PixelToRobotTransform — 3-point affine calibration
└── robot/
├── arm.py # RobotArm high-level API (move, draw, park)
├── kinematics.py # 2-link inverse kinematics
├── protocol.py # Serial command formatting
└── serial_connection.py # USB serial communication
Located in firmware/sketch_dec27a.ino. Serial command protocol:
| Command | Description |
|---|---|
SERVO <ch> <µs> |
Set servo to exact microsecond position |
MOVE <base> <elbow> <ms> |
Cosine-eased interpolated move |
LIFT UP / LIFT DOWN |
Pen control |
PARK |
Return to calibrated parking position |
CAL / SAVE / GETCAL |
Calibration storage in EEPROM |
Flash using Arduino IDE with ESP32 board support.
- Python 3.10+
- Arduino IDE with ESP32 board support
- The hardware listed above
# Clone the repository
git clone https://github.com/maayanmag/doodle_mind.git
cd doodle_mind
# Create a virtual environment and install dependencies
python -m venv venv
source venv/bin/activate
pip install -r requirements.txtOpen firmware/sketch_dec27a.ino in Arduino IDE. Select ESP32 Dev Module, set the correct serial port, and upload.
python src/robot_calibrate.pySet center positions, pen lift/down values, and the parking position. Values are saved to config/robot_calibration.json and synced to the ESP32's EEPROM.
See config/robot_calibration.example.json for the expected format.
python src/robot_align.py- The camera feed appears — position the camera directly above the paper
- Press C to start 3-point calibration
- For each point:
- Use WASD to move the arm to a position on the paper
- Click where the pen tip appears on screen
- Press ENTER to confirm
- Spread the 3 points across the reachable area for best accuracy
Calibration data is saved to config/. See the *.example.json files there for reference.
- Scribble on the paper with a red pen
- Press SPACE — the robot detects your scribbles, picks the largest unstamped one, and draws a straight stamp near it
- Press SPACE again for the next
- Press R to reset and let the robot revisit all scribbles
| Key | Action |
|---|---|
| C | Start 3-point calibration |
| SPACE | Stamp the next unstamped scribble |
| R | Reset stamp tracking |
| M | Toggle red-mask debug view |
| WASD | Manual arm control |
| Q | Park arm and quit |
doodle_mind/
├── src/ # Python application
│ ├── robot_align.py # Main entry point
│ ├── robot_calibrate.py # Servo calibration tool
│ ├── processing.py # Image processing pipeline
│ ├── analysis.py # Scribble analysis (centroid, PCA, metrics)
│ ├── calibration.py # Paper boundary calibration
│ ├── camera.py # Camera device management
│ ├── alignment/ # Pixel-to-servo affine mapping
│ └── robot/ # Serial communication & kinematics
├── firmware/ # ESP32 Arduino firmware
│ └── sketch_dec27a.ino
├── config/ # Calibration files
│ ├── alignment.example.json # Template configs (tracked)
│ ├── calibration.example.json
│ └── robot_calibration.example.json
├── docs/ # Design specification documents
│ ├── spec.md
│ └── stage_two_spec.md
├── images/ # Presentation photos
├── archive/ # Earlier development scripts
├── requirements.txt
├── LICENSE
└── README.md
- The robot draws in black, the human in red — this color separation is how the system tells its own marks from yours.
- Unreachable scribbles (outside the arm's range) are marked with a red ✕ on screen and automatically skipped.
- Already-stamped scribbles appear as gray outlines with green ✓ marks.
- All movements use cosine-eased interpolation for smooth, quiet motion.
- Calibration files in
config/are machine-specific and gitignored. Use the*.example.jsonfiles as templates.
- Patrick Tresset — Companion (2025): A drawing robot + LLM system for co-creative sketching, developed in the EACVA project (Embodied Agents in Contemporary Visual Art). Transforms the robot from executor to playful partner through speech and drawing.
Doodle Mind — M.Des Industrial Design, Bezalel Academy of Arts and Design, Jerusalem, 2025