Skip to content

Partial sim with gesture control

terrytaylorbonn edited this page Feb 13, 2024 · 19 revisions

(6 Feb 2024). Uses mediapipe AI to recognize hand gestures (open, closed right hand) to control the simulated copter flight. In this example holding up an open right hand (in front of usb camera) causes the sim copter to pause the mission (the CC could then take over control). A close right hand (fist) continues the mission.

image image

In script I modified for this example the above screenshot does not show the tracing (actually it does, but not long enough for you to see it). Below are examples from the unmodified script that show the tracing.

image image

For (draft) details see the first chapter in

  • Gdrive
  • Document B1-3a-1_pymavlink_mediapipe_vxx_24.xxxx.docx first chapter section "24.0206-1 guesture control (works)"

FYI: There are many mediapipe variations to use. These are also interesting.

Drawing with the finger (left) or (right) using full body gestures (this would best for controlling a copter outside, but these mediapipe examples (that I found) do not contain the recognition part, only the tracing)... maybe do this in the near future).

image image

Note that gestures are not exactly a part of my eventual goal. But I thought it would be interesting to be able to control a copter with gestures. Note that mediapipe has a full suite of AI tools.

image

Home

Getting into the air ASAP

(0) Reference


EPIC 1 - Build/fly FPV drone

(1) FPV simulators (inav notes 0608)

(2) SBeeF405/INAV (1a)

(3) SBeeF405/BF (1b)

(4) SBeeF405/AP (1c)


EPIC 2 - Build/fly Pixhawk drone

(5) Pix6c/PX4 (2a)

(6) Pix6c/AP (2b)


EPIC 3 - Add AI to Pixhawk drone

(7) AI cc + cam

(8) AI CC+FW

(9a) SIH (frame+world sim)


EPIC 4 – Basic Autonomy

(13) CC AI (+ Mavlink) (2)

(11) FC_Mavlink_API (via CC/GCS APIs) (5.3)

(13c) CC autonomy algorithms


EPIC 5 – Advanced Autonomy

(20) Kalman KK (+Python,vectornav)

(14) Firmware dev (5)

(22) AI


EPIC 6 – Project management 24.0903


EPIC 7 – Mission platforms / special projects

(15) Mission platforms (6b)

(16) Special projects (5.6,5.5)


EPIC 8 – PITS (Pie in the sky)

(12) ROS (+ROSMAV) (was E4)

(14b) Matlab (was E5/4(HITL))

(9b) HITL (frame+world sim)

(10) (skip) SITL (total sim) (1) (was E4)

Clone this wiki locally