Skip to content

Add MediaPipe Pose backend to MGT-python#334

Merged
alexarje merged 2 commits intomasterfrom
copilot/add-mediapipe-integration
Apr 13, 2026
Merged

Add MediaPipe Pose backend to MGT-python#334
alexarje merged 2 commits intomasterfrom
copilot/add-mediapipe-integration

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 13, 2026

MediaPipe offers modern, CPU-friendly pose estimation with 33 landmarks and no manual model management. This PR wires it into MGT-python as a first-class pose() backend alongside the existing OpenPose/Caffe path.

Core changes

  • _pose_estimator.py — Rewrites MediaPipePoseEstimator to use the mediapipe Tasks API (mp.tasks.vision.PoseLandmarker), required since mediapipe 0.10 removed the old mp.solutions.pose.Pose. Model files (lite/full/heavy .task, ~8–28 MB) are auto-downloaded on first use and cached in musicalgestures/models/. Drops deprecated static_image_mode parameter.

  • _pose.py — Connects MediaPipe to the existing pose() workflow:

    • Routes model='mediapipe' to a new _pose_mediapipe() helper (frame-by-frame FFmpeg pipe → MediaPipePoseEstimator → overlay skeleton → write video)
    • Adds MEDIAPIPE_POSE_CONNECTIONS (35 pairs covering all 33 landmarks) for skeleton rendering (cyan lines + red circles)
    • Adds _save_pose_txt() for exporting 33-landmark data to CSV/TSV/TXT with human-readable headers
  • tests/test_new_features.py — Adds TestMediaPipePoseIntegration (10 tests covering connections, data export, model URLs, routing); fixes test_mediapipe_no_mediapipe_raises for the new attribute name.

Usage

import musicalgestures as mg

mv = mg.MgVideo("dancer.avi")
mv.pose(model="mediapipe")          # auto-downloads model on first call
mv.pose(model="mediapipe", data_format=["csv", "tsv"])  # multi-format export

# Or use the estimator directly
from musicalgestures import MediaPipePoseEstimator
est = MediaPipePoseEstimator(model_complexity=0)   # 0=lite, 1=full, 2=heavy
result = est.predict_frame(bgr_frame)              # PoseEstimatorResult, shape (33, 3)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_full/float16/1/pose_landmarker_full.task
    • Triggering command: /usr/bin/python python -m pytest tests/test_new_features.py::TestPoseEstimator -v get --global t pull.rebase (http block)
    • Triggering command: /usr/bin/python python -m pytest tests/test_new_features.py -v --global ame git (http block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -v unset --global k/_temp/copilot-developer-action-main/dist/gh-gpgsign/gh-gpgsign-linux-x86_64 credential.usernld (http block)
  • https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task
    • Triggering command: /usr/bin/curl curl -s -o /home/REDACTED/work/MGT-python/MGT-python/musicalgestures/models/pose_landmarker_lite.task REDACTED -w %{http_code} nfig/composer/vendor/bin/git r (http block)
    • Triggering command: /usr/bin/curl curl -I REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue Apr 13, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Add Mediapipe integration to MGT Add MediaPipe Pose backend to MGT-python Apr 13, 2026
Copilot AI requested a review from alexarje April 13, 2026 16:11
@alexarje alexarje marked this pull request as ready for review April 13, 2026 16:23
@alexarje alexarje merged commit d2e6535 into master Apr 13, 2026
2 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mediapipe

2 participants