PhotoCoach

A real-time iOS photography coach. It tells the photographer how to move the camera and the subject how to pose — fast on-device models every frame, plus a VLM (Claude / GPT-4o / Gemini) for the occasional deeper, human-sounding tip. The design principle, validated end-to-end in a 3D harness: geometry owns continuous control (framing), the VLM owns discrete semantics (facing, pose, light) — and the device verifies that advice was executed before asking for more.

Deeper docs: docs/ — architecture, guidance signals, the full evaluation story, models, roadmap, decision log. The most complete experiment narrative is docs/a3-closed-loop-experiment.md.

What the coaching loop can do (measured, not vibes)

All results from the Blender AE37 street harness with a realistic human model, judged blind by an independent VLM (GPT-4o, both presentation orders) with Claude as a bias gauge. Full write-ups in docs/.

The A3 closed loop — geometric camera driver + VLM subject coach. From a bad start (subject far away, back turned), geometry walks the camera in and re-frames while the VLM turns the subject:

When the framing is already good, geometry does nothing and only the subject signal fires — the case a camera-only coach can never fix:

Key numbers (N=15 paired starts, verified start distribution): decomposed A3 beats the single joint VLM 5–1 head-to-head (Claude gauge agrees 10–4); the single VLM is net-negative vs the start (2–5) on varied moderate defects and wrecked 1/5 good starts — A3 wrecked 0/5. The facing signal carries most of the edge (A3 vs geometry-only: 4–0).

Aesthetics are a target, not a mechanism. The same geometric driver can aim at any framing preset; a 5-scene taste test (both judges, all pairs, both orders) picked the targets: rule-of-thirds ≥ centred everywhere, half-body loses on scenic backdrops. Thirds-with-nose-room is now the default full-body target:

Every paid run is gated by free verification layers — offline self-checks, Blender ground-truth box cross-checks, geometry-only dry runs, and a rendered start gallery for human sign-off (8 backdrops: 4 street segments × 2 look directions):

Architecture

Two loops, glued by execution verification (advise → watch it happen → only then advise again):

CameraController ──frames──▶ FrameAnalyzer ──FrameInsights──▶ GuidanceCoordinator ──▶ SwiftUI
   (AVFoundation)            (Vision requests)                      │
                                                                    ├─ CompositionEngine   (camera moves, every frame)
                                                                    ├─ PoseRetriever ─ DiversityReranker (scene→poses, Greedy DPP)
                                                                    ├─ PoseGuidanceEngine  (posing cues)
                                                                    └─ VLMCoach            (deep tips, throttled)

GuidedSession (multi-turn VLM coaching with execution verification)
   ConvergingCoach ──one move──▶ user executes ──▶ verified by:
      rotation: CoreMotion attitude (MotionMeasurer)
      translate/zoom: subject-box displacement (FrameDisplacementEstimator)
   ──converged──▶ ready-to-shoot

The fast pipeline is pure value types (FrameInsights, BodyPose, GuidanceCue), unit-testable without a camera — see PhotoCoachTests/.

How it maps to the two papers

Paper	Role in the app	On-device adaptation
InstaPose: Scene-Aware Pose Recommendation via ViT + Diversity-Optimized Reranking (ICCVW 2025)	Suggests poses that suit the scene, spoken-style cues	ViT-L/16 + FAISS over 247k → VNGenerateImageFeaturePrintRequest + cosine retrieval over a bundled skeleton gallery; Greedy DPP reranking ported verbatim (DiversityReranker); pose correction → PoseGuidanceEngine vs live VNDetectHumanBodyPose.
Camera View Adjustment Prediction for Improving Image Composition (2021)	Tells the photographer how to move: pan / tilt / zoom / level	View-adjustment model → CompositionEngine (8-direction + magnitude from Vision cues); a trained CoreML model drops in behind the same interface. The eval harness's geometric camera driver (a3_geometry_camera) is the exact-ground-truth version of this signal.

Repo layout

path	contents
PhotoCoach/	The iOS app (Xcode 26 project, file-system-synchronized groups).
PhotoCoachTests/	Unit tests: guidance round-trips, coach parsing, motion/visual execution tracking, Vision fixtures.
eval/	Python evaluation: judge_eval.py (calibrated blind VLM judges, Claude + GPT-4o), scorer/labels training scripts.
eval/scene3d/	The 3D harnesses — core/ shared engine, person/ active person-in-scene line (A3), scenery/ concluded scenery line, sim/ pure-Python era. See eval/scene3d/README.md.
docs/	Design + experiment write-ups; docs/images/ the gallery above.

Build & run

open PhotoCoach.xcodeproj

1. Confirm signing under the PhotoCoach target → Signing & Capabilities.
2. Run on a real device (camera + Vision + CoreMotion need hardware).

Deployment target iOS 26.2; Swift 5 mode with SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor. The pure algorithms also type-check standalone without full Xcode (CLT-only machine friendly):

cd PhotoCoach
xcrun swiftc -typecheck Pose/BodyPose.swift Guidance/GuidanceModels.swift \
  Scene/PoseLibrary.swift Scene/PoseRetriever.swift Scene/DiversityReranker.swift \
  Pose/PoseGuidanceEngine.swift Composition/CompositionEngine.swift \
  Composition/CompositionModel.swift

VLM keys — app: Settings (gear) → provider → paste key → Save (Keychain only). Without a key everything runs except the VLM coach. VLM keys — eval scripts: dotenv-style ~/.photocoach_key (ANTHROPIC_API_KEY=… / OPENAI_API_KEY=…), never committed.

Evaluation methodology (the short version)

Three questions, three layers:

1. Is the guidance logic correct? Round-trip/perturbation unit tests (RoundTripGuidanceTests, GuidedSessionTests, MotionMeasurerTests, FrameDisplacementEstimatorTests) — instant, no images.
2. Is perception accurate? VisionPerceptionTests on fixture images; the realistic person model is verified detectable by Vision (front and back).
3. Does the photo actually improve? The eval/scene3d closed loops with ground-truth geometry + independent blind VLM judging — this is where the A3 / preset results above come from.

Hard-won methodology rules (each learned the expensive way, see docs/a3-closed-loop-experiment.md): verify start distributions with ground truth before paying for a run; angle units are radians (hfov degrees) — aim cameras by numeric solve, never by hand; judges get both presentation orders; and the coach must never also be the judge (measured bias ≈ 0, but an independent judge retires the question).

Status & next steps

• Done: on-device execution verification (rotation + visual displacement), anti-flicker guidance UI, A3 decomposition validated, thirds default, preset library with a distance-aware move engine.
• Next: on-device end-to-end run of the Swift GuidedSession loop; a zoom/hfov move (退后变焦) in converging_coach.apply_move_fine + a3_geometry_camera for distortion-free tight framing; half-body retest on cluttered backgrounds; second-scene replication of the A3 result; ghost skeleton overlay for pose matching.