Local beauty retouch with frame-by-frame mask tracking

[liangqirong94-oss]

Feature request: local beauty retouch with frame-by-frame mask tracking

Hi, I am looking for a way to automate a very specific CapCut / JianYing desktop editing workflow for beauty product ad videos.

Short version

I need editable draft automation for:

duplicate source clip as upper PIP layer -> apply beauty retouch only on that upper layer -> add a local mask -> keyframe the mask frame by frame / stroke by stroke as the cosmetic product touches and spreads on the face

This should not be full-face smoothing. The retouch should only appear where the foundation stick / brush / finger / product material has already touched the skin.

Why this matters

In beauty product videos, the viewer needs to see the product effect gradually appear on the face. If the whole face is retouched from the beginning, the product demonstration becomes meaningless.

Correct behavior:

• Before the product touches the face: no retouch.
• When the product touches the cheek / nose / under-eye area: retouch only that local area.
• As the product is blended: the mask follows each stroke.
• The lower original video track remains unchanged.
• The final result stays editable inside CapCut / JianYing, not only as an exported video.

Required draft features

• Add or duplicate an upper picture-in-picture video segment.
• Attach beauty/retouch effects only to that upper segment.
• Add a rectangular or custom mask to that upper segment.
• Keyframe mask position, width, height, rotation, and feather.
• Support multiple keyframes per product stroke.
• Avoid media-missing issues when saving the draft.
• Detect whether newer JianYing drafts are encrypted or unsupported.

Ideal output

Given an existing draft and a product-application segment, the automation should create:

• original lower track preserved,
• upper local retouch PIP layer,
• local mask attached to that upper layer,
• multiple mask keyframes following the product-on-face area,
• validation report with overlay count, mask count, keyframe count, and missing media count.

Questions

1. Does capcut-cli currently support writing mask keyframes for existing video segments?
2. Can mask position/size/rotation/feather be keyframed through the current schema?
3. Is there any known limitation with newer encrypted JianYing Pro drafts?
4. Would this workflow fit as a new command or plugin in this project?
5. If direct draft editing is not possible, would a UI-automation fallback be the recommended path?

I can provide a sample JianYing draft, source video, and reference screenshots showing the expected local-mask retouch behavior.

[renezander030]

Hi @liangqirong94-oss, thanks for the detailed writeup, this is a great use case.

Quick answers:

1 & 2. Masks and keyframes both exist today, but they do not yet meet in the middle. keyframe animates the segment transform (position/scale/rotation/alpha/volume), not the mask geometry. So a static local mask works now; a mask that follows each stroke does not yet. That is exactly the gap your request points at.

3. JianYing 6.0+ encrypts draft_content.json. capcut decrypt and capcut version detect it but cannot edit encrypted drafts. For now, pin to JianYing 5.9 or use CapCut, where everything stays plain JSON and editable.

4. Yes, this fits as a natural extension: adding mask position/width/height/rotation/feather as keyframeable properties, plus a helper to duplicate a clip onto an upper PIP layer. I opened #44 to track it.

5. Direct draft editing is the right path here, not UI automation, since it keeps the result editable inside CapCut like you want. UI automation would only be a fallback for encrypted drafts.

The sample draft, source video, and screenshots you offered would genuinely help me get the mask-keyframe schema right the first time. If you can attach or link them here, I will build against your real workflow.

Thanks again,
Rene