Live2D with Lipsync (using audio file/link) #122

RaSan147 · 2023-12-14T00:31:54Z

Solving issues mentioned in #117

Changes: 1. updated model.motion(group, index, priority) to model.motion(group, index, priority, sound, volume, expression); 2. added model.stopSpeaking() 3. updated readme.MD with demos 4. Workflow will save files to dist (won't be gitignored) 5. Praying this new change doesn't break

voice volume expressions are now optional arg {name: value, ....}

guansss

Thanks again for the PR! I think we are getting close but some changes are still needed as described in the comments.

I noticed that some of the code is not properly linted. After making changes to the code, please run npm run lint:fix to automatically fix the linting errors, and address any remaining errors manually. (except for the triple slash reference errors, which I will fix later)

After you finish theses changes, I'll be adding some tests to make sure this feature works as expected.

src/cubism-common/MotionManager.ts

src/cubism-common/SoundManager.ts

guansss · 2023-12-14T11:45:20Z

src/cubism4/Cubism4InternalModel.ts

@@ -248,6 +257,11 @@ export class Cubism4InternalModel extends InternalModel {
        this.coreModel.addParameterValueById(this.idParamBodyAngleX, this.focusController.x * 10); // -10 ~ 10
    }

+
+    updateFacialEmotion(mouthForm: number) {


Could a name like setMouthForm be better? Because it's not changing the entire facial expression but only the mouth form. Also, update implies that this function will do some computations other than setting the value, so set will be more suitable here.

As a new API, this method should also be added to Cubism2InternalModel for consistency.

I'll test and run on the cubism 2 (well the issue is i tried and failed to set up the development env on my local system, but the github action worked fine even the codespace failed, i know my skill issue) So probably won't be able to run the npm lint (will try)

The development guide in DEVELOPMENT.md was a bit messy and I've rewritten it, now I guess there won't be problems if you follow the steps (if there is please let me know!)

It's not your issue but the codespaces being problematic with submodules, browser testing etc. So better run it locally.

Thanks a lot 😭

I decided to remove this method because setting this param is pretty straightforward and isn't really worth adding a method for it.

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

also remove cache buster and autoplay

into for_PR

RaSan147 · 2024-03-07T12:05:03Z

Yeah, lets give them options (and specify in the docs) and let them pick

xumx · 2024-03-11T06:37:24Z

A new behavior is breaking some existing tests, that is, when a motion with a sound is playing, the model does not allow another motion to start even if it has a higher priority, because there's already a playing audio.

I tried removing that audio check but just got some other errors. Anyway, I think this is mainly because we didn't have a thorough design for how to reconcile motions with sounds (model.motion()) and lipsync audios (model.speak()).

So here we go! My intuitive idea is, motions shouldn't be disallowed to play because of a playing audio, and motion sounds should have a higher priority than lipsync audios. So if a motion is going to play and it has a sound, the current lipsync audio should be canceled; and if it doesn't have a sound, the lipsync audio should keep playing along with the motion.

Could you share your thoughts on this?

Why can't both speaking audio and motion sound be played together?

guansss · 2024-03-11T07:12:56Z

@xumx Yeah that's exactly what my latest comment was talking about.

guansss · 2024-03-11T10:32:03Z

Hey @RaSan147 , is there a reason why these two calculations are different? I wonder if they can be consistent so I can move them into InternalModel.

pixi-live2d-display/src/cubism2/Cubism2InternalModel.ts

Lines 250 to 266 in b00b64b

    
           let value = this.motionManager.mouthSync(); 
        
           let min_ = 0; 
        
           const max_ = 1; 
        
           const bias_weight = 1.2; 
        
           const bias_power = 0.7; 
        
           if (value > 0.0) { 
        
               min_ = 0.4; 
        
           } 
        
           value = Math.pow(value, bias_power); 
        
           value = clamp(value * bias_weight, min_, max_); 
        
           for (let i = 0; i < this.motionManager.lipSyncIds.length; ++i) { 
        
               this.coreModel.setParamFloat( 
        
                   this.coreModel.getParamIndex(this.motionManager.lipSyncIds[i]!), 
        
                   value, 
        
               ); 
        
           }

pixi-live2d-display/src/cubism4/Cubism4InternalModel.ts

Lines 225 to 236 in b00b64b

    
           let value = this.motionManager.mouthSync(); 
        
           let min_ = 0; 
        
           const max_ = 1; 
        
           const weight = 1.2; 
        
           if (value > 0) { 
        
               min_ = 0.4; 
        
           } 
        
           value = clamp(value * weight, min_, max_); 
        
           for (let i = 0; i < this.motionManager.lipSyncIds.length; ++i) { 
        
               model.addParameterValueById(this.motionManager.lipSyncIds[i], value, 0.8); 
        
           }

RaSan147 · 2024-03-11T11:12:59Z

Hey @RaSan147 , is there a reason why these two calculations are different? I wonder if they can be consistent so I can move them into InternalModel.

pixi-live2d-display/src/cubism2/Cubism2InternalModel.ts

Lines 250 to 266 in b00b64b

let value = this.motionManager.mouthSync();

let min_ = 0;

const max_ = 1;

const bias_weight = 1.2;

const bias_power = 0.7;

if (value > 0.0) {

min_ = 0.4;

}

value = Math.pow(value, bias_power);

value = clamp(value * bias_weight, min_, max_);

for (let i = 0; i < this.motionManager.lipSyncIds.length; ++i) {

this.coreModel.setParamFloat(

this.coreModel.getParamIndex(this.motionManager.lipSyncIds[i]!),

value,

);

}

pixi-live2d-display/src/cubism4/Cubism4InternalModel.ts

Lines 225 to 236 in b00b64b

let value = this.motionManager.mouthSync();

let min_ = 0;

const max_ = 1;

const weight = 1.2;

if (value > 0) {

min_ = 0.4;

}

value = clamp(value * weight, min_, max_);

for (let i = 0; i < this.motionManager.lipSyncIds.length; ++i) {

model.addParameterValueById(this.motionManager.lipSyncIds[i], value, 0.8);

}

Sorry, i might forgot to update the another one. Would recommend adding bias_power and weighted one (otherwise lips don't move well).

guansss · 2024-04-15T07:18:27Z

Finally it's ready to merge! Before I merge it, are there any changes you would like to make or suggest?

RaSan147 · 2024-04-17T14:24:38Z

Sorry didn't notice, gimme a bit time, testing...

RaSan147 · 2024-04-17T14:27:21Z

btw can you please check the PR i've sent you on cubism folder repo...
that should fix the process not found (or you may tweak the results a bit)

RaSan147 · 2024-04-17T14:37:56Z

package.json

vite requires terser in this version (my fresh install was not working without it, so kindly add it the deps
i fixed it with npm install terser

RaSan147 · 2024-04-17T14:52:58Z

src/cubism-common/LipSync.ts

add another option, force or priority, if force or higher priority, will stop current audio, otherwise current one will play.

will add onFinish and onError callback as option

RaSan147 · 2024-04-17T15:05:05Z

Well, I'm gonna miss motion(...., {sound})
it was a great option (since its optional feature, removing it is kinda feeling like a bad idea)
that would help retain certain posture and motion while speaking. Also the expression and many more things are missing from the PR_version...
😥

liyao1520 · 2024-05-08T06:07:31Z

期待，更新

RaSan147 · 2024-05-08T07:33:01Z

Gotta re-test and look for compatible way to shift from patch to official version

liyao1520 · 2024-05-09T10:46:19Z

#150

live2d official website The demonstration video of the model can flexibly display mouth movements, and the lip-syncing looks quite natural. Demo Video

In this example model, not only can the mouth opening be set based on audio information, but vowel mouth shapes can also be set by adjusting 'ParamA', 'ParamE', 'ParamI', 'ParamO', 'ParamU'.

model.internalModel.coreModel.setParameterValueById('ParamMouthOpenY', mouthY)
model.internalModel.coreModel.setParameterValueById('ParamA', 0.3)

I feel there might be better methods to achieve lip-syncing. Can the model be set to correspond to the mouth shape based on the audio?

Also, Alibaba Cloud's TTS can output the time position of each Chinese character/English word in the audio. How can the model play the audio, and can it set the corresponding mouth shape based on the phonetic information?

Seeking guidance from the experts! 🙏

RaSan147 · 2024-05-25T02:47:03Z

Finally it's ready to merge! Before I merge it, are there any changes you would like to make or suggest?

Whenever you're ready. Thanks for all your hard works

RaSan147 added 22 commits October 6, 2023 18:16

Automated report

d378339

Update index.js links

3ff3a53

Update test.yml

8539e76

Update test.yml

cf0f4c4

Update test.yml

f53e25a

Fix Lip sync. Breaking change

7deba12

voice volume expressions are now optional arg {name: value, ....}

Automated report

06bfbd3

Update README.md and added video

1e89254

Merge branch 'master' into master

34831e8

Fix type error on ci test

2d0c512

clear dist, using workflow to get output files

2617eae

removed Version number

de764a1

Now supports other b64 audio

2877f47

Remove audio url validation

25ddef6

rename speakUp -> speak

b4a394a

rename resetMotions -> stopMotions

6f3a779

return false when audio play fails

e878d79

Added crossorigin for speak voice source

031ca86

Place missing CrossOrigin arg

ae1c923

Update MotionManager.ts

c0cb7f5

Update readme as per PR

c8eca3f

guansss requested changes Dec 14, 2023

View reviewed changes

RaSan147 and others added 7 commits December 14, 2023 19:05

Check any base64 data for audio

7f8ec7a

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

Remove autoplay attr from audio

06d7255

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

Remove cache blocker.

96a6f82

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

remove wav only blob condition

9fc5a21

also remove cache buster and autoplay

Merge branch 'guansss:master' into for_PR

b005f73

RAN npm run lint:fix

ff1a588

Merge branch 'for_PR' of https://github.com/RaSan147/pixi-live2d-display

0495f5b

into for_PR

guansss added 14 commits March 18, 2024 18:52

feat: extract lip sync code to LipSync class

b511d74

test: fix broken test procedure

12e9530

test: add lip sync test

ee42229

chore: fix failing npm install

6f3cd68

feat: skip lip sync analyzer when no audio is playing

40cd3f3

fix: model.motion() parameters should be optional

638053e

feat: default lip sync id for cubism 4 models

9a9d20f

feat: make options.crossOrigin also work for audios

2ecdb53

test: add aborted lip sync test

9cb5c38

test: make lip sync test more stable

f53509b

test: make lip sync test more stable (for real)

d69a771

feat: merge lipSync.analyze() into lipSync.getValue()

94a7ea1

test: temporarily disable parallelism to fix random failures

87476a4

revert: remove updateFacialEmotion method

718215d

RaSan147 commented Apr 17, 2024

View reviewed changes

DominicStewart approved these changes Jun 7, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Live2D with Lipsync (using audio file/link) #122

Live2D with Lipsync (using audio file/link) #122

RaSan147 commented Dec 14, 2023

guansss left a comment

guansss Dec 14, 2023

RaSan147 Dec 14, 2023

guansss Dec 15, 2023

RaSan147 Dec 15, 2023

guansss Apr 15, 2024

RaSan147 commented Mar 7, 2024

xumx commented Mar 11, 2024

guansss commented Mar 11, 2024

guansss commented Mar 11, 2024

RaSan147 commented Mar 11, 2024

guansss commented Apr 15, 2024

RaSan147 commented Apr 17, 2024

RaSan147 commented Apr 17, 2024

RaSan147 Apr 17, 2024

RaSan147 Apr 17, 2024

RaSan147 Apr 17, 2024

RaSan147 commented Apr 17, 2024

liyao1520 commented May 8, 2024

RaSan147 commented May 8, 2024

liyao1520 commented May 9, 2024

RaSan147 commented May 25, 2024

Live2D with Lipsync (using audio file/link) #122

Are you sure you want to change the base?

Live2D with Lipsync (using audio file/link) #122

Conversation

RaSan147 commented Dec 14, 2023

guansss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RaSan147 commented Mar 7, 2024

xumx commented Mar 11, 2024

guansss commented Mar 11, 2024

guansss commented Mar 11, 2024

RaSan147 commented Mar 11, 2024

guansss commented Apr 15, 2024

RaSan147 commented Apr 17, 2024

RaSan147 commented Apr 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RaSan147 commented Apr 17, 2024

liyao1520 commented May 8, 2024

RaSan147 commented May 8, 2024

liyao1520 commented May 9, 2024

RaSan147 commented May 25, 2024