Live2D with Lipsync (Outdated) #117

RaSan147 · 2023-10-06T12:35:49Z

No description provided.

Changes: 1. updated model.motion(group, index, priority) to model.motion(group, index, priority, sound, volume, expression); 2. added model.stopSpeaking() 3. updated readme.MD with demos 4. Workflow will save files to dist (won't be gitignored) 5. Praying this new change doesn't break

voice volume expressions are now optional arg {name: value, ....}

guansss

Sorry for the long wait! You did a great job on this PR and I really appreciate the effort you put into it.

I've added a few comments for things that need to be addressed before this can be merged.

.github/workflows/test.yml

.gitignore

guansss · 2023-12-01T10:26:19Z

README.md

+#### Feel free to support the Maintainer:
+<a href="https://www.buymeacoffee.com/RaSan147" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>
+


It's okay to put this in your fork, but not into the main repository.

Will make another fork 😅 copy pasted it in all my repo. Sorry.

guansss · 2023-12-01T10:29:25Z

README.md

@@ -97,16 +100,24 @@ import { Live2DModel } from 'pixi-live2d-display/cubism2';
 import { Live2DModel } from 'pixi-live2d-display/cubism4';
 ```

-#### Via CDN
+#### Via CDN (lipsync patched)


This and the (not lipsync patch) above is unnecessary after this feature is merged.

Yeah will change that.

guansss · 2023-12-01T10:30:29Z

README.md

+<script src="https://cdn.jsdelivr.net/gh/RaSan147/pixi-live2d-display@v0.4.0-ls-2/dist/index.min.js"></script>

 <!-- if only Cubism 2.1 -->
-<script src="https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism2.min.js"></script>
+<script src="https://cdn.jsdelivr.net/gh/RaSan147/pixi-live2d-display@v0.4.0-ls-2/dist/cubism2.min.js"></script>

 <!-- if only Cubism 4 -->
-<script src="https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism4.min.js"></script>
+<script src="https://cdn.jsdelivr.net/gh/RaSan147/pixi-live2d-display@v0.4.0-ls-2/dist/cubism4.min.js"></script>


These links should point to the main repository.

guansss · 2023-12-01T11:52:11Z

src/cubism-common/SoundManager.ts

@@ -1,7 +1,7 @@
 import { logger, remove } from '@/utils';

 const TAG = 'SoundManager';
-const VOLUME = 0.5;
+const VOLUME = 0.9;


This should not be changed. 0.9 would be too loud as a default volume.

However since the lip movement value is dependent on the audio volume, lowering it makes unusual movement (like mumbling)

Is it possible to add a multiplier after getting the value? Some thing like amplitude *= 2

nope, like if you set volume to 0, there will be no response, no matter how the audio or multiplier is (Browser JS API, cant change).... But yeah multiplier can be really great, will try to add one.

tried, sending Amp variable to analizer is hard. Let it be...

guansss · 2023-12-01T11:56:21Z

src/cubism-common/SoundManager.ts

@@ -39,6 +41,8 @@ export class SoundManager {

        audio.volume = this._volume;
        audio.preload = 'auto';
+        audio.autoplay = true;


autoplay should not be true because we are not immediately playing the audio. When config.motionSync is on, we'll postpone it until the related motion/expression is ready.

Thanks for the info

guansss · 2023-12-01T12:20:49Z

src/cubism-common/SoundManager.ts

@@ -39,6 +41,8 @@ export class SoundManager {

        audio.volume = this._volume;
        audio.preload = 'auto';
+        audio.autoplay = true;
+        audio.crossOrigin = "anonymous";


crossOrigin should not be set by default, otherwise if the audio's source is cross-origin but the server does not respond with a CORS header, the audio will be blocked by the browser.

It should be an option, for example, model.speak('audio.mp3', { crossOrigin: true })

guansss · 2023-12-01T12:25:12Z

src/cubism2/Cubism2InternalModel.ts

@@ -33,9 +34,12 @@ export class Cubism2InternalModel extends InternalModel {
    angleZParamIndex: number;
    bodyAngleXParamIndex: number;
    breathParamIndex: number;
+    mouthFormIndex: number;


This property seems unused.

I do remember it using somewhere in cubism 2, cubism 4 has it by default. It takes the mouthForm id...

guansss · 2023-12-01T12:30:01Z

src/cubism4/Cubism4InternalModel.ts

+    updateFacialEmotion(mouthForm: number) {
+        this.coreModel.addParameterValueById(this.idParamMouthForm, mouthForm); // -1 ~ 1
+    }


This is not related to lip sync, I suppose?

Thats the main thinggggg. 😅 It makes the lips move.

RaSan147 · 2023-12-07T12:31:25Z

Sorry for the late reply.
I'll make a branch with the changes you mentioned (also fixing a lot of merge conflict from the latest update)
Thanks for taking your time and looking into it. This project is awesome.

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

also remove cache buster and autoplay

into for_PR

9a9d20f 638053e

RaSan147 added 9 commits October 6, 2023 18:16

Automated report

d378339

Update index.js links

3ff3a53

Update test.yml

8539e76

Update test.yml

cf0f4c4

Update test.yml

f53e25a

Fix Lip sync. Breaking change

7deba12

voice volume expressions are now optional arg {name: value, ....}

Automated report

06bfbd3

Update README.md and added video

1e89254

guansss requested changes Dec 1, 2023

View reviewed changes

RaSan147 added 12 commits December 7, 2023 05:21

Merge branch 'master' into master

34831e8

Fix type error on ci test

2d0c512

clear dist, using workflow to get output files

2617eae

removed Version number

de764a1

Now supports other b64 audio

2877f47

Remove audio url validation

25ddef6

rename speakUp -> speak

b4a394a

rename resetMotions -> stopMotions

6f3a779

return false when audio play fails

e878d79

Added crossorigin for speak voice source

031ca86

Place missing CrossOrigin arg

ae1c923

Update MotionManager.ts

c0cb7f5

RaSan147 changed the title ~~Live2D with Lipsync (using audio file/link)~~ Live2D with Lipsync (Outdated) Dec 14, 2023

Update readme as per PR

c8eca3f

RaSan147 mentioned this pull request Dec 14, 2023

Live2D with Lipsync (using audio file/link) #122

Open

RaSan147 and others added 4 commits December 14, 2023 19:05

Check any base64 data for audio

7f8ec7a

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

Remove autoplay attr from audio

06d7255

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

Remove cache blocker.

96a6f82

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

remove wav only blob condition

9fc5a21

also remove cache buster and autoplay

RaSan147 and others added 29 commits December 31, 2023 03:13

Merge branch 'guansss:master' into master

1a5dd5e

Merge branch 'guansss:master' into for_PR

b005f73

RAN npm run lint:fix

ff1a588

Merge branch 'for_PR' of https://github.com/RaSan147/pixi-live2d-display

0495f5b

into for_PR

Merge remote-tracking branch 'origin/for_PR'

44d62a4

reverting readme with this repo links

75d50c9

fix startmotion, getting error

1cd47d6

All test passed. Everything OK

65f31e2

revert: undo unexpected changes

92af59b

fix: model.speak() not returning true on success

b00b64b

Bring patch from for_pr

0409b60

9a9d20f 638053e

Merge branch 'update'

8ba8fa5

Fix artifact upload on failed test

f35e417

Update test.yml

35ce10d

Update test.yml

aab633a

Upload dist for jsdelivr support

3655244

Update .gitignore

e91f3aa

Automated report

4e58ef7

Update README.md

16b0e6f

credit fatalgoth

8e73cff

credit windjackz

b73f657

Update pixijs link README.md

af0126d

Update README.md

18f7c36

add callback function to motion and speak

ea3de0e

fix Cannot find type definition file for 'offscreencanvas'.

c66259c

Automated report

3dcc936

Update README.md

e720485

patch node.env.NODE_ENV issue (action failed to update)

6f899f3

Fix #6 for crossOrigin docs

41bdca4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Live2D with Lipsync (Outdated) #117

Live2D with Lipsync (Outdated) #117

RaSan147 commented Oct 6, 2023

guansss left a comment

guansss Dec 1, 2023

RaSan147 Dec 7, 2023

guansss Dec 1, 2023

RaSan147 Dec 7, 2023

guansss Dec 1, 2023

guansss Dec 1, 2023

RaSan147 Dec 7, 2023

guansss Dec 11, 2023

RaSan147 Dec 11, 2023

RaSan147 Dec 14, 2023

guansss Dec 1, 2023

RaSan147 Dec 7, 2023

guansss Dec 1, 2023

RaSan147 Dec 30, 2023

guansss Dec 1, 2023

RaSan147 Dec 7, 2023

guansss Dec 1, 2023

RaSan147 Dec 7, 2023

RaSan147 commented Dec 7, 2023

		#### Feel free to support the Maintainer:
		<a href="https://www.buymeacoffee.com/RaSan147" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

Live2D with Lipsync (Outdated) #117

Are you sure you want to change the base?

Live2D with Lipsync (Outdated) #117

Conversation

RaSan147 commented Oct 6, 2023

guansss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RaSan147 commented Dec 7, 2023