It doesn't work on WebGL built with Unity 2021

[uezo]

It works on WebGL built with Unity 2020 but it doesn't with 2021.

I found that the type of WEBAudio.audioInstances is changed from array to object (hash), and this change causes this bug in uLipSyncWebGL.jslib.
https://github.com/uezo/uLipSyncWebGL/blob/main/Plugins/uLipSyncWebGL.jslib#L11

And, I also found that reconnecting gain node to outputHookNode periodically(e.g. every 200ms) is needed to keep lip syncing continuously. I don't know why...🤔

gain -x-> hook ---> dest

[zhuchenyang]

Would you update this for Unity 2021?
I'm waiting for it. 😃

[uezo]

Hi @zhuchenyang ,
Sorry for make you waiting.
But this disconnecting issue is not clearly documented, maybe an implicit spec or bug.
I will share you a work around for this later, but I don't know it have some side effects or not.

[zhuchenyang]

Hello, I just make it run on both 2020 and 2021.
In line 11: for (var i = 0; i < WEBAudio.audioInstances.length; i++) => for (var i = 1; i <= WEBAudio.audioInstanceIdCounter; i++) .
Maybe the reason is "length" is not supported in chronium based browers.

[ArEnSc]

Hello, I just make it run on both 2020 and 2021. In line 11: for (var i = 0; i < WEBAudio.audioInstances.length; i++) => for (var i = 1; i <= WEBAudio.audioInstanceIdCounter; i++) . Maybe the reason is "length" is not supported in chronium based browers.

what version of 2020 and 2021 did you get this to work on? I see it connecting in the webgl code but it doesn't do anything to the blendshapes

[ArEnSc]

Ok further check I get a string of all 0's coming back despite connecting ....

[ArEnSc]

So it turns out this is the issue @zhuchenyang is right about the query how ever the browser is preventing the audio context from starting up for chrome. So this sorta fixes it but I think I have to find a way to keep that context resumed.

  public void Update() {
            if (Input.GetKeyDown(KeyCode.Return)) {
            #if UNITY_WEBGL && !UNITY_EDITOR
            ulipSyncScript = GetComponent<uLipSyncScript>();
            if (ulipSyncScript != null)
            {
                InitWebGLuLipSync(gameObject.name, "SetAudioSampleData");
            }
            Debug.Log("Started Audio Context");
            #endif
            }
        }

        private void SetAudioSampleData(string inputString)
        {
            
            // if (!ulipSyncScript.audioSourceProxy)
            // {
                Debug.Log(inputString);
                var samplingData = inputString.Split(',').Select(s => Convert.ToSingle(s)).ToArray();
                //Debug.Log($"{samplingData}");
                ulipSyncScript.OnDataReceived(samplingData, 1);
            //}
        }```
        

[ArEnSc]

The audio context was not allowed to start is the message I get from chrome

[Gabri94x]

Unity Editor Version 2022.1.24 / 2021.3.13 LTS.
Hello there! by using these editor versioni, the lip syinc works bad.
By logging the inputString value i get all almost zero values in webgl build,

In other hand printing in the editor the inputString log is right.

There is any solution?

[ArEnSc]

Unity Editor Version 2022.1.24 / 2021.3.13 LTS. Hello there! by using these editor versioni, the lip syinc works bad. By logging the inputString value i get all almost zero values in webgl build, In other hand printing in the editor the inputString log is right. There is any solution?

the values are correct if you are getting them from the webgl version I get 0's all the time

[uezo]

My workaround.

mergeInto(LibraryManager.library, {
    InitWebGLuLipSync: function(targetObjectNamePtr, targetMethodNamePtr) {
        const targetObjectName = UTF8ToString(targetObjectNamePtr);
        const targetMethodName = UTF8ToString(targetMethodNamePtr);

        const outputHookNode = WEBAudio.audioContext.createScriptProcessor();
        outputHookNode.onaudioprocess = function (stream) {
            SendMessage(targetObjectName, targetMethodName, event.inputBuffer.getChannelData(0).join(','));
        };

        const connectAudioNodes = function (audioInstance) {
            if (audioInstance != null && audioInstance.hasOwnProperty("gain")) {
                // connect gain -> outputHookNode
                audioInstance.gain.connect(outputHookNode);
                // connect outputHookNode -> dest (dummy: no output data will go to dest)
                outputHookNode.connect(WEBAudio.audioContext.destination);
                console.log("Connected audio nodes successfully");
                return true;
            } else {
                return false;
            }
        };

        const jobId = setInterval(function() {
            for (var key in WEBAudio.audioInstances) {
                if (connectAudioNodes(WEBAudio.audioInstances[key])) {
                    // Continuously reconnect gain -> outputHookNode (they will be disconnected silently, I don't know why...)
                    setInterval(function() {
                        WEBAudio.audioInstances[key].gain.connect(outputHookNode);
                    }, 200);
                    clearInterval(jobId);
                    break;
                }
            }
        }, 200);
    },
});

[ArEnSc]

@uezo have you considered using the new api ? maybe this is deprecated and they broke it by accident

[uezo]

Hi @ArEnSc , "new api" means WEBAudio.audioInstanceIdCounter?
Actually I found WEBAudio.audioInstanceIdCounter returns number at runtime, but I couldn't find the official document about that explains the spec of this api.

And, we have to make out the way to access each audio instance in the for loop.

if (WEBAudio.audioInstances[i] != null && WEBAudio.audioInstances[i].hasOwnProperty("gain")) {

Do you have any ideas?

[ArEnSc]

@uezo >

sadly my knowledge of JS is limited by the new api looks like
ScriptProcessorNode

[Gabri94x]

My workaround.

mergeInto(LibraryManager.library, {
    InitWebGLuLipSync: function(targetObjectNamePtr, targetMethodNamePtr) {
        const targetObjectName = UTF8ToString(targetObjectNamePtr);
        const targetMethodName = UTF8ToString(targetMethodNamePtr);

        const outputHookNode = WEBAudio.audioContext.createScriptProcessor();
        outputHookNode.onaudioprocess = function (stream) {
            SendMessage(targetObjectName, targetMethodName, event.inputBuffer.getChannelData(0).join(','));
        };

        const connectAudioNodes = function (audioInstance) {
            if (audioInstance != null && audioInstance.hasOwnProperty("gain")) {
                // connect gain -> outputHookNode
                audioInstance.gain.connect(outputHookNode);
                // connect outputHookNode -> dest (dummy: no output data will go to dest)
                outputHookNode.connect(WEBAudio.audioContext.destination);
                console.log("Connected audio nodes successfully");
                return true;
            } else {
                return false;
            }
        };

        const jobId = setInterval(function() {
            for (var key in WEBAudio.audioInstances) {
                if (connectAudioNodes(WEBAudio.audioInstances[key])) {
                    // Continuously reconnect gain -> outputHookNode (they will be disconnected silently, I don't know why...)
                    setInterval(function() {
                        WEBAudio.audioInstances[key].gain.connect(outputHookNode);
                    }, 200);
                    clearInterval(jobId);
                    break;
                }
            }
        }, 200);
    },
});

i tried this but doesnt works in webgl build for me, character just open and closes mouth

[uezo]

@ArEnSc I don't have considered new APIs such as AudioWorklets and AudioWorkletNode for now. Just workaround. But I understand ScriptProcessorNode is deprecated and I have to replace them in the (near) future. Thank you.

[uezo]

@Gabri94x

character just open and closes mouth

Isn't it lip sync? Does it work on native platform, not WebGL? If not, I guess uLipSync is not configured correctly.

[Gabri94x]

in the editor everything works fine, avatar talking has a very nice and smooth mouth shaping, but when i build to the webgl platform, its just open the mouth when it talks, and close it when it stops to talk

[PaoloConteVVIP]

Hello, have you found any solution to run this on newer unity/broser versions?
I'm using 2021.3.21 (LTS) and sadly the lipsync on web doesn't work anymore.

[Solidsoldier12]

Any progress on making the lipsync work on webgl builds?

[uezo]

@Solidsoldier12 This works.
#1 (comment)

[Solidsoldier12]

@Solidsoldier12 This works.
#1 (comment)

I tried it but still doesn't seem to work man🥲
Gonna do some more deep dive and experimenting with it, and gonna let you know!
I think the problem is that the for loop isn't getting executed
The one in setInterval.

[uezo]

@Solidsoldier12 oh...
Checking what is set to WEBAudio.audioInstances will help you.

[Solidsoldier12]

@Solidsoldier12 oh...
Checking what is set to WEBAudio.audioInstances will help you.

Will do and let you know👍

[Solidsoldier12]

The WEBAudio.audioInstances has nothing in it when i console.log it, it just prints {}. I think that's why the for loop cannot find a key in it, can you suggest a fix or a workaround, your above workaround isn't working.
Unity version: 2021.

[sinansonlu]

Could it be a browser problem? I tested with Chrome and visemes was not working, when I tested with Microsoft Edge it worked.

[Morgan-6Freedom]

Still does not work in Unity 2022.
I don't know how to implement that workaround

[uezo]

Hi @Morgan-6Freedom,
Copy the code of that workaround and replace whole code of uLipSyncWebGL.jslib with it.

[Morgan-6Freedom]

Thank you. I will try that.
I change my Unity Version to 2020.3.45f and it's also broken :<
I will keep you in touch with the workaround !

[Morgan-6Freedom]

I used baked lipsync for my project (and it worked). Did not had the chance to test the workaround sorry.

[hansolGib]

im using unity 2021.3.21 and the workaround is working. but when I add another game object with audio source (I was thinking to use it as background music), then the avatar does lip-sync not only for the avatar's audio but also the background music. it seems inputString in SetAudioSampleData is mixed audio data having avatar's audio and background music. in the jslib, how can I separate them?
I only want the avatar to lip-sync with the avatar audio.

[Tongzhou-Yu]

My workaround.

mergeInto(LibraryManager.library, {
    InitWebGLuLipSync: function(targetObjectNamePtr, targetMethodNamePtr) {
        const targetObjectName = UTF8ToString(targetObjectNamePtr);
        const targetMethodName = UTF8ToString(targetMethodNamePtr);

        const outputHookNode = WEBAudio.audioContext.createScriptProcessor();
        outputHookNode.onaudioprocess = function (stream) {
            SendMessage(targetObjectName, targetMethodName, event.inputBuffer.getChannelData(0).join(','));
        };

        const connectAudioNodes = function (audioInstance) {
            if (audioInstance != null && audioInstance.hasOwnProperty("gain")) {
                // connect gain -> outputHookNode
                audioInstance.gain.connect(outputHookNode);
                // connect outputHookNode -> dest (dummy: no output data will go to dest)
                outputHookNode.connect(WEBAudio.audioContext.destination);
                console.log("Connected audio nodes successfully");
                return true;
            } else {
                return false;
            }
        };

        const jobId = setInterval(function() {
            for (var key in WEBAudio.audioInstances) {
                if (connectAudioNodes(WEBAudio.audioInstances[key])) {
                    // Continuously reconnect gain -> outputHookNode (they will be disconnected silently, I don't know why...)
                    setInterval(function() {
                        WEBAudio.audioInstances[key].gain.connect(outputHookNode);
                    }, 200);
                    clearInterval(jobId);
                    break;
                }
            }
        }, 200);
    },
});

I have tried in Unity 2021 and 2022, but neither works. https://yuuuuu.net/ChatYokAI_9/

[DBigagli]

Hi, I'm trying this on Unity 2022.2.18 with the workaround you mentioned but I'm having the same issue as @Gabri94x , where my avatar is keeping just the mouth open still when talking and closed when not. As him, in the editor it works fine so I believe it is configured properly. There could be some kind of differences in value scales that maybe push all blendshapes to the limit cap ?

[uezo]

@DBigagli, could you please verify the scale difference as you suggested?
Additionally, if possible, could you test this issue on different browsers and machines to ascertain whether it is environment-dependent?

[uezo]

I've just released v0.3 that supports Unity 2021.
https://github.com/uezo/uLipSyncWebGL/releases/tag/v0.3