Rhubarb executable can generate cues from audio file but Capture does not generate cues inside Blender

[davebs]

Hello, I am having a bit of a strange issue. I am using Blender 3.5

The problem is that when I load an audio file into rhubarb in blender and run Capture, it goes through progress and then at the end nothing happens and no error message is returned. Cues are not generated.

I tried troubleshooting by verifying that rhubarb.exe can generate cues from the same audio file, and it does. I can see the desired cues print out to the console by running rhubarb -r pocketSphinx PATH_TO_WAV_FILE

Is there anything I can do to better troubleshoot or does anyone have any ideas for what I might try instead? Thank you!

[davebs]

I notice that when I click Capture button inside the plugin/inside blender, it says "Capture (Running)" and runs, but when it finishes, it never says "complete" or anything, it just says "Capture (Running)" still. This leads me to think it may be an issue with me using a longer audio file (about a minute of audio) and may be related to multithreading or the background process not finishing properly.

Also tried running as administrator thinking it might be a permissions issue somehow but no dice.

So I guess I need to find the code that handles the finishing task.

[davebs]

I put a print statement that prints 'TRYING SOMETHING THAT MIGHT BLOCK' and 'FINISHED SOMETHING THAT MIGHT BLOCK' as so:

    def read_process_stderr_line(self) -> None:
        assert self.was_started
        if self.has_finished:
            return

        # (stdout, stderr) = self.process.communicate(timeout=timeout)
        print('START POLL')
        exit_code = self.process.poll()
        print('END POLL')
        if exit_code is not None:  # Process has finished
            # Collect the output just in case
            self.collect_output_sync(ignore_timeout_error=True)
            self.last_exit_code = exit_code
            if exit_code != 0:
                raise RuntimeError(f"Rhubarb binary exited with a non-zero exit code {exit_code}")
            return

        try:
            # Rhubarb binary is reporting progress on the stderr. Read next line.
            # This would eventually block.
            print('TRYING SOMETHING THAT MIGHT BLOCK')
            n = next(self.process.stderr)  # type: ignore
            self.stderr += n
            print('FINISHED SOMETHING THAT MIGHT BLOCK')
        except StopIteration:
            log.debug("EOF reached while reading the stderr")  # Process has just terminated

When progress gets to 100%, it prints TRYING SOMETHING THAT MIGHT BLOCK and then does not print the line about FINISHED. So I think the problem is that this section blocks:

            n = next(self.process.stderr)  # type: ignore
            self.stderr += n

How to fix this is an issue I am currently struggling with.

[davebs]

Ok, I fixed my own issue. I added a check to see if progress had reached 100% and if so, prevent it from reading further from self.process.stderr and instead runs self.process.stdout.readlines() to then update self.stdout. This works on Blender 3.6, but fwiw I was having the original trouble on Blender 3.4 and 3.5 as well. So this doesn't seem to be a common issue people are experiencing. It feels pretty hacky but it gets the job done for now.

In my case, I just replace rhubarb_command.py with the following changed file. I don't know that it won't break other things or will work on other people's computer, so I am including my solution here but not submitting a pull request:

import json
import logging
import os
import pathlib
import platform
import re
import traceback
from collections import defaultdict
from queue import Empty, SimpleQueue
from subprocess import PIPE, Popen, TimeoutExpired
from threading import Event, Thread
from time import sleep
from typing import Any, Dict, List, Optional

from rhubarb_lipsync.rhubarb.mouth_shape_data import MouthCue

log = logging.getLogger(__name__)


class RhubarbParser:
    version_info_rx = re.compile(r"version\s+(?P<ver>\d+\.\d+\.\d+)")

    LOG_LEVELS_MAP: dict[str, int] = defaultdict(lambda: logging.TRACE)  # type: ignore
    LOG_LEVELS_MAP.update(
        {
            "Fatal": logging.CRITICAL,  # TODO Verify
            "Error": logging.ERROR,
            "Info": logging.INFO,
        }
    )

    @staticmethod
    def parse_version_info(stdout: str) -> str:
        m = re.search(RhubarbParser.version_info_rx, stdout)
        if m is None:
            return ""
        return m.groupdict()["ver"]

    @staticmethod
    def parse_status_infos(stderr: str) -> list[Dict]:
        """Parses the one-line json(s) produced by rhubarb binary.
        Each report is a json on separate line"""
        if not stderr:
            return []
        return [RhubarbParser.parse_status_info_line(l) for l in stderr.splitlines()]

    @staticmethod
    def parse_status_info_line(stderr_line: str) -> Dict[str, Any]:
        if not stderr_line:
            return {}
        try:
            return json.loads(stderr_line)
            # { "type":"start", "file":"1.ogg", "log":{"level":"Info","message": "Application startup." } }
            # { "type": "failure", "reason": ...
            # { "type": "progress", "value": 0.17,
        except json.JSONDecodeError:
            log.exception(f"Failed to parse status line '{stderr_line[:100]}'")
            return {}

    @staticmethod
    def parse_lipsync_json(stdout: str) -> List[Dict]:
        """Parses the main lipsync output json. Return only the list of mouthCues"""
        # { "metadata": { "soundFile": "1.ogg", "duration": 5.68},
        #   "mouthCues": [ { "start": 0.00, "end": 0.28, "value": "X" } ... ] }

        if not stdout:
            return []
        try:
            j = json.loads(stdout)
            return j["mouthCues"]
        except json.JSONDecodeError:
            log.exception(f"Failed to parse main rhubarb output json. '{stdout[:200]}...'")
            return []

    @staticmethod
    def lipsync_json2MouthCues(cues_json: List[Dict]) -> List[MouthCue]:
        return [MouthCue.of_json(c_json) for c_json in cues_json]


class RhubarbCommandWrapper:
    """Wraps low level operations related to the lipsync executable."""

    thread_wait_timeout = 5

    def __init__(self, executable_path: pathlib.Path, recognizer="pocketSphinx", extended=True, extra_args=[]) -> None:
        self.executable_path = executable_path
        self.recognizer = recognizer
        self.use_extended = extended
        self.process: Optional[Popen] = None
        self.stdout = ""
        self.stderr = ""
        self.last_exit_code: Optional[int] = None
        self.extra_args = extra_args
        self.IS_FINISHED = False

    @staticmethod
    def executable_default_filename() -> str:
        return "rhubarb.exe" if platform.system() == "Windows" else "rhubarb"

    def config_errors(self) -> Optional[str]:
        if not self.executable_path:
            return "Configure the Rhubarb lipsync executable file path in the addon preferences. "

        if not self.executable_path.exists():
            return f"The '{self.executable_path}' doesn't exist."
        if not self.executable_path.is_file():
            return f"The '{self.executable_path}' is not a valid file."
        # Zip doesn't maintain file flags, set as executable
        os.chmod(self.executable_path, 0o744)
        return None

    def build_lipsync_args(self, input_file: str, dialog_file: Optional[str] = None) -> list[str]:
        dialog = ["--dialogFile", dialog_file] if dialog_file else []
        extended = ["--extendedShapes", "GHX"] if self.use_extended else []
        return [
            str(self.executable_path),
            "-f",
            "json",
            "--machineReadable",
            *extended,
            "-r",
            self.recognizer,
            *dialog,
            input_file,
        ]

    def build_version_args(self) -> list[str]:
        return [str(self.executable_path), "--version"]

    def open_process(self, cmd_args: List[str]) -> None:
        assert not self.was_started
        assert not self.config_errors(), self.config_errors()
        self.stdout = ""
        self.stderr = ""
        self.last_exit_code = None
        log.info(f"Starting process\n{cmd_args}")
        # universal_newlines forces text mode
        self.process = Popen(self.extra_args + cmd_args, stdout=PIPE, stderr=PIPE, universal_newlines=True)

    def close_process(self) -> None:
        if self.was_started:
            log.debug(f"Terminating the process {self.process}")
            self.process.terminate()
            self.process.wait(RhubarbCommandWrapper.thread_wait_timeout)
            # Consume any reminding output, this would also close the process io streams
            self.process.communicate(timeout=5)
            log.debug("Process terminated")
        self.process = None

    def get_version(self) -> str:
        """Execute `lipsync --version` to get the current version of the binary. Synchroinous call."""
        self.close_process()
        args = self.build_version_args()
        self.open_process(args)
        self.collect_output_sync(ignore_timeout_error=False)
        return RhubarbParser.parse_version_info(self.stdout)

    def log_status_line(self, log_json: dict) -> None:
        # {'log': {'level': 'Info', 'message': 'Msg'}}]
        if not log_json or "log" not in log_json:
            return  # Not log key included in the progress line
        level = log_json["log"]["level"]
        msg = log_json["log"]["message"]

        log.log(RhubarbParser.LOG_LEVELS_MAP[level], f"Rhubarb: {msg}")

    def lipsync_start(self, input_file: str, dialog_file: Optional[str] = None) -> None:
        """Start the main lipsync command. Process runs in background"""
        self.close_process()
        args = self.build_lipsync_args(input_file, dialog_file)
        self.open_process(args)

    def lipsync_check_progress(self) -> int | None:
        """Reads the stderr of the lipsync command where the progress and status in being reported.
        Note this call blocks until there is status update available on stderr.
        The rhubarb binary provides the status update few times per seconds typically.
        """
        assert self.was_started, "Process not started. Can't check progress"
        if self.has_finished:
            self.close_process()
            return 100
        self.stderr = ""
        self.read_process_stderr_line()
        if not self.stderr:
            return None
        status_lines = RhubarbParser.parse_status_infos(self.stderr)
        if not status_lines:
            return None
        for s in status_lines:
            self.log_status_line(s)
        by_type = {j["type"]: j for j in status_lines if j}
        if "failure" in by_type:
            raise RuntimeError(f"Rhubarb binary failed:\n{by_type['failure']['reason']}")
        if "progress" not in by_type:
            return None
        v = by_type["progress"]["value"]
        if v*100 == 100:
            self.IS_FINISHED = True
        return int(v * 100)

    @property
    def was_started(self) -> bool:
        """Whether the process has been triggered already. Might be still running running or have finished"""
        return self.process is not None

    @property
    def has_finished(self) -> bool:
        """
        Whether the process has finished. Either sucessfully or with an error.
        When True the process is not running and the last_out and the last_error are complete
        """
        # if not self.was_started:
        #    return False
        return self.last_exit_code is not None

    @property
    def is_running(self) -> bool:
        return self.was_started and not self.has_finished

    def get_lipsync_output_json(self) -> list[dict]:
        """Json - parsed output of the lipsync capture process"""
        assert self.has_finished, "Output is not available since the process has not finisehd"
        return RhubarbParser.parse_lipsync_json(self.stdout)

    def get_lipsync_output_cues(self) -> list[MouthCue]:
        """Json - parsed output of the lipsync capture process"""
        json = self.get_lipsync_output_json()
        return RhubarbParser.lipsync_json2MouthCues(json)

    def collect_output_sync(self, ignore_timeout_error=True, timeout=1) -> None:
        """
        Waits (with a timeout) for the process to finish. Then collects its std out and std error
        """
        assert self.was_started
        try:
            (stdout, stderr) = self.process.communicate(timeout=1)  # Consume any reminding output
            self.stderr += stderr
            self.stdout += stdout
        except TimeoutExpired:
            log.warn("Timed out while waiting for process to finalize outputs")
            if not ignore_timeout_error:
                raise
        self.close_process()

    def read_process_stderr_line(self) -> None:
        assert self.was_started
        if self.has_finished:
            return

        # (stdout, stderr) = self.process.communicate(timeout=timeout)
        exit_code = self.process.poll()
        if exit_code is not None:  # Process has finished
            # Collect the output just in case
            self.collect_output_sync(ignore_timeout_error=True)
            self.last_exit_code = exit_code
            if exit_code != 0:
                raise RuntimeError(f"Rhubarb binary exited with a non-zero exit code {exit_code}")
            return

        try:
            # Rhubarb binary is reporting progress on the stderr. Read next line.
            # This would eventually block.
            if self.IS_FINISHED != True:
                n = next(self.process.stderr)  # type: ignore
                self.stderr += n
            else:
                o = self.process.stdout.readlines()
                self.stdout += ''.join(o)
        except StopIteration:
            log.debug("EOF reached while reading the stderr")  # Process has just terminated


class RhubarbCommandAsyncJob:
    """Additional wrapper over the RhubarbCommandWrapper which handles asynchronious progress-updates."""

    thread_wait_timeout = 5

    def __init__(self, cmd: RhubarbCommandWrapper) -> None:
        assert cmd
        self.cmd = cmd
        self.thread: Optional[Thread] = None
        self.queue: SimpleQueue[tuple[str, Any]] = SimpleQueue()
        self.last_progress = 0
        self.last_exception: Optional[Exception] = None
        self.last_cues: list[MouthCue] = []
        self.stop_event = Event()

    def _thread_run(self) -> None:
        """Runs on a separate threads, pushing progress message via Q"""
        log.trace("Entered progress check thread")  # type: ignore
        while True:
            try:
                if self.cmd.has_finished:
                    log.trace("Process finished")  # type: ignore
                    break
                if self.stop_event.is_set():
                    log.trace("Stop event received")  # type: ignore
                    break  # Cancelled
                progress = self.cmd.lipsync_check_progress()

                if progress is None:
                    sleep(0.1)
                else:
                    self.last_progress = progress
                    self.queue.put(("PROGRESS", progress))

            except Exception as e:
                log.error(f"Unexpected error while checking the progress status {e}")
                self.last_exception = e
                traceback.print_exc()
                self.queue.put(("EXCEPTION", e))
                raise
        log.debug("Progress check thread exit")

    def join_thread(self) -> None:
        if not self.thread:
            return

        log.debug(f"Joining thread {self.thread}")
        try:
            self.thread.join(RhubarbCommandWrapper.thread_wait_timeout)
            if self.thread.is_alive():
                log.error(f"Failed to join the thread after waiting for {RhubarbCommandWrapper.thread_wait_timeout} seconds.")
        except:
            log.error("Failed to join the thread")
            traceback.print_exc()
        finally:
            self.queue = SimpleQueue()
            self.thread = None

    def lipsync_check_progress_async(self) -> int | None:
        if self.cmd.has_finished:  # Finished, do some auto-cleanup
            self.join_thread()
            self.cmd.close_process()
            return 100
        if not self.thread:
            log.debug("Creating status-check thread")
            self.stop_event.clear()
            self.thread = Thread(target=self._thread_run, name="StatusCheck", daemon=True)
            self.thread.start()

        try:
            (msg, obj) = self.queue.get_nowait()
            log.trace(f"Received {msg}={obj}")  # type: ignore
            if msg == 'PROGRESS':
                return int(obj)
            if msg == 'EXCEPTION':
                raise obj  # Propagate exception from the thread
            assert False, f"Received unknown message {msg}"
        except Empty:
            return None

    def cancel(self) -> None:
        log.info("Cancel request. Stopping the process and the status thread.")
        self.stop_event.set()
        self.join_thread()
        self.cmd.close_process()

    def get_lipsync_output_cues(self) -> list[MouthCue]:
        if self.last_cues:  # Cached
            return self.last_cues
        if not self.cmd.has_finished:  # Still in progress (rhubarb bin can't deliver partial results)
            return []
        if not self.cmd.stdout:  # No output, probably failed
            return []
        self.last_cues = self.cmd.get_lipsync_output_cues()  # Cache the result
        return self.last_cues

    @property
    def failed(self) -> bool:
        if self.last_exception:
            return True
        if self.cmd.has_finished:
            if self.cmd.last_exit_code != 0:
                return True
        return False

    @property
    def status(self) -> str:
        if not self.cmd.was_started and not self.cmd.has_finished:
            # Not started yet. Or cancelled
            return "Failed" if self.failed else "Stopped"
        if self.cmd.has_finished:
            return "Done" if self.get_lipsync_output_cues() else "No data"
        return "Running"

[okikiola12]

same here, after loading the audio file in the addon and when I click the capture, it works fine but nothing happens. Not binding to the rig or the mesh.

[Premik]

Thanks @davebs for checking this and finding where the issue is.

For me this happens when the output json is bigger that the system pipe buffer size. On my box it is 64kb which means any audio longer ~4 minutes blocks the rhubar executable from terminating. I guess this buffer could be smaller on other systems so could break for much shorter clips

I've put the sdtin reader on a separate thread to be safe.

Would be great if you could test the v1.0.2 release if it still fixes your original issue.

@okikiola12 Just to be sure, did you also get an empty list after the Capture process finished?

If so, please try the latest version above. Note after you get the cues captured you still need to proceed to the second tab to do the actual mapping and baking

[okikiola12]

I just downloaded the new
update version which is 1.0.2 now, and the cue remains empty even after the capture has been initiated

it still persists. probably there is a need to look at the buffer size max allow value again. My files are over 10 min.

i also tested this version with a file that is less than 2min and the cue remained empty still

[davebs]

I will test the updated version later, I notice @okikiola12 seems to be experiencing the same issue as I did where the capture is still listed as Running so isn't returning from execution. In my hacky fix, I remember even after I got it to stop the process, I still needed to do self.process.stdout.readlines() to get the JSON which then gets read and put into the cue list. So basically, even if stderr gets fully read, the json from stdout wasn't being read out so I wasn't seeing anything appear.

try:
            # Rhubarb binary is reporting progress on the stderr. Read next line.
            # This would eventually block.
            if self.IS_FINISHED != True:
                n = next(self.process.stderr)  # type: ignore
                self.stderr += n
            else:
                o = self.process.stdout.readlines() # <---- this line
                self.stdout += ''.join(o)
        except StopIteration:
            log.debug("EOF reached while reading the stderr")  # Process has just terminated

I'm busy on some other things today but I will give the updates a try soon.

[ACMOIDRE]

Guys can make small tutorial, i have did everything as official repo but still struggling to understand i'm i doing everything correct or wrong

[ACMOIDRE]

Sorry,
I'm in hurry.
thank you @Premik Continuing development of addon

[Premik]

@davebs Indeed when the stdout is not fully read the process hangs. But that is what should be fixed in the v1.0.2. At least I hope :)

@okikiola12
I'm no longer able to reproduced the hanging myself. I've even tried 11 minutes audio, using windows (wine), Blender 3.5.1 with no luck. To narrow this down, can you please trying:

• Some very short audio. Just few seconds. For example the files used by the automatic test.
• Download the new v1.0.3 version which has improved the logging a bit.
  • In the plug Preference set the log level to trace
  • Open the system console using the Windows main menu/Toggle System Console (https://blender.stackexchange.com/questions/145890/where-is-the-system-console)
  • Run the capture and check the logs
  • Or/and set the log file in the Preferences so the logs should get saved to a file and attach the log.txt here:

This is how the trace looks from a normal run:

INFO:rhubarb_lipsync.rhubarb.rhubarb_command:Starting process
['/home/premik/.config/blender/3.6/scripts/addons/rhubarb_lipsync/bin/rhubarb', '-f', 'json', '--machineReadable', '--extendedShapes', 'GHX', '-r', 'pocketSphinx', '/home/premik/Desktop/Blender/../../../../wrk/dev/rhubarb-lipsync/tests/data/en_male_electricity.ogg']
DEBUG:rhubarb_lipsync.blender.rhubarb_operators:Operator execute
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:Creating reader threads
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Entered _stdout_thread_run  reader thread
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Entered _stderr_thread_run  reader thread
INFO:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Application startup. Input file: /home/premik/Desktop/Blender/../../../../wrk/dev/rhubarb-lipsync/tests/data/en_male_electricity.ogg.
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 0%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 1%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 2%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 3%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 4%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 5%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 6%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=0
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=1
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=2
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=3
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=4
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=5
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=6
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 68%
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=69
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Progress: 100%
INFO:rhubarb_lipsync.rhubarb.rhubarb_command:Rhubarb: Application terminating normally.
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:Consumed stdout. Read 16 lines.
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Received PROGRESS=100
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Process finished
TRACE:rhubarb_lipsync.rhubarb.rhubarb_command:Process finished
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:_stderr_thread_run thread exit
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:_stdout_thread_run thread exit
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:Joining thread <Thread(StdOut, stopped daemon 139734449909760)>
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:Joining thread <Thread(StdError-StatusCheck, stopped daemon 139734422646784)>
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:Terminating the process <Popen: returncode: 0 args: ['/home/premik/.config/blender/3.6/scripts/addon...>
DEBUG:rhubarb_lipsync.rhubarb.rhubarb_command:Process terminated
INFO:rhubarb_lipsync.blender.rhubarb_operators:Operator finished
INFO:rhubarb_lipsync.blender.rhubarb_operators:Added 8 cues to the list
Info: Capture @0 Done

@ACMOIDRE Is the quick start on the main page of any help? I know it is not that exciting like a video but there are some screecasts too..

[okikiola12]

A good job well done,

However, this are the issues that I encountered.

I downloaded the version 1.0.3 ,

after installations, I load the earlier wave file again. but it still persists.
however, when I loaded the test file that you attached there, the cue was generated. so I close and restart my blender, and I loaded my WAV file of 11 min again,

this time around the cue was captured and it was loaded successfully..

and,

now, when click on the mapping. i was only able to see two rigs here.

and i generated them like that as shown below.


which was successful with no error and could see them in my NLA..

Info Created new NLA track RLPS Tra.txt

however, when I dragged them, the my rig did not move...

again find the attached text..

finally, one more thing,
the log file is disabled even when the trace is selected..

[Premik]

Thanks for testing this out and the very detailed description.

after installations, I load the earlier wave file again. but it still persists.

That is expected. Nothing was changed in the v1.0.3 . Only some extra logging was added.

however, when I loaded the test file that you attached there, the cue was generated.

That is interesting. It looks like there is some concurrency/racing issue if it sometimes works and sometimes doesn't for the same input. I'll try to run more tests to see if I would be able to reproduce this.

now, when click on the mapping. i was only able to see two rigs here.
however, when I dragged them, the my rig did not move...

I see I really need to create some video with a real rig and test this out.
I would also like to add some Faceit integration in the future. From what I remember Faceit can only generate ShapeKey "Action" for each cue. But ShapeKey Action support is not yet implemented. It is visible in the ui but it does nothing

But Faceit also have single long Action faceit_shape_action where each cue-shape is on a different frame (with 10 frames spaces). This format is also not supported as is. But you can:

• Open this faceit_shape_action in DopeSheet/Action editor.
• For each cue type (A,B,C, ... X) find the best match. For instance the A and X cues are at frame 1. The E could be frame 20. You can click the ? for a guidance:

• Create a new Action (give it some nice name) for each cue type and copy+paste the keyframes from the faceit action to this new one (to frame 1).
• Populate the mapping drop downs with those new actions.

finally, one more thing, the log file is disabled even when the trace is selected..

These log file is additional option. The log level increases amount of messages logged to system console and also to the file. But file is only used if there is a file specified (blank in your screenshot). You would need to put something like c:\mylog.txt to the log file and it would save logs there.

[okikiola12]

in blender 3.4 and Rhubarb 3...., I have used Postlib in the past instead of shape-keys with Faceit. which is, because the Rhubarb 3.... works well with Faceit in version 3.4 of Blender.
I understand that with the recent update, poseLib has been relocated to the sidebar. thus making it possible for graphical icons.

this update makes Rhubarb 3... incompatible with version 3.5 of Blender anymore.

however, I am trying to see if this new Rhubarb version 1.0.3 can be manipulated in a certain way.

That is, I want to create a pose asset as seen here.

and rename it according to the preferred mouth shape. which I have done.
I noticed that I was able to see the created pose here in the mapping..

so creating different mouth poses and selecting them here. the desired mouth pose can be generated with the cue.
I hope it will work..

one more thing, I tend to use Faceit rather than Rigify because Faceit offers more deformative bone than Rigify. Hence the reason though.

many thanks for this addon though...

[okikiola12]

Hi, i have been able to create multiple poses as seen here,

I bake it and it was successful, however when drag the slide on my timeline, nothing happened.
and when I check my NLA, I can see the bake cues,

however, when I check my dop sheet, there are no baked cues.

[Premik]

Hi @okikiola12
sorry for taking so long to reply and thanks for the very detailed description.

I understand that with the recent update, poseLib has been relocated to the sidebar. thus making it possible for graphical icons.

My understanding is the old poseLib has been completely removed and replaced by the inbuilt Pose Library plugin. And now any Action on armature-bones marked as an Asset is considered a pose.

This lipsync plugin works with any Action. Whether it is marked as an asset or not. Anyway I guess you've already figured this out :)

one more thing, I tend to use Faceit rather than Rigify because Faceit offers more deformative bone than Rigify. Hence the reason though.

I'd like to add better Faceit support next. I've realized the Action Clip in the NLA has frame start-end. So it could "clip" a sub-range from the long faceit_shape_action for each cue-shape. Avoiding the need to have each cue-shape in a separate Action.

bake it and it was successful, however when drag the slide on my timeline, nothing happened.
and when I check my NLA, I can see the bake cues,

You have an active Action (orange) in your FaceRig object which overrides the two underlying NLA tracks:

You need to unlink it:

however, when I check my dop sheet, there are no baked cues.

To see the keyframes in the Dope sheet you need to further bake the NLA tracks to a new Action. This is standard Blender functionality. There is a screenshot in the main readme.

I've also added a video tutorial. My animation skills are horrible but I hope it would still help with the basic concepts..

[okikiola12]

Well done,
I was able to use both the asset editor and post assets when creating poses.
both work for me. but I will be going with post assets because it is easy.

the NLA was generated for both. when the slider slides. the mouth was deformed which is good, even when using faceit it rig.

with these, I have no problem using it with faceit.

and I was able to bake the two generated NLA as seen here.

this can be considered a successful release.

in the upcoming version, it would be nice if the last part could be automated. that is, from the NLA down to Baking.

Thanks, with these, I can say that I am ready for blender 4 . 0

[Premik]

Good stuff. Glad you figure it all out..

in the upcoming version it will be nice if the last part can be automated. that from the NLA down to Baking.

I think this could be done eventually.
Btw did you have to tweak the generated animation anyhow? Usually some tweaking is required and I always thought tweaking the NLA strips is generally easier compared to moving the keyframes in Dope sheet or similar.

[okikiola12]

yes, I normally tweak the keyframe that is because, I work with wave audio that is mixed with many voices.
so, it is easier to cut out part of it in the timeline, because. the keyframes are easily seen at some specific frames.

and the rate at which the mouth and so other deformation bone open can be tweaks.
so by dragging the generated keyframe on the timeline or in dope sheets make it more easier and convenient .

once again, thanks for this addon.
Good job.

[okikiola12]

@Premik also looks at the baking aspect too. after baking, the blend file runs to gigabytes.
unlike the previous version when I compare both. that one was 300+ MB even with a keyframe baked, while this was 2.67 GB with just one baked keyframe.

one more thing, when trying to bring another the cue to the NLA an error was thrown as seen below.

which is because I work with more than 1 character in blend scene.

what I did was to rename them myself.

and, I had do the other character in another blend file before. renaming the RLSP track before copying it.
that solves the conflict issues.

Again, I believe just like I said a few days ago, automating the baking action down to the dope or timeline keyframe might solve the conflict issues.

finally, try to update your code to blender 4. although nothing breaks apart from those areas highlighted.
because, when installing the addon. it say this addon was written in a previous version

Perfects

working well on blender 4 as it was in blender 3.6

[Premik]

yes, I normally tweak the keyframe that is because, I work with wave audio that is mixed with many voices.
so, it is easier to cut out part of it in the timeline, because. the keyframes are easily seen at some specific frames.

I see. Thanks for the explanation.

and the rate at which the mouth and so other deformation bone open can be tweaks.

Good to know. Note you can also tweak the Playback Scale and Blend In/Out of a NLA strip to achieve similar result (I believe)..

also looks at the baking aspect too. after baking, the blend file runs to gigabytes.

I've tried with 10 minutes ogg audio and couldn't get the .blend file grow over ~40MB. But I haven't tried with faceit plugin yet. Here by baking you mean the final bake of the two NLA track to a single Action, right?

unlike the previous version when I compare both. that one was 300+ MB even with a keyframe baked, while this was 2.67 GB with just one baked keyframe.

Can you specify the "previous version"? Do you mean like the blender_rhubarb_lipsync_ng say v1.0.1 or the very old plugin (without the _ng)?

What can help reduce the file size:

• Remove the Capture object from the RLPS: Sound setup. But this is usually in MB range even for long audio.
• When baking the final Action tick the Clean Curves:

It is also possible to cleanup already baked keyframes but it was not that efficient in reducing the final size from what I've tried.

one more thing, when trying to bring another the cue to the NLA an error was thrown as seen below.
and, I had do the other character in another blend file before. renaming the RLSP track before copying it. that solves the conflict issues.

NLA doesn't allow strips on the same track to overlap. Usually one can simply pressed the Remove strips button. Provided you have already baked the NLA tracks to a new single Action you don't need the strips anymore. You can then refurbish the Track pair for another capture:

Or just create new pair of tracks in the Strip placement settings by unselecting the existing pair (x icon) and creating new using the other button:

There is also a chance something is wrong with the track selection. NLA tracks are not so called ID objects so they can't be directly referenced and saved using the Blender API. So the reference is done by the track name, index and Object.

finally, try to update your code to blender 4. although nothing breaks apart from those areas highlighted. because, when installing the addon. it say this addon was written in a previous version

I'll update the bl_info structure in the new release. Although when I tried to install the plugin into fresh Blender v4.0.1 it didn't say a thing.

[okikiola12]

ok, I will have to look at the Playback Scale and Blend In/Out again.
my audio file is formatted to wave. i will try to convert it to Ogg and see. again, the initial baking, "i.e NLA baking" doesn't cause the blend file to become suddenly big with faceit addons.

it only becomes way bigger in the final bake of the two NLA tracks into a single Action.

I am referring to the very old addon for reference sake.

ok, I will try these two steps here.

**Remove the Capture object from the RLPS: Sound setup. But this is usually in the MB range even for long audio.
When baking the final Action tick the Clean Curves**

I hope it fixes the memory increase.

The Remove strips button was clicked on the new character even when the NLA had never been baked on the new character.
still, there were conflict issues.

yes, I think. combining the NLA track to a single one should fix it I believe.

yeah, I tried that too.
that is to create a new pair of tracks in the Strip placement settings by unselecting the existing pair.
still, there were conflict issues.

yeah, please have a look at the track selection.

An error popped up in the first version. blender 4.0. however, I have tried to install it again in the latest version 4.01.

Thanks for this add-on. I believe all these are little fixes.

[okikiola12]

perfect now, the blend file reduced drastically from 2.3 GB to 230 MB.
now, those areas that were highlighted earlier have been resolved now.

that is, the baked keyframe are now seen in the dope sheet and timeline,
graph editor now works perfectly.

i believe the conflict issues will be resolved as well since NLA track can be deleted. haven't tried it with multiple characters but I believe it will be resolved.

just one area remaining, that is if it is possible though.
the keyframe baking time is over 4 hours which is high when compared to the previous version "the NG version" using the same audio file. it was under 2 hours. again, I hope I am not asking for too much though. and finally, any permission to release a video guide. that way, people who intend to use pose assets and Faceit addon can learn from the tutorial as well.

All I can say is a good job well done.

[Tibodelanvale1]

Can help please?
I install addon, add sound and have this if press capture:

I try 3.4 | 3.6 | 4.0 blender versions, all have same error.
File not long, try ogg, mp3 and wav files, dont help.

[Premik]

Hi @Tibodelanvale1

I assume the files you tried have different format (stereo/mono). But to be sure please also try to convert the file via the plugin. As it sets the format to the one known to work with the rhubarb binary:

• In the Preferences tick the Always show the convert buttons.
• In the Sound setup and cues capture panel press the new Convert to Wav button. This will convert to a wav.
• Try to run capture again with the wav if it makes any difference.

You can also inspect the Additional info section in the Sound setup panel to see more details about the sound file. You can also try with any files from the tests data which are known to work.

If still no luck enable more verbose logs and save them to a file:

• In the Preference Change the Log level to TRACE and
• Populate the Log file. Set it somewhere like D:\Download\log.txt.
• Run the Capture and inspect the log.txt if it contains more details.
• There should be also the exact command which failed. Like ['blender/4.0/scripts/addons/rhubarb_lipsync/bin/rhubarb', '-f', 'json', '--machineReadable', '--extendedShapes', 'GHX', '-r', 'pocketSphinx', 'sound.ogg'] so you can try to run it from cmd to get some clue why it fails.
• Attach the log.txt here

[Tibodelanvale1]

Yes it help!
now i have sound, but cant make mapping(

I makeaction target, have pose library (try with and without it) but dont see any example what i can add. i try change different buttons what to show and what not to show (under the word mapping) but nothing gives results

[Premik]

You have only-shapekeys Actions filter enabled:

So untick it if you have enabled it accidentaly.
Or more likely that means you have a Mesh selected (where this filter is activated by default) instead of an Armature. Typically the mapping should be be created on the Armature Object (unless you want to animate using shape-keys).

Edit: I see now you've already tried these filter buttons. If you can't list any of the Actions that is strange indeed. Are those action coming from a library perhaps?

[Tibodelanvale1]

I'm trying with a random skeleton, I created it, but it doesn't work either. Do you have a Discord? It will be easier to figure it out there, and then post an answer for people if someone comes across something similar. (my is maestromans)

[Tibodelanvale1]

@Premik it work with blender 4.0 ? Maybe i need only some speshial verion without else addons? (have no idea.. )

[Premik]

Seems you do all right. So have no clue at this stage why it doesn't show any Actions. It should work with any recent Blender (3.5 to 4.3) and I don't think there could be any inference with others plugins.

Could you attach your test .blend file?

I have Discord premik12345 but not using it actively. Would have been better to create a new Github ticket too.

One more thing you could try: Open the Scripting Console and type list(D.actions). It should list the Actions in the projects:

@Tibodelanvale1 might be also worth trying the old v1.0.3 since these drop-downs are new.

[okikiola12]

I will have to test the latest version myself. from the last release. there haven't been any issues from my end here. apart from the time taken to bake the NLA to the keyframe. there haven't been one.

I think I have gotten used to the NLA editing before the final baking. it seems I prefer it now, which is understandable.

again I hope the long-hour baking issues will be sorted now in this new version 1.1..
I would test it myself and revert back.

@Tibodelanvale1 you can always try this site for converting your audio to wave files.

https://audio.online-convert.com/convert/mpeg-to-wav

[Tibodelanvale1]

@Premik
this all action edits what i have and console sent them too
Here file:
https://dropmefiles.com/kfn55
I delete almost all to make it smaller (still dont work)
Use blender 4.0.0

[okikiola12]

well, from my own end here.
the addon works perfectly fine with no issues at all.

@Tibodelanvale1 what type of rig are using rigify or some paid one ?

[Tibodelanvale1]

@okikiola12
i try rigify and just few bones, both done work

[Premik]

@ Tibodelanvale1

Thanks for the file. It was really helpful.
I found there was a bug in the plugin which makes the action listing fail when there is any Action without a key-frame. The bpy.data.actions['D-'], bpy.data.actions['E-']] Actions are blank.
Unfortunatelly Blender only prints such errors to the system console (which is bit tricky to get displayed) and doesn't show it anywhere else.
Either way I've improved the error handling and fixed this bug. Please try out the v1.1.1 at your best convenience.

Btw did you found what was the original issue with the ogg file? Was it stereo vs mono or something else? There is already a check for bitrates >16k which are not supported. But if there is more to be check I could add another one.

[Premik]

Hi @okikiola12
sorry for not addressing your issues earlier..

i believe the conflict issues will be resolved as well since NLA track can be deleted. haven't tried it with multiple characters but I believe it will be resolved.

Would be good to have some steps to reproduce this eventually if a fix is desired. I assume it this not a blocking issue anyway.

the keyframe baking time is over 4 hours which is high when compared to the previous version "the NG version" using the same audio file. it was under 2 hours. again,

That is pretty insane :-) Those 4 hours is the file NLA->Action bake, right? That operation is something which comes from Blender and it is probably already in C++. But I guess it is only single-threaded (like many things in Blender).
Not much I could do about it.
The baking in the old pre-NG version was done very differently. In theory it is possible to add more baking methods, not just the bake-to-NLA. But it is not trivial so for me not worth the effort. Perhaps if it was like 4 minutes vs 4 hours :-)

any permission to release a video guide. that way, people who intend to use pose assets and Faceit addon can learn from the tutorial as well.

Yes, please go ahead. You don't need my permission to do that if I understand you correctly. Btw I'm (slowly) working on direct Faceit integration so it would be much easier to use then. This new Action drop-down was a prerequisite.

I think I have gotten used to the NLA editing before the final baking. it seems I prefer it now, which is understandable.

Well done. Any bigger project from Blender Studio I opened used this Nobody Likes Animator. So I assumed it is what everybody use.. :)

again I hope the long-hour baking issues will be sorted now in this new version 1.1..

Have to crash your expectations since there were no changes in that area...

you can always try this site for converting your audio to wave files.

Blender is actually pretty capable in converting between various audio formats. But the convert buttons are only shows when the plugin thinks the sound format is not compatible (unless forced in the preferences).

[Tibodelanvale1]

@Premik hi!
Glad i can little help you :3
Now all work! I have no idea what wrong with sound, but i press button what you say and sound work good too.

[okikiola12]

alright,
sorry for the late reply.
been very busy.
check this link for the tutorial. https://odysee.com/rhubarbs:6f1f98652c15cc490a455ed9673bf72f0257f56e
on the c++ issues. I guess we will just have to take it like that.
I have resolved to bake my NLA in parts. that way, the long annoying time is prevented as a result.

[okikiola12]

@Premik The auto blend in & out seems to be removed in v 1.2.1.

[Premik]

Hi @okikiola12
I'm currently reworking the stip-placement to simplify it and make it produce better results out-of-the box. So probably I removed that checkbox already. Did realize anybody would want that auto-blend enabled.

Either way you can simply enable Auto Blend on the strip properties after the baking is done:

• Select all the strips in the NLA (a key).
• 2x Shift-click any of the already selected strip again to make it active. This should show the side panel.
• Alt+click the Auto Blend In/Out to distribute the change to all the selected strips.

[okikiola12]

@Premik enables that makes the mouth to open and close naturally. without it. lip will be synced like a robot (i.e. opening and closing of the mouth in an unrealistic way).

[Premik]

@okikiola12 I see what you mean. The new strip-placing/baking method should produce better results out-of-the box with less tweaks. Still wip:

NewBakingPreview2.mp4

[okikiola12]

@Premik yh, the video you shared seam to be corrupted. please share again

@okikiola12 I see what you mean. The new strip-placing/baking method should produce better results out-of-the box with less tweaks. Still wip:
NewBakingPreview2.mp4

@Premik yh, the video you shared seam to be corrupted. please share again

[Premik]

@okikiola12 Seems FF doesn't like the video format but Chrome is ok. This one is with slightly different parameters:

2.mp4

It is just a short preview anyway. I'll created another one with proper description later.

I think the main reason for this robotic look is when the animation "freeze", ie. doesn't change on several consequential frames. The new baking methods address this better.

[okikiola12]

@okikiola12 Seems FF doesn't like the video format but Chrome is ok. This one is with slightly different parameters:
2.mp4

It is just a short preview anyway. I'll created another one with proper description later.

I think the main reason for this robotic look is when the animation "freeze", ie. doesn't change on several consequential frames. The new baking methods address this better.

looking at my attached two pictures here, I think I understand your point.

but without the auto blend in/out the mouth will open and close quickly after each frame range. but with auto blend, it keeps the mouth and also mixes the frame. e.g, if A is open wide and B is O shape, instead of having the mouth close and open to O. the mouth will only close a little automatically and open in O shape again.

[Premik]

I think I see your point now. I've already added the auto-blend flag back in the last code-base (not released yet).

I didn't realize blending of two adjacent strips only make sense when they overlap. Otherwise the strip actually mix/blend with some default pose. And that makes the mouth closing and opening in between cues.
Have to rethink that new baking method considering this..

[okikiola12]

I think I see your point now. I've already added the auto-blend flag back in the last code-base (not released yet).

I didn't realize blending of two adjacent strips only make sense when they overlap. Otherwise the strip actually mix/blend with some default pose. And that makes the mouth closing and opening in between cues. Have to rethink that new baking method considering this..

yeah,