Issue relating to Piper Training and monotonic_align

[Gaming-Doggy]

I was trying to train a model based on ljspeech today, but ran into an error when trying to run my code.
(as I was using a multi-speaker dataset I used the --resume_from_single_speaker_checkpoint as recommended)

Code I attempted to run was:

python3 -m piper_train --dataset-dir ~/piper/my.training --accelerator 'gpu' --devices 1 --batch-size 32 --validation-split 0.0 --num-test-examples 0 --max_epochs 10000 --resume_from_single_speaker_checkpoint ~/piper/lj/ljspeech-2000.ckpt --checkpoint-epochs 1 --precision 32

The error that I received was:

File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/cat/piper/src/python/piper_train/__main__.py", line 10, in <module> from .vits.lightning import VitsModel File "/home/cat/piper/src/python/piper_train/vits/lightning.py", line 15, in <module> from .models import MultiPeriodDiscriminator, SynthesizerTrn File "/home/cat/piper/src/python/piper_train/vits/models.py", line 10, in <module> from . import attentions, commons, modules, monotonic_align File "/home/cat/piper/src/python/piper_train/vits/monotonic_align/__init__.py", line 4, in <module> from .monotonic_align.core import maximum_path_c ModuleNotFoundError: No module named 'piper_train.vits.monotonic_align.monotonic_align.core'

I tried finding a solution to this on my own, but wasn't able to find any documentation about this. I wasn't running this in a docker file simply because I have seen people provide mixed opinions on whether or not it is necessary. I had no issues earlier with the preprocessing.
There is the chance that maybe this issue could be related to me using WSL, but I am not sure.

Any help or suggestions would be much appreciated!

[systemofapwne]

Have you ran ./build_monotonic_align.sh before?

Regarding docker: This was a life saver for me. This project depends on outdated libraries and binaries that building a training environment on my linux system was out of question. The only help was building a Dockerfile that was based on an image, that includes the old software needed (with some caveats).

My two-staged Dockerfile looks like this

# Use the official PyTorch image as the base image
ARG BASE_IMAGE=nvcr.io/nvidia/pytorch:22.03-py3
ARG PIPER_VERSION=c0670df63daf07070c9be36b5c4bed270ad72383
ARG PYTHON_VERSION=3.10.13
ARG PYTHON_BIN=python3.10

########## Build python
FROM ${BASE_IMAGE} AS pythonbuilder
ARG PYTHON_VERSION

# Install dependencies needed for building Python
ENV DEBIAN_FRONTEND  noninteractive
RUN apt-get update && apt install -y \
    git build-essential zlib1g-dev libbz2-dev \
    liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
    libgdbm-dev liblzma-dev tk-dev lzma lzma-dev libgdbm-dev libffi-dev

RUN mkdir -pv /src && mkdir -pv /build
WORKDIR /src

RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
RUN tar zxvf Python-${PYTHON_VERSION}.tgz

WORKDIR /src/Python-${PYTHON_VERSION}
# Prefix is not only setting the destination where "make altinstall" puts the files, but also compiles in certain path such, that any program
# that will build against this python version, expect header files etc to be there -> We install into a clean /usr/local and then move the install files to /build
RUN ./configure --enable-optimizations --prefix=/usr/local
RUN make -j8

# Make clean destination which we then copy over to the actual container
RUN rm -rf /usr/local && mkdir -pv /usr/local
RUN make altinstall

RUN mv /usr/local/* /build




########## Build piper-train
FROM ${BASE_IMAGE}
ARG PIPER_VERSION
ARG PYTHON_BIN

# Copy python from pythonbuilder stage
RUN mkdir -pv /usr/local/
COPY --from=pythonbuilder /build/ /usr/local

# Set environment variables for Numba cache directory
ENV NUMBA_CACHE_DIR=.numba_cache

# Install dependencies and tools for training
ENV DEBIAN_FRONTEND  noninteractive
RUN apt update && apt install -y \
    git build-essential espeak-ng ffmpeg && \
    rm -rf /var/lib/apt/lists/*

# Prepare venv for piper
RUN /usr/local/bin/${PYTHON_BIN} -m venv /.venv
# Automatically activate the virtual environment when entering the container via 'docker exec -it <container name> bash'
RUN echo "source /.venv/bin/activate" >> /etc/bash.bashrc

# Prepare piper
RUN mkdir -pv /src
WORKDIR /src
# 
RUN git clone https://github.com/rhasspy/piper.git && cd piper && git checkout ${PIPER_VERSION}
WORKDIR /src/piper/src/python

# Upgrade pip
RUN source /.venv/bin/activate && pip install "pip<24"
# Install latest numpy 1.x and tochmetrics 0.x to avoid RTX 4000 issues (https://github.com/rhasspy/piper/issues/295)
RUN source /.venv/bin/activate && pip install "numpy<2" "torchmetrics<1"
# Install piper dependencies
RUN source /.venv/bin/activate && pip install pip wheel setuptools && \
    pip install -r requirements.txt
# Build piper-train
RUN source /.venv/bin/activate && pip install -e . && ./build_monotonic_align.sh

# Actual training directory: Mount your data folder in here
RUN mkdir -pv /training
WORKDIR /training

# Makes the container stay up
STOPSIGNAL SIGKILL
CMD tail -f /dev/null

and the container is built by docker compose up --build -d via this compose.yml

services:
  app:
    build: .
    container_name: training
    environment:
      - NVIDIA_VISIBLE_DEVICES=all                          # GPU Support
      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility    # GPU Support
    runtime: nvidia                                         # GPU Support
    volumes:
      - "./:/training:rw"
    shm_size: '4gb'

This needs of course the nvidia-container-toolkit to be installed on the system (which should be linux - All untested, if this works on wsl)

With this stack set up, I just entered the docker container via docker exec -it training bash and ran the training.

I hope this helps you along.