V2.0.6 (#497)

* Fix transcription export issue

* Update for G2P and language modeling on Windows

* Bug fixes

README.md

@@ -31,15 +31,13 @@ in your environment of choice.
 If you'd like to install a local version of MFA or want to use the development set up, the easiest way is first create the dev environment from the yaml in the repo root directory:
 
 ```
-conda env create -n mfa-dev -f environment.yml  # Linux or mac
-conda env create -n mfa-dev -f environment_win.yml  #  Windows
+conda env create -n mfa-dev -f environment.yml
 ```
 
 Alternatively, the dependencies can be installed via:
 
 ```
-conda install -c conda-forge python=3.8 kaldi sox librosa biopython praatio tqdm requests colorama pyyaml  # All platforms
-conda install -c conda-forge pynini openfst baumwelch ngram  # Additional dependencies to install on Linux or Mac
+conda install -c conda-forge python=3.8 kaldi sox librosa biopython praatio tqdm requests colorama pyyaml pynini openfst baumwelch ngram
 ```
 
 MFA can be installed in develop mode via:

docs/source/changelog/changelog_2.0.rst

@@ -5,6 +5,16 @@
 2.0 Changelog
 *************
 
+2.0.6
+=====
+
+- Added G2P and language model training support to Windows
+- Fixed a bug where exporting trained models to the current working directory would not work :github_issue:`494`
+- Fixed a crash in exporting transcriptions to TextGrids
+- Added support for parsing out longer quoted strings :github_issue:`492`
+- Fix error message for files with no file extensions :github_issue:`495`
+- Fix PhoneSetType error for some models trained on earlier versions :github_issue:`496` and :github_issue:`484`
+
 2.0.5
 =====
 

docs/source/installation.rst

@@ -18,10 +18,6 @@ All platforms
 
 3. Ensure you're in the new environment created (:code:`conda activate aligner`)
 
-.. warning::
-
-   Windows native install is not fully supported in 2.0.  G2P functionality will be unavailable due to Pynini supporting only Linux and MacOS. To use G2P functionality on Windows, please set up the :xref:`wsl` and use the Bash console to continue the instructions.
-
 Upgrading from non-conda version
 ================================
 
@@ -45,7 +41,7 @@ If the Conda installation above does not work or the binaries don't work on your
 
 .. note::
 
-   You can also clone the conda forge feedstocks for `OpenFst <https://github.com/conda-forge/openfst-feedstock>`_, `SoX <https://github.com/conda-forge/sox-feedstock>`_, `Kaldi <https://github.com/conda-forge/kaldi-feedstock>`_, and `MFA <https://github.com/conda-forge/montreal-forced-aligner-feedstock>`_ and run them with `conda build <https://docs.conda.io/projects/conda-build/en/latest/>`_ to build for your specific system.
+   You can also clone the conda-forge feedstocks for `OpenFst <https://github.com/conda-forge/openfst-feedstock>`_, `SoX <https://github.com/conda-forge/sox-feedstock>`_, `Kaldi <https://github.com/conda-forge/kaldi-feedstock>`_, and `MFA <https://github.com/conda-forge/montreal-forced-aligner-feedstock>`_ and run them with `conda build <https://docs.conda.io/projects/conda-build/en/latest/>`_ to build for your specific system.
 
 Installing via pip
 ------------------
@@ -76,8 +72,7 @@ MFA uses a temporary directory for commands that can be specified in running com
 Supported functionality
 =======================
 
-Currently in the 2.0 beta, supported functionality is fragmented across platforms.  Native support for features
-is as follows.  Note that Windows can use Windows Subsystem for Linux to use the Linux version as necessary.
+As of version 2.0.6, all features are available on all platforms.  Prior to this version, G2P and language model training was unavailable on native Windows, but could be used with Windows Subsystem for Linux (WSL).
 
 .. list-table::
    :header-rows: 1
@@ -105,7 +100,7 @@ is as follows.  Note that Windows can use Windows Subsystem for Linux to use the
           <span class='rst-table-cell supported'>Yes</span>
      - .. raw:: html
 
-          <span class='rst-table-cell not-supported'>No</span>
+          <span class='rst-table-cell supported'>Yes</span>
      - .. raw:: html
 
           <span class='rst-table-cell supported'>Yes</span>
@@ -116,7 +111,7 @@ is as follows.  Note that Windows can use Windows Subsystem for Linux to use the
           <span class='rst-table-cell supported'>Yes</span>
      - .. raw:: html
 
-          <span class='rst-table-cell not-supported'>No</span>
+          <span class='rst-table-cell supported'>Yes</span>
      - .. raw:: html
 
           <span class='rst-table-cell supported'>Yes</span>
@@ -138,7 +133,7 @@ is as follows.  Note that Windows can use Windows Subsystem for Linux to use the
           <span class='rst-table-cell supported'>Yes</span>
      - .. raw:: html
 
-          <span class='rst-table-cell not-supported'>No</span>
+          <span class='rst-table-cell supported'>Yes</span>
      - .. raw:: html
 
           <span class='rst-table-cell supported'>Yes</span>

docs/source/user_guide/corpus_creation/training_lm.rst

@@ -6,9 +6,9 @@ Train a new language model  ``(mfa train_lm)``
 MFA has a utility function for training ARPA-format ngram :term:`language models`, as well as merging with a pre-existing model.
 
 
-.. warning::
+.. note::
 
-   Please note that this functionality is not available on Windows natively, however, you can install it using :xref:`wsl`, see :ref:`installation` for more details.
+   As of version 2.0.6, users on Windows can run this command natively without requiring :xref:`wsl`, see :ref:`installation` for more details.
 
 Command reference
 -----------------

docs/source/user_guide/dictionary.rst

@@ -15,6 +15,12 @@ Pronunciation dictionary format
 
 .. _text_normalization:
 
+.. warning::
+
+   As of 2.0.5, dictionaries have a firmer format of requiring tab-delimited columns (words, pronunciations, etc), and space-delimited pronunciations to avoid confusions in automatically interpreting dictionary format for phonesets that include numbers like X-SAMPA.
+
+   If your dictionary uses spaces as the delimiter between orthography and pronunciations, you can re-encode it with tabs in a text editor that has regex search and replace support. The regex pattern :code:`^(\S+)\s+` replaced with :code:`\1\t` or :code:`$1\t`, depending on the text editor in question, will replace the first whitespace in every line with a tab.
+
 Text normalization and dictionary lookup
 ========================================
 

docs/source/user_guide/workflows/dictionary_generating.rst

@@ -26,9 +26,9 @@ grapheme-to-phoneme mapping.
 
 See :ref:`dict_generating_example` for an example of how to use G2P functionality with a premade example.
 
-.. warning::
+.. note::
 
-   Please note that this functionality is not available on Windows natively, however, you can install it using :xref:`wsl`, see :ref:`installation` for more details.
+   As of version 2.0.6, users on Windows can run this command natively without requiring :xref:`wsl`, see :ref:`installation` for more details.
 
 Command reference
 -----------------

docs/source/user_guide/workflows/g2p_train.rst

@@ -17,9 +17,9 @@ The G2P model output will be a .zip file like the acoustic model generated from
 
 See :ref:`g2p_model_training_example` for an example of how to train a G2P model with a premade toy example.
 
-.. warning::
+.. note::
 
-   Please note that this functionality is not available on Windows natively, however, you can install it using :xref:`wsl`, see :ref:`installation` for more details.
+   As of version 2.0.6, users on Windows can run this command natively without requiring :xref:`wsl`, see :ref:`installation` for more details.
 
 .. _g2p_phonetisaurus_training:
 

montreal_forced_aligner/abc.py

@@ -32,7 +32,7 @@
 from sqlalchemy.orm import Session
 
 from montreal_forced_aligner.exceptions import KaldiProcessingError, MultiprocessingError
-from montreal_forced_aligner.helper import comma_join, load_configuration
+from montreal_forced_aligner.helper import comma_join, load_configuration, mfa_open
 
 if TYPE_CHECKING:
     from argparse import Namespace
@@ -615,7 +615,7 @@ def cleanup(self) -> None:
 
     def save_worker_config(self) -> None:
         """Export worker configuration to its working directory"""
-        with open(self.worker_config_path, "w") as f:
+        with mfa_open(self.worker_config_path, "w") as f:
             yaml.dump(self.configuration, f)
 
     def _validate_previous_configuration(self, conf: MetaDict) -> bool:

montreal_forced_aligner/acoustic_modeling/base.py

@@ -23,6 +23,7 @@
 from montreal_forced_aligner.corpus.features import FeatureConfigMixin
 from montreal_forced_aligner.db import Utterance
 from montreal_forced_aligner.exceptions import KaldiProcessingError
+from montreal_forced_aligner.helper import mfa_open
 from montreal_forced_aligner.models import AcousticModel
 from montreal_forced_aligner.utils import (
     KaldiProcessWorker,
@@ -265,7 +266,7 @@ def initialize_training(self) -> None:
         try:
             self._trainer_initialization()
         except Exception as e:
-            with open(dirty_path, "w"):
+            with mfa_open(dirty_path, "w"):
                 pass
             if isinstance(e, KaldiProcessingError):
 
@@ -413,7 +414,7 @@ def acc_stats(self) -> None:
                         pbar.update(num_utterances + errors)
 
         log_path = os.path.join(self.working_log_directory, f"update.{self.iteration}.log")
-        with open(log_path, "w") as log_file:
+        with mfa_open(log_path, "w") as log_file:
             acc_files = []
             for a in arguments:
                 acc_files.extend(a.acc_paths.values())
@@ -455,7 +456,7 @@ def acc_stats(self) -> None:
         average_logdet_sum = 0
         average_logdet_frames = 0
         for a in arguments:
-            with open(a.log_path, "r", encoding="utf8") as f:
+            with mfa_open(a.log_path, "r") as f:
                 for line in f:
                     m = avg_like_pattern.search(line)
                     if m:
@@ -542,15 +543,15 @@ def train(self) -> None:
                 self.train_iteration()
             self.finalize_training()
         except Exception as e:
-            with open(dirty_path, "w"):
+            with mfa_open(dirty_path, "w"):
                 pass
             if isinstance(e, KaldiProcessingError):
 
                 logger = logging.getLogger(self.identifier)
                 log_kaldi_errors(e.error_logs, logger)
                 e.update_log_file(logger)
             raise
-        with open(done_path, "w"):
+        with mfa_open(done_path, "w"):
             pass
         self.log_info("Training complete!")
         self.log_debug(f"Training took {time.time() - begin} seconds")

montreal_forced_aligner/acoustic_modeling/lda.py

@@ -15,6 +15,7 @@
 from montreal_forced_aligner.abc import KaldiFunction
 from montreal_forced_aligner.acoustic_modeling.triphone import TriphoneTrainer
 from montreal_forced_aligner.data import MfaArguments
+from montreal_forced_aligner.helper import mfa_open
 from montreal_forced_aligner.utils import (
     KaldiProcessWorker,
     Stopped,
@@ -93,7 +94,7 @@ def __init__(self, args: LdaAccStatsArguments):
 
     def _run(self) -> typing.Generator[typing.Tuple[int, int]]:
         """Run the function"""
-        with open(self.log_path, "w", encoding="utf8") as log_file:
+        with mfa_open(self.log_path, "w") as log_file:
             for dict_id in self.dictionaries:
                 ali_path = self.ali_paths[dict_id]
                 feature_string = self.feature_strings[dict_id]
@@ -177,7 +178,7 @@ def __init__(self, args: CalcLdaMlltArguments):
     def _run(self) -> typing.Generator[int]:
         """Run the function"""
         # Estimating MLLT
-        with open(self.log_path, "w", encoding="utf8") as log_file:
+        with mfa_open(self.log_path, "w") as log_file:
             for dict_id in self.dictionaries:
                 ali_path = self.ali_paths[dict_id]
                 feature_string = self.feature_strings[dict_id]
@@ -427,7 +428,7 @@ def lda_acc_stats(self) -> None:
         acc_list = []
         for x in arguments:
             acc_list.extend(x.acc_paths.values())
-        with open(log_path, "w", encoding="utf8") as log_file:
+        with mfa_open(log_path, "w") as log_file:
             est_lda_proc = subprocess.Popen(
                 [
                     thirdparty_binary("est-lda"),
@@ -527,7 +528,7 @@ def calc_lda_mllt(self) -> None:
         previous_mat_path = os.path.join(self.working_directory, "lda.mat")
         new_mat_path = os.path.join(self.working_directory, "lda_new.mat")
         composed_path = os.path.join(self.working_directory, "lda_composed.mat")
-        with open(log_path, "a", encoding="utf8") as log_file:
+        with mfa_open(log_path, "a") as log_file:
             macc_list = []
             for x in arguments:
                 macc_list.extend(x.macc_paths.values())

montreal_forced_aligner/acoustic_modeling/monophone.py

@@ -15,6 +15,7 @@
 from montreal_forced_aligner.acoustic_modeling.base import AcousticModelTrainingMixin
 from montreal_forced_aligner.data import MfaArguments
 from montreal_forced_aligner.exceptions import KaldiProcessingError
+from montreal_forced_aligner.helper import mfa_open
 from montreal_forced_aligner.utils import KaldiProcessWorker, Stopped, thirdparty_binary
 
 if typing.TYPE_CHECKING:
@@ -70,7 +71,7 @@ def __init__(self, args: MonoAlignEqualArguments):
 
     def _run(self) -> typing.Generator[typing.Tuple[int, int]]:
         """Run the function"""
-        with open(self.log_path, "w", encoding="utf8") as log_file:
+        with mfa_open(self.log_path, "w") as log_file:
             for dict_id in self.dictionaries:
                 fst_path = self.fst_ark_paths[dict_id]
                 ali_path = self.ali_ark_paths[dict_id]
@@ -277,7 +278,7 @@ def mono_align_equal(self) -> None:
                         pbar.update(num_utterances + errors)
 
         log_path = os.path.join(self.working_log_directory, "update.0.log")
-        with open(log_path, "w") as log_file:
+        with mfa_open(log_path, "w") as log_file:
             acc_files = []
             for x in arguments:
                 acc_files.extend(sorted(x.acc_paths.values()))
@@ -321,7 +322,7 @@ def _trainer_initialization(self) -> None:
         shared_phones_path = os.path.join(self.worker.phones_dir, "sets.int")
         init_log_path = os.path.join(self.working_log_directory, "init.log")
         temp_feats_path = os.path.join(self.working_directory, "temp_feats")
-        with open(init_log_path, "w") as log_file:
+        with mfa_open(init_log_path, "w") as log_file:
             subprocess.call(
                 [
                     thirdparty_binary("subset-feats"),