tts: migrate to torch source builds; torchaudio: init at 0.13.0

[mweinelt]

Description of changes

I'm working on migrating tts and the trainer package away from the binary packages that won't be cached on hydra due to their size.

To that end I'm trying to package up torchaudio.

There are also some other cleanups in there and a cherry-pick of the latest torch bump from #202769

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[mweinelt]

The build currently fails because it can't find some header file:

torchaudio> FAILED: torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o 
torchaudio> /nix/store/dq0xwmsk1g0i2ayg6pb7y87na2knzylh-gcc-wrapper-11.3.0/bin/g++ -DINCLUDE_KALDI -DINCLUDE_SOX -DUSE_C10D_GLOO -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -Dlibtorchaudio_EXPORTS -I/build/source -isystem /nix/store/hsy4nya1yb9sn8c40863x8igck4z9bzi-python3.10-torch-1.13.0/lib/python3.10/site-packages/torch/include -isystem /nix/store/hsy4nya1yb9sn8c40863x8igck4z9bzi-python3.10-torch-1.13.0/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -Wall -D_GLIBCXX_USE_CXX11_ABI=1 -O3 -DNDEBUG -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -fopenmp -std=gnu++14 -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o.d -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o -c /build/source/torchaudio/csrc/kaldi.cpp
torchaudio> /build/source/torchaudio/csrc/kaldi.cpp:2:10: fatal error: feat/pitch-functions.h: No such file or directory
torchaudio>     2 | #include "feat/pitch-functions.h"
torchaudio>       |          ^~~~~~~~~~~~~~~~~~~~~~~~
torchaudio> compilation terminated.

The header file is part of the ptyhon3.pkgs.torchaudio.kaldi.dev output and kaldi' is in buildInputs, but since kaldi does not provide pkg-config support 🤷

include/kaldi/feat/pitch-functions.h

Tried

• -DCMAKE_INCLUDE_PATH=${lib.getDev kaldi'}/include/kaldi
• -DCMAKE_CXX_FLAGS=-I${lib.getDev kaldi'}/include/kaldi,
• NIX_CFLAGS_COMPILE = "-I${lib.getDev kaldi'}/include/kaldi";

without success.

[r-burns]

I was able to get a bit further by adding the include path for openfst as well:

--- a/pkgs/development/python-modules/torchaudio/default.nix
+++ b/pkgs/development/python-modules/torchaudio/default.nix
@@ -116,6 +116,11 @@ buildPythonPackage {
     kaldi = kaldi';
   };
 
+  NIX_CFLAGS_COMPILE = [
+    "-I${lib.getDev kaldi'}/include/kaldi"
+    "-I${lib.getDev kaldi'}/include/openfst"
+  ];
+
   meta = with lib; {
     changelog = "https://github.com/pytorch/audio/releases/tag/v${version}";
     description = "Data manipulation and transformation for audio signal processing, powered by PyTorch";

But then I encountered this error:

/build/source/torchaudio/csrc/kaldi.cpp:20:53: error: no matching function for call to ‘kaldi::VectorBase<float>::VectorBase(const at::Tensor&)’
   20 |   ::kaldi::VectorBase<::kaldi::BaseFloat> input(wave);
      |                                                     ^
In file included from /nix/store/g29h1jj0vi14585iv7nb35vbcwqj05vs-kaldi-2022-03-06-dev/include/kaldi/base/kaldi-error.h:35,
                 from /nix/store/g29h1jj0vi14585iv7nb35vbcwqj05vs-kaldi-2022-03-06-dev/include/kaldi/feat/pitch-functions.h:32,
                 from /build/source/torchaudio/csrc/kaldi.cpp:2:
/nix/store/g29h1jj0vi14585iv7nb35vbcwqj05vs-kaldi-2022-03-06-dev/include/kaldi/matrix/kaldi-vector.h:398:34: note: candidate: ‘kaldi::VectorBase<Real>::VectorBase(const kaldi::VectorBase<Real>&) [with Real = float]’
  398 |   KALDI_DISALLOW_COPY_AND_ASSIGN(VectorBase);
      |                                  ^~~~~~~~~~

[mweinelt]

Fails to link libsox right now, but progresses way further. Got some help by andir :)

torchaudio> /nix/store/92h8cksyz9gycda22dgbvvj2ksm01ca4-binutils-2.39/bin/ld: cannot find -llibsox: No such file or directory
torchaudio> collect2: error: ld returned 1 exit status

[Mic92]

If you run ninja || NIX_DEBUG=1 ninja -j1 than you might be able to get more verbose output from the compiler and binutils wrapper when linking libsox.so