Support for librosa needed

[jefflawr]

Hello,
I compiled some simple scripts that worked (Hello world, playing a clip with librosa, graphing a sine wave with matplotlib & numpy) but the current test to graph a spectrogram with matplotlib and librosa fails. The executable compiles fine but exits without producing the graph I get from the interpreter. This problem is dashing my hopes for a larger project -- any assistance welcome.
Thanks,
Jeff Lawrence

I'm using Windows 10 running under Parallels on a Macbook

python -m nuitka --version
1.7.10
Commercial: None
Python: 3.11.4 | packaged by Anaconda, Inc. | (main, Jul 5 2023, 13:47:18) [MSC v.1916 64 bit (AMD64)]
Flavor: Anaconda Python
Executable: C:\Users\jeffreylawrence\anaconda3\envs\accentprax5\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Version C compiler: ~\AppData\Local\Nuitka\Nuitka\Cache\downloads\gcc\x86_64\11.3.0-14.0.3-10.0.0-msvcrt-r3\mingw64\bin\gcc.exe (gcc).

Installation:
I used intelliJ IDE package installation feature to install nuitka and all other dependencies in a conda virtualenv. I installed Anaconda python 3.11 using Anaconda's installer package

The specific PyPI names and versions -- I'm sorry, but I don't know how to determine this

relevant output from > python -m pip freeze
...
librosa==0.10.1
...
matplotlib @ file:///C:/b/abs_49b2acwxd4/croot/matplotlib-suite_1679593486357/work
...
Nuitka==1.7.10
...
numpy==1.24.0
...
PyAudio==0.2.13
...
<<are these the PyPl names and versions referred to above? >>
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pyparsing @ file:///C:/ci_311/pyparsing_1678502182533/work
PyQt5==5.15.7
PyQt5-sip @ file:///C:/ci_311/pyqt-split_1676428895938/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work

Nuitka compile options (these worked fine with prior test examples):
python -m nuitka --enable-plugin=tk-inter --standalone --onefile --disable-console stft_spectralcentroids.py

Example code -- stft_spectralcentroids.py (uses any wave file clip "OneWord.wav"):
import librosa
import librosa.display
import numpy as np
import matplotlib.pyplot as plt

audio_file = "OneWord.wav" # a short voice-audio clip
graph_data, sample_rate = librosa.load(audio_file, sr=48000, mono=True)
plt.figure(figsize=(10, 5))
graph_ax = plt.gca()

spec, phase = librosa.magphase(librosa.stft(y=graph_data))
centroids = librosa.feature.spectral_centroid(S=spec)
times = librosa.times_like(centroids)
librosa.display.specshow(librosa.amplitude_to_db(spec, ref=np.max),
y_axis='log', x_axis='time', ax=graph_ax)
graph_ax.plot(times, centroids[0], label='Spectral centroid', color='w')

plt.show()

[kayhayen]

When trying to reproduce this, I didn't go with Anaconda, to not see its specific problem immediately. I noticed that librosa is nowadays employing a lazy_loader usage, much like seen already with skimage and sklearn and lately also huggingface_hub and I have been working to add enhancements that it can resolve those lazy loads at compile time automatically, and that kind of works for some things, but I need to figure out at least one more case.

I believe librosa worked before, but I am not sure of that now. Doesn't matter, this needs to be done. Hopefully that is what plagues you there. Unfortunately you were uncapable of producing an error output. You are probably not reading what the issue template said about disabled consoles, so I don't think there is anything I can know really about Anaconda yet. Might look at it once PyPI works.

[kayhayen]

So, librosa is using numba for parts, but rather unconditionally, and with decorators that will crash. I managed to do the lazy_loader automatic handling of the provided .pyi lazy loads, to be that, and be included as necessary, that will be super useful for a lot of scientific packages. For numba, we cannot support it, but we need to provide a stub, such that it doesn't explode dependencies with code that will never be able to work, and instead of using their JIT on a remote host, maybe compile it locally, but there seem to be more complex forms as well. But for numpy.testing we already provide a stub like that, and maybe librosa is usable in large parts without it, since that's only parts of librosa.core.audo that seem to use it.

Not sure when I get to add that. I will try for 1.9, since that will be simple to do and might remove a lot of bloat for cases of hard importing numba. What Nuitka recommends, namely to just not include it, will just give an ImportError during startup when librosa is importing the load function.

[jefflawr]

Hello Kay,
I appreciate your time and attention to this issue.
I know you like to keep chat to a minimum here, but want to thank you sincerely for your comments. I have to look up many of the concepts in your explanations and am learning a lot about program structure.
I'll try not including numba for now and if that doesn't work will wait patiently for 1.9.
Best regards,
Jeff

[kayhayen]

So, current factory will do librosa lazy loader just fine, and actually pretty efficient. I have yet to check if now skimage and sklearn work as well.

However, the numba angle is unresolved. That stub module replacement for numba would probably do a good job and get a large subset of librosa to work, hard to tell now. For now it will be more of a priority to get that mechanism that I added there stabilized, and applied to e.g. the huggingface_hub variation, but I will eventually get to that numba work as well.

[kayhayen]

The lazy loader changes are on current develop. Not sure if I can get the numba solution, I am feeling the need to prepare for 3.12 release readiness.

[kayhayen]

Part of the 1.9 release I just made. It seems using librosa pulls in numba and the warning, but programs will still work, since it is not used most of the time.

[wl-junlin]

hello, I still cant sucecess with librosa

• env: ubuntu20.04, python 3.11.5, Nuitka '1.9.7', librosa 0.10.1

• i use command
  nuitka3 --standalone --onefile tests/test.py
  to compile

• below is the compile message

Nuitka-Options:INFO: Used command line options: --standalone --onefile tests/test.py
Nuitka:INFO: Starting Python compilation with Nuitka '1.9.7' on Python '3.11' commercial grade 'not installed'.
Nuitka-Plugins:WARNING: anti-bloat: Undesirable import of 'numba' (intending to avoid 'numba') in 'librosa.core.spectrum' (at
Nuitka-Plugins:WARNING: '/home/me/miniconda3/lib/python3.11/site-packages/librosa/core/spectrum.py:12') encountered. It may slow down
Nuitka-Plugins:WARNING: compilation.
Nuitka-Plugins:WARNING: Complex topic! More information can be found at https://nuitka.net/info/unwanted-module.html
Nuitka-Plugins:WARNING: options-nanny: Using module 'numba' (version 0.58.1) with incomplete support due untrue condition 'not use_numba and
Nuitka-Plugins:WARNING: standalone': Numba is not yet working with Nuitka standalone, try to use '--noinclude-numba-mode', but it may fail.
Nuitka-Plugins:INFO: anti-bloat: Not including '_bisect' automatically in order to avoid bloat, but this may cause: may slow down by using fallback implementation.
Nuitka-Plugins:INFO: anti-bloat: Not including '_json' automatically in order to avoid bloat, but this may cause: may slow down by using fallback implementation.
Nuitka-Plugins:INFO: anti-bloat: Not including 'matplotlib' automatically in order to avoid bloat, but this may cause: plotting will lack matplotlib.
Nuitka:INFO: Completed Python level compilation and optimization.
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Running data composer tool for optimal constant value handling.
Nuitka:INFO: Running C compilation via Scons.
Nuitka-Scons:INFO: Backend C compiler: gcc (gcc 11).
Nuitka-Scons:INFO: Backend linking program with 907 files (no progress information available for this stage).
Nuitka-Scons:WARNING: You are not using ccache, re-compilation of identical code will be slower than necessary. Use your OS package manager to
Nuitka-Scons:WARNING: install it.
Nuitka-Plugins:INFO: data-files: Included data file 'librosa/core/intervals.msgpack' due to package data for 'librosa.core.intervals'.
Nuitka-Plugins:INFO: dll-files: Found 1 file DLLs from llvmlite.binding.ffi installation.
Nuitka-Postprocessing:INFO: Creating single file from dist folder, this may take a while.
Nuitka-Onefile:INFO: Running bootstrap binary compilation via Scons.
Nuitka-Scons:INFO: Onefile C compiler: gcc (gcc 11).
Nuitka-Scons:INFO: Onefile linking program with 1 files (no progress information available for this stage).
Nuitka-Scons:WARNING: You are not using ccache, re-compilation of identical code will be slower than necessary. Use your OS package manager to
Nuitka-Scons:WARNING: install it.
Nuitka-Onefile:INFO: Using compression for onefile payload.
Nuitka-Onefile:INFO: Onefile payload compression ratio (20.99%) size 413489570 to 86787222.
Nuitka-Onefile:INFO: Keeping onefile build directory 'test.onefile-build'.
Nuitka:INFO: Keeping dist folder 'test.dist' for inspection, no need to use it.
Nuitka:INFO: Keeping build directory 'test.build'.
Nuitka:INFO: Successfully created 'test.bin'.

• the test.py was written as

  import sys
  from loguru import logger
  import librosa
  logger.remove()  # remove default log handler
  logger.add(sys.stderr, level="INFO")
  
  if __name__ == "__main__":
      audio_file = "tests/4sec.wav"
      sample_rate = 16000
      print('process audio file ', audio_file)
      waveform, _ = librosa.core.load(audio_file, sr=sample_rate)
      s = librosa.stft(waveform)
      logger.info("stft computed, shape: {}, dtype: {}".format(s.shape, s.dtype))
  

• after compile suceede, i ran test.bin, got error message:

process audio file tests/4sec.wav
Traceback (most recent call last):
File "/tmp/onefile_285719_1705472870_807632/test.py", line 20, in
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "/tmp/onefile_285719_1705472870_807632/librosa/core/audio.py", line 1135, in
File "/tmp/onefile_285719_1705472870_807632/numba/stencils/stencil.py", line 816, in stencil
File "/tmp/onefile_285719_1705472870_807632/numba/stencils/stencil.py", line 825, in decorated
File "/tmp/onefile_285719_1705472870_807632/numba/core/compiler.py", line 334, in run_frontend
File "/tmp/onefile_285719_1705472870_807632/numba/core/interpreter.py", line 1363, in interpret
File "/tmp/onefile_285719_1705472870_807632/numba/core/byteflow.py", line 118, in run
File "/tmp/onefile_285719_1705472870_807632/numba/core/byteflow.py", line 281, in _is_implicit_new_block
File "/tmp/onefile_285719_1705472870_807632/numba/core/byteflow.py", line 1571, in get_inst
File "/tmp/onefile_285719_1705472870_807632/numba/core/bytecode.py", line 269, in getitem
KeyError: 2

[kayhayen]

Posting below closed issues is a surefire way to get ignored @wl-junlin

[kayhayen]

New issues with pull issue template are expected in case a problem resurfaces.