pymnn inference quality is unstable

[kmn1024]

平台(如果交叉编译请再附上交叉编译目标平台):

Platform(Include target platform as well if cross-compiling):

Model converter compilation, model conversion, and pymnn compilation were all done on device (==Orange Pi 5 Pro, using CPU, which is Arm A76+A55).

Github版本:

Github Version:

Tested on both 2.8.3 (2972fe7) and 2.8.4 (5895243)

编译方式:

Compiling Method

For conversion (hifigan audio generation model with randomness removed):

> cmake -DMNN_BUILD_CONVERTER=ON .. && make -j4
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at 3rd_party/protobuf/cmake/CMakeLists.txt:2 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- 
-- 3.19.0.0
-- Use Threadpool, forbid openmp
-- >>>>>>>>>>>>>
-- MNN BUILD INFO:
-- 	System: Linux
-- 	Processor: aarch64
-- 	Version: 2.8.3
-- 	Metal: OFF
-- 	OpenCL: OFF
-- 	OpenGL: OFF
-- 	Vulkan: OFF
-- 	ARM82: ON
-- 	oneDNN: OFF
-- 	TensorRT: OFF
-- 	CoreML: OFF
-- 	NNAPI: OFF
-- 	CUDA: OFF
-- 	OpenMP: OFF
-- 	BF16: OFF
-- 	ThreadPool: ON
-- 	Hidden: TRUE
-- 	Build Path: /home/orangepi/MNN_convert/build
-- 	CUDA PROFILE: OFF
-- WIN_USE_ASM: 
-- Enabling AArch64 Assemblies
-- Enable INT8 SDOT
-- Onnx: 
-- Configuring done (0.0s)
-- Generating done (0.2s)
-- Build files have been written to: /home/orangepi/MNN_convert/build
[  0%] Built target MNNCV
....

Conversion is successful, and testMNNFromOnnx.py is also successful (with randomness removed from the model code):

> python ../tools/script/testMNNFromOnnx.py /home/orangepi/platform/theplatform/resources/tts/decoder-det.onnx DEBUG
Dir exist
onnx/test.onnx
('asr', (1, 512, 190))
('F0_pred', (1, 380))
('N_pred', (1, 380))
('ref', (1, 128))
['output']
inputs:
asr
onnx/
F0_pred
onnx/
N_pred
onnx/
ref
onnx/
outputs:
onnx/output.txt (1, 1, 76000)
onnx/
The device support i8sdot:1, support fp16:1, support i8mm: 0
Start to Convert Other Model Format To MNN Model..., target version: 2.8
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:46: ONNX Model ir version: 8
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:47: ONNX Model opset version: 16
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /generator/f0_upsamp/Resize_output_0 has empty input, the index is 1
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /generator/m_source/l_sin_gen/Resize_output_0 has empty input, the index is 1
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /generator/m_source/l_sin_gen/Resize_1_output_0 has empty input, the index is 1
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /decode.3/upsample/Resize_output_0 has empty input, the index is 1
Start to Optimize the MNN Net...
inputTensors : [ ref, asr, F0_pred, N_pred, ]
outputTensors: [ output, ]
Converted Success!
Check convert result by onnx, thredhold is 0.01
asr
F0_pred
N_pred
ref
output: output
output: (1, 1, 76000, )
TEST_SUCCESS

The device support i8sdot:1, support fp16:1, support i8mm: 0
Start to Convert Other Model Format To MNN Model..., target version: 2.8
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:46: ONNX Model ir version: 8
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:47: ONNX Model opset version: 16
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /generator/f0_upsamp/Resize_output_0 has empty input, the index is 1
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /generator/m_source/l_sin_gen/Resize_output_0 has empty input, the index is 1
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /generator/m_source/l_sin_gen/Resize_1_output_0 has empty input, the index is 1
[15:38:44] /home/orangepi/MNN_convert/tools/converter/source/onnx/onnxConverter.cpp:142: Check it out ==> /decode.3/upsample/Resize_output_0 has empty input, the index is 1
Start to Optimize the MNN Net...
inputTensors : [ ref, asr, F0_pred, N_pred, ]
outputTensors: [ output, ]
Converted Success!
Check convert result by onnx, thredhold is 0.01
asr
F0_pred
N_pred
ref
output: output
output: (1, 1, 76000, )
TEST_SUCCESS

I converted the model using a simple command:

./MNNConvert -f ONNX --modelFile /home/orangepi/platform/theplatform/resources/tts/decoder-det.onnx --MNNModel /home/orangepi/platform/theplatform/resources/tts/decoder-det.mnn --bizCode biz --keepInputFormat true

pymnn was installed using https://github.com/alibaba/MNN/blob/master/pymnn/INSTALL.md. No problems (again, I tried both 2.8.3 and 2.8.4; the final result was the same).

USE_INTERNAL: False
USE_TRT: False
USE_CUDA: False
USE_OPENCL: False
USE_VULKAN: False
USE_RENDER: False
Building with python wheel with package name  MNN
['/home/orangepi/platform/MNN/pymnn_build', '/home/orangepi/platform/MNN/pymnn_build/tools/train', '/home/orangepi/platform/MNN/pymnn_build/tools/cv', '/home/orangepi/platform/MNN/pymnn_build/source/backend/tensorrt', '/home/orangepi/platform/MNN/pymnn_build/source/backend/cuda']
running install
/home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
writing MNN.egg-info/PKG-INFO
writing dependency_links to MNN.egg-info/dependency_links.txt
writing entry points to MNN.egg-info/entry_points.txt
writing requirements to MNN.egg-info/requires.txt
writing top-level names to MNN.egg-info/top_level.txt
reading manifest file 'MNN.egg-info/SOURCES.txt'
writing manifest file 'MNN.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-aarch64/egg
running install_lib
running build_py
copying MNN/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN
copying MNN/numpy/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/numpy
copying MNN/tools/mnnquant.py -> build/lib.linux-aarch64-cpython-312/MNN/tools
copying MNN/tools/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/tools
copying MNN/tools/mnnconvert.py -> build/lib.linux-aarch64-cpython-312/MNN/tools
copying MNN/tools/mnn.py -> build/lib.linux-aarch64-cpython-312/MNN/tools
copying MNN/nn/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/nn
copying MNN/data/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/data
copying MNN/expr/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/expr
copying MNN/optim/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/optim
copying MNN/cv/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/cv
copying MNN/numpy/random/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/numpy/random
copying MNN/numpy/linalg/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/numpy/linalg
copying MNN/tools/utils/__init__.py -> build/lib.linux-aarch64-cpython-312/MNN/tools/utils
copying MNN/tools/utils/getkey.py -> build/lib.linux-aarch64-cpython-312/MNN/tools/utils
copying MNN/tools/utils/log.py -> build/lib.linux-aarch64-cpython-312/MNN/tools/utils
running build_ext
creating build/bdist.linux-aarch64/egg
copying build/lib.linux-aarch64-cpython-312/_tools.cpython-312-aarch64-linux-gnu.so -> build/bdist.linux-aarch64/egg
creating build/bdist.linux-aarch64/egg/MNN
creating build/bdist.linux-aarch64/egg/MNN/numpy
copying build/lib.linux-aarch64-cpython-312/MNN/numpy/__init__.py -> build/bdist.linux-aarch64/egg/MNN/numpy
creating build/bdist.linux-aarch64/egg/MNN/numpy/random
copying build/lib.linux-aarch64-cpython-312/MNN/numpy/random/__init__.py -> build/bdist.linux-aarch64/egg/MNN/numpy/random
creating build/bdist.linux-aarch64/egg/MNN/numpy/linalg
copying build/lib.linux-aarch64-cpython-312/MNN/numpy/linalg/__init__.py -> build/bdist.linux-aarch64/egg/MNN/numpy/linalg
creating build/bdist.linux-aarch64/egg/MNN/tools
copying build/lib.linux-aarch64-cpython-312/MNN/tools/mnnquant.py -> build/bdist.linux-aarch64/egg/MNN/tools
copying build/lib.linux-aarch64-cpython-312/MNN/tools/__init__.py -> build/bdist.linux-aarch64/egg/MNN/tools
copying build/lib.linux-aarch64-cpython-312/MNN/tools/mnnconvert.py -> build/bdist.linux-aarch64/egg/MNN/tools
copying build/lib.linux-aarch64-cpython-312/MNN/tools/mnn.py -> build/bdist.linux-aarch64/egg/MNN/tools
creating build/bdist.linux-aarch64/egg/MNN/tools/utils
copying build/lib.linux-aarch64-cpython-312/MNN/tools/utils/__init__.py -> build/bdist.linux-aarch64/egg/MNN/tools/utils
copying build/lib.linux-aarch64-cpython-312/MNN/tools/utils/getkey.py -> build/bdist.linux-aarch64/egg/MNN/tools/utils
copying build/lib.linux-aarch64-cpython-312/MNN/tools/utils/log.py -> build/bdist.linux-aarch64/egg/MNN/tools/utils
creating build/bdist.linux-aarch64/egg/MNN/nn
copying build/lib.linux-aarch64-cpython-312/MNN/nn/__init__.py -> build/bdist.linux-aarch64/egg/MNN/nn
creating build/bdist.linux-aarch64/egg/MNN/data
copying build/lib.linux-aarch64-cpython-312/MNN/data/__init__.py -> build/bdist.linux-aarch64/egg/MNN/data
creating build/bdist.linux-aarch64/egg/MNN/expr
copying build/lib.linux-aarch64-cpython-312/MNN/expr/__init__.py -> build/bdist.linux-aarch64/egg/MNN/expr
copying build/lib.linux-aarch64-cpython-312/MNN/__init__.py -> build/bdist.linux-aarch64/egg/MNN
creating build/bdist.linux-aarch64/egg/MNN/optim
copying build/lib.linux-aarch64-cpython-312/MNN/optim/__init__.py -> build/bdist.linux-aarch64/egg/MNN/optim
creating build/bdist.linux-aarch64/egg/MNN/cv
copying build/lib.linux-aarch64-cpython-312/MNN/cv/__init__.py -> build/bdist.linux-aarch64/egg/MNN/cv
copying build/lib.linux-aarch64-cpython-312/_mnncengine.cpython-312-aarch64-linux-gnu.so -> build/bdist.linux-aarch64/egg
byte-compiling build/bdist.linux-aarch64/egg/MNN/numpy/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/numpy/random/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/numpy/linalg/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/mnnquant.py to mnnquant.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/mnnconvert.py to mnnconvert.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/mnn.py to mnn.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/utils/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/utils/getkey.py to getkey.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/tools/utils/log.py to log.cpython-312.pyc
build/bdist.linux-aarch64/egg/MNN/tools/utils/log.py:66: SyntaxWarning: invalid escape sequence '\S'
  res = os.popen("reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\ /v MachineGuid").read().strip().split(" ")[-1].lower()
byte-compiling build/bdist.linux-aarch64/egg/MNN/nn/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/data/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/expr/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/optim/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/MNN/cv/__init__.py to __init__.cpython-312.pyc
creating stub loader for _mnncengine.cpython-312-aarch64-linux-gnu.so
creating stub loader for _tools.cpython-312-aarch64-linux-gnu.so
byte-compiling build/bdist.linux-aarch64/egg/_mnncengine.py to _mnncengine.cpython-312.pyc
byte-compiling build/bdist.linux-aarch64/egg/_tools.py to _tools.cpython-312.pyc
creating build/bdist.linux-aarch64/egg/EGG-INFO
copying MNN.egg-info/PKG-INFO -> build/bdist.linux-aarch64/egg/EGG-INFO
copying MNN.egg-info/SOURCES.txt -> build/bdist.linux-aarch64/egg/EGG-INFO
copying MNN.egg-info/dependency_links.txt -> build/bdist.linux-aarch64/egg/EGG-INFO
copying MNN.egg-info/entry_points.txt -> build/bdist.linux-aarch64/egg/EGG-INFO
copying MNN.egg-info/requires.txt -> build/bdist.linux-aarch64/egg/EGG-INFO
copying MNN.egg-info/top_level.txt -> build/bdist.linux-aarch64/egg/EGG-INFO
writing build/bdist.linux-aarch64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__._mnncengine.cpython-312: module references __file__
__pycache__._tools.cpython-312: module references __file__
creating 'dist/MNN-2.8.4-py3.12-linux-aarch64.egg' and adding 'build/bdist.linux-aarch64/egg' to it
removing 'build/bdist.linux-aarch64/egg' (and everything under it)
Processing MNN-2.8.4-py3.12-linux-aarch64.egg
removing '/home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages/MNN-2.8.4-py3.12-linux-aarch64.egg' (and everything under it)
creating /home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages/MNN-2.8.4-py3.12-linux-aarch64.egg
Extracting MNN-2.8.4-py3.12-linux-aarch64.egg to /home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages
/home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages/MNN-2.8.4-py3.12-linux-aarch64.egg/MNN/tools/utils/log.py:66: SyntaxWarning: invalid escape sequence '\S'
  res = os.popen("reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\ /v MachineGuid").read().strip().split(" ")[-1].lower()
Adding MNN 2.8.4 to easy-install.pth file
Installing mnn script to /home/orangepi/miniconda3/envs/theplatform_rockwhisp/bin
Installing mnnconvert script to /home/orangepi/miniconda3/envs/theplatform_rockwhisp/bin
Installing mnnquant script to /home/orangepi/miniconda3/envs/theplatform_rockwhisp/bin

Installed /home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages/MNN-2.8.4-py3.12-linux-aarch64.egg
Processing dependencies for MNN==2.8.4
Searching for numpy==1.26.4
Best match: numpy 1.26.4
Adding numpy 1.26.4 to easy-install.pth file
Installing f2py script to /home/orangepi/miniconda3/envs/theplatform_rockwhisp/bin

Using /home/orangepi/miniconda3/envs/theplatform_rockwhisp/lib/python3.12/site-packages
Finished processing dependencies for MNN==2.8.4

Inference with pymnn, the quality of audio produced is equal to onnx output 90% of the time. The other 10% of the time, output quality is horrible. And the output quality is fixed after model initialization - if the first output is high quality, all outputs afterwards are high quality. If the first output is poor, all outputs afterwards are poor.

This is my model initialization and inference code:

class MnnWrapper(SubmodelWrapper):
    def __init__(self, resource_dir, model_args):
        model_basename, self.mnn_runtime_manager, self.inputs, outputs = model_args
        self.mnn_net = mnn_nn.load_module_from_file(
            os.path.join(resource_dir, model_basename), self.inputs, outputs, runtime_manager=self.mnn_runtime_manager)
        self.mnn_net.train(False)
        print(self.mnn_net.get_info())

    def infer(self, model_kwargs):
        assert len(model_kwargs) == len(self.inputs), (model_kwargs, self.inputs)
        mnn_args = [mnn_np.array(model_kwargs[input_name].tolist(), dtype=model_kwargs[input_name].dtype, copy=False) for input_name in self.inputs]
        mnn_outputs = self.mnn_net.forward(mnn_args)
        outputs = [np.copy(mnn_output.read()) for mnn_output in mnn_outputs]  # Copy seems necessary, otherwise gets corrupted.                                                                                                                                                                                                                                                                                                            
        return outputs

...
class TTSEngine():
    def __init__(self)
        self.mnn_runtime_manager = mnn_nn.create_runtime_manager(({
            'precision': 'high',
            'backend': 'CPU',
            },))

        self.decoder = MnnWrapper(('decoder.mnn', self.mnn_runtime_manager, [ 'ref', 'asr', 'F0_pred', 'N_pred', ], ['output']))
...
    def infer(self, text):
...
        decoder_sess_kwargs = {'ref': ref, 'asr': asr, 'F0_pred': F0_pred, 'N_pred': N_pred}
        return self.decoder.infer(decoder_sess_kwargs)[0]

[kmn1024]

Furthermore, it seems the problem only happens if the initialization+inference code is in a separate Process (production environment). In a single thread, single process test case, the problem seems to never occur.

[kmn1024]

It seems like adding dynamic=True to mnn_nn.load_module_from_file fixes the problem! However, it makes inference ~50% slower, even slower than ONNX =(

dynamic=True makes sense, since decoder input shape always changes, and output shape also always changes.

Is there anything I can do to improve inference speed?

[jxt1234]

Dynamic=True will load module as expr function and decrease inference speed. It's for mnn to train model.
Could you upload a project to reproduce the problem?

[jxt1234]

You can try not use raw numpy. Fully use MNN.numpy instead of numpy. The numpy data convert with MNN may cause error.

[kmn1024]

Thanks for your help, @jxt1234 !

Can you explain more about fully using MNN.numpy? The values of decoder_sess_kwargs needs to be numpy, since it is computed by numpy based code. Is there a safer way to convert numpy to MNN.numpy? I don't mind if the conversion is slow, since model inference is the biggest bottleneck.

Also, is there a safer way to convert back from MNN.numpy to numpy? Again, I don't mind if that is slow.

[kmn1024]

@jxt1234 I have uploaded a simple test to reproduce the issue:
https://mega.nz/file/pPVTGbBT#nCKr3OvKnXD8IiMHaGFG-4ZMW3455625qKxOYSRpiLA

Once you download and expand, there are 3 components:

decoder_iso_test.py
requirements.txt
resources/...

resources/... has the decoder MNN file, along with required inputs.

python decoder_iso_test.py will load the model, do inference, and then play the generated audio (note: you might have to adjust the pyaudio stream output_device_index; or you can save the wav file and check it with any audio player). If you run it 10 times, you are most likely going to hear low quality output at least once. You can also verify that adding dynamic=True to load_module_from_file makes the problem go away.

Please do let me know if you have problems running this test, and once again, thanks for helping!