linux voxcpm2的webui关于torch.compile的问题

[hotdogarea]

linux 环境下 开启torch.compile= true时 启动webui推理时报错，关闭compile可以正常运行但3080ti下，rtf在一点几

• To create a public link, set share=True in launch().
  Traceback (most recent call last):
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/gradio/queueing.py", line 785, in process_events
  response = await route_utils.call_process_api(
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/gradio/route_utils.py", line 358, in call_process_api
  output = await app.get_blocks().process_api(
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/gradio/blocks.py", line 2172, in process_api
  result = await self.call_function(
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/gradio/blocks.py", line 1634, in call_function
  prediction = await anyio.to_thread.run_sync( # type: ignore
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/anyio/to_thread.py", line 63, in run_sync
  return await get_async_backend().run_sync_in_worker_thread(
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
  return await future
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 986, in run
  result = context.run(func, *args)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/gradio/utils.py", line 1059, in wrapper
  response = f(*args, **kwargs)
  File "/root/VoxCPM/appV2.py", line 616, in _generate
  sr, wav_np = demo.generate_tts_audio(
  File "/root/VoxCPM/appV2.py", line 436, in generate_tts_audio
  return self._generate_tts_audio_main_thread(
  File "/root/VoxCPM/appV2.py", line 587, in _generate_tts_audio_main_thread
  raise error_queue.get()
  File "/root/VoxCPM/appV2.py", line 570, in inference_worker
  result = self._generate_tts_audio_worker_thread(
  File "/root/VoxCPM/appV2.py", line 502, in _generate_tts_audio_worker_thread
  wav = current_model.generate(**generate_kwargs)
  File "/root/VoxCPM/src/voxcpm/core.py", line 155, in generate
  return next(self._generate(*args, streaming=False, **kwargs))
  File "/root/VoxCPM/src/voxcpm/core.py", line 276, in _generate
  for wav, _, _ in generate_result:
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
  response = gen.send(None)
  File "/root/VoxCPM/src/voxcpm/model/voxcpm2.py", line 927, in _generate_with_prompt_cache
  latent_pred, pred_audio_feat = next(inference_result)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
  response = gen.send(None)
  File "/root/VoxCPM/src/voxcpm/model/voxcpm2.py", line 992, in _inference
  feat_embed = self.feat_encoder(feat) # [b, t, h_feat]
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
  return fn(*args, **kwargs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  File "/root/VoxCPM/src/voxcpm/modules/locenc/local_encoder.py", line 17, in forward
  def forward(self, x):
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
  return fn(*args, **kwargs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1100, in forward
  return compiled_fn(full_args)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 321, in runtime_wrapper
  all_outs = call_func_at_runtime_with_args(
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 124, in call_func_at_runtime_with_args
  out = normalize_as_list(f(args))
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 667, in inner_fn
  outs = compiled_fn(args)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 488, in wrapper
  return compiled_fn(runtime_args)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1478, in call
  return self.current_callable(inputs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1008, in run
  return compiled_fn(new_inputs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 398, in deferred_cudagraphify
  fn, out = cudagraphify(model, inputs, new_static_input_idxs, *args, **kwargs)
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 420, in cudagraphify
  manager = get_container(device_index).get_tree_manager()
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 341, in get_container
  container_dict = get_obj(local, "tree_manager_containers")
  File "/root/miniconda3/envs/voxcpm/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 336, in get_obj
  assert torch._C._is_key_in_tls(attr_name)
  AssertionError

[a710128]

可以参考这个文档中的内容：https://voxcpm.readthedocs.io/zh-cn/latest/faq.html#torch-compile-errors-on-first-run
从之前的经验来看，这一般是某些 PyTorch、Triton 与 einops 的版本组合与 torch.compile 不兼容。

如果需要高效的部署推理可以尝试 https://voxcpm.readthedocs.io/zh-cn/latest/deployment/nanovllm_voxcpm.html

[Xiaodu233]

Windows下使用python-3.11，torch-2.11.0+cu128，triton_windows-3.6.0.post26，einops-0.8.2，在编译后推理几次后，推理内容仅包含Control Instruction和Target Text，未启用其他模式和功能，两个参数为默认，（几次尝试后似乎是在第五次）就会一直输出几乎一致的错误日志：

报错内容（点击展开）

Traceback (most recent call last):
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\gradio\queueing.py", line 785, in process_events
   response = await route_utils.call_process_api(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\gradio\route_utils.py", line 358, in call_process_api
   output = await app.get_blocks().process_api(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\gradio\blocks.py", line 2172, in process_api
   result = await self.call_function(
            ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\gradio\blocks.py", line 1634, in call_function
   prediction = await anyio.to_thread.run_sync(  # type: ignore
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\anyio\to_thread.py", line 63, in run_sync
   return await get_async_backend().run_sync_in_worker_thread(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 2518, in run_sync_in_worker_thread
   return await future
          ^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 1002, in run
   result = context.run(func, *args)
            ^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\gradio\utils.py", line 1059, in wrapper
   response = f(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\app.py", line 362, in _generate
   sr, wav_np = demo.generate_tts_audio(
                ^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\app.py", line 340, in generate_tts_audio
   wav = current_model.generate(**generate_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\src\voxcpm\core.py", line 155, in generate
   return next(self._generate(*args, streaming=False, **kwargs))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\src\voxcpm\core.py", line 276, in _generate
   for wav, _, _ in generate_result:
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 40, in generator_context
   response = gen.send(None)
              ^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\src\voxcpm\model\voxcpm2.py", line 927, in _generate_with_prompt_cache
   latent_pred, pred_audio_feat = next(inference_result)
                                  ^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 40, in generator_context
   response = gen.send(None)
              ^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\src\voxcpm\model\voxcpm2.py", line 992, in _inference
   feat_embed = self.feat_encoder(feat)  # [b, t, h_feat]
                ^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_dynamo\eval_frame.py", line 472, in __call__
   return super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
   return forward_call(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_dynamo\eval_frame.py", line 1024, in compile_wrapper
   return fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
   return forward_call(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\src\voxcpm\modules\locenc\local_encoder.py", line 17, in forward
   def forward(self, x):
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_dynamo\eval_frame.py", line 1263, in _fn
   return fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_functorch\aot_autograd.py", line 1200, in forward
   return compiled_fn(full_args)
          ^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 580, in runtime_wrapper
   all_outs = call_func_at_runtime_with_args(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_functorch\_aot_autograd\utils.py", line 138, in call_func_at_runtime_with_args
   out = normalize_as_list(f(args))
                           ^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 2298, in __call__
   return self.compiled_fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 783, in wrapper
   return compiled_fn(runtime_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 1011, in inner_fn
   outs = compiled_fn(args)
          ^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_inductor\output_code.py", line 656, in __call__
   return self.current_callable(inputs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Name\AppData\Local\Temp\torchinductor_Name\3e\c3ejitypqkjykmc43c6kwhxj5oyrnntsmac4rahk354z75d73qi4.py", line 2872, in call
   (buf284,) = self.partitions[0](partition0_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_inductor\compile_fx.py", line 1875, in run
   return compiled_fn(new_inputs)  # type: ignore[arg-type]
          ^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_inductor\cudagraph_trees.py", line 436, in deferred_cudagraphify
   fn, out = cudagraphify(model, inputs, new_static_input_idxs, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_inductor\cudagraph_trees.py", line 494, in cudagraphify
   manager = get_container(device_index).get_tree_manager()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_inductor\cudagraph_trees.py", line 364, in get_container
   container_dict = get_obj(local, "tree_manager_containers")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "E:\AI_Tools\VoxCPM\.venv\Lib\site-packages\torch\_inductor\cudagraph_trees.py", line 359, in get_obj
   assert torch._C._is_key_in_tls(attr_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

此时若重新启动，在加载主模型时，python会弹出内存错误窗口，条件是包含Control Instruction和Target Text

截图内容（点击展开）

当只包含Target Text时，会重新预热模型
不太清楚是哪个方面的问题

[liuxin99]

看你的情况，是已经compile成功了是吗？看看这个issue有没有帮助呢：#36 (comment)

[450083aa]

同样的问题，碰到assert torch._C._is_key_in_tls(attr_name)报错，在torch2.9.1cu128 torch2.10cu130都复现

[a710128]

同样的问题，碰到assert torch._C._is_key_in_tls(attr_name)报错，在torch2.9.1cu128 torch2.10cu130都复现

#107 这个应该是多线程推理的问题，如果需要多线程的话可以用 nanovllm-voxcpm 来跑。

[450083aa]

同样的问题，碰到assert torch._C._is_key_in_tls(attr_name)报错，在torch2.9.1cu128 torch2.10cu130都复现

最新版代码已无法复现该问题，应该是修复了

[Xiaodu233]

是的我这边用最新代码测试没有问题了，用的环境都没有改动。