When installing ART via the following pyproject file:
[project]
name = "agentdac"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"json-repair>=0.59.10",
"math-verify>=0.9.0",
"openpipe-art[backend]==0.5.17",
"wrapt-timeout-decorator>=1.5.1",
]I get the following dependency error:
...
/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
...
...
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
wandb: Currently logged in as: giladfrid009 (giladfrid009-technion-israel-institute-of-technology) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.25.0
wandb: Run data is saved locally in /home/fre.gilad/source/AgentDaC/wandb/run-20260527_115527-Qwen3-14B_05_27_11_55
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run Qwen3-14B_05_27_11_55
wandb: ⭐️ View project at https://wandb.ai/giladfrid009-technion-israel-institute-of-technology/math_dac
wandb: 🚀 View run at https://wandb.ai/giladfrid009-technion-israel-institute-of-technology/math_dac/runs/Qwen3-14B_05_27_11_55
wandb: Detected [litellm, openai] in use.
wandb: Use W&B Weave for improved LLM call tracing. Weave is installed but not imported. Add `import weave` to the top of your script.
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/art/unsloth/service.py:934: UserWarning: WARNING: Unsloth should be imported before [trl, transformers, peft] to ensure all optimizations are applied. Your code may run slower or encounter memory issues without these optimizations.
Please restructure your imports with 'import unsloth' at the top of your file.
import unsloth
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
==((====))== Unsloth 2026.3.3: Fast Qwen3 patching. Transformers: 4.57.3. vLLM: 0.17.0.
\\ /| NVIDIA H200 NVL. Num GPUs = 1. Max memory: 139.719 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.10.0+cu128. CUDA: 9.0. CUDA Toolkit: 12.8. Triton: 3.6.0
\ / Bfloat16 = TRUE. FA [Xformers = 0.0.35. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:36<00:00, 6.05s/it]
unsloth/Qwen3-14B does not have a padding token! Will use pad_token = <|PAD_TOKEN|>.
Traceback (most recent call last):
File "/home/fre.gilad/source/AgentDaC/experiments/math/run.py", line 61, in <module>
Runner().run()
File "/home/fre.gilad/source/AgentDaC/experiments/experiment_runner.py", line 324, in run
asyncio.run(self._main())
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/nest_asyncio.py", line 30, in run
return loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
^^^^^^^^^^
File "/home/fre.gilad/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/home/fre.gilad/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/asyncio/tasks.py", line 277, in __step
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/experiments/experiment_runner.py", line 251, in _main
model = await load_art_model(
^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/src/utils/loaders.py", line 43, in load_art_model
await model.register(backend, _openai_client_config=art_config.openai_config)
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/art/model.py", line 1003, in register
base_url, api_key = await backend._prepare_backend_for_training(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/art/local/backend.py", line 402, in _prepare_backend_for_training
host, port = await service.start_openai_server(config=resolved_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/art/unsloth/service.py", line 477, in start_openai_server
self._state.trainer.save_model(lora_path)
^^^^^^^^^^^
File "/home/fre.gilad/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/functools.py", line 1001, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/art/unsloth/service.py", line 959, in _state
unsloth.FastLanguageModel.get_peft_model(
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/unsloth/models/llama.py", line 3122, in get_peft_model
model = _get_peft_model(model, lora_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/mapping_func.py", line 122, in get_peft_model
return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/peft_model.py", line 1955, in __init__
super().__init__(model, peft_config, adapter_name, **kwargs)
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/peft_model.py", line 129, in __init__
self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 315, in __init__
self.inject_adapter(self.model, adapter_name, low_cpu_mem_usage=low_cpu_mem_usage, state_dict=state_dict)
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 913, in inject_adapter
self._create_and_replace(
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/tuners/lora/model.py", line 269, in _create_and_replace
new_module = self._create_new_module(lora_config, adapter_name, target, device_map=device_map, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/tuners/lora/model.py", line 418, in _create_new_module
new_module = dispatcher(target, adapter_name, config=lora_config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/tuners/lora/torchao.py", line 142, in dispatch_torchao
if not is_torchao_available():
^^^^^^^^^^^^^^^^^^^^^^
File "/home/fre.gilad/source/AgentDaC/.venv/lib/python3.11/site-packages/peft/import_utils.py", line 143, in is_torchao_available
raise ImportError(
ImportError: Found an incompatible version of torchao. Found version 0.15.0, but only versions above 0.16.0 are supportedI see that torchao==0.15.0 is pinned via the [backend] package of ART.
But actually simply uninstalling torchao after-the-fact completely resolves the issue. (uv pip uninstall torchao)
To be fair i didnt perform extensive testing and validated that nothing else breaks as result of removing torchao.
At least on the temporal-clue examples removing torchao resolved the issue.
When installing ART via the following pyproject file:
I get the following dependency error:
I see that torchao==0.15.0 is pinned via the
[backend]package of ART.But actually simply uninstalling torchao after-the-fact completely resolves the issue. (
uv pip uninstall torchao)To be fair i didnt perform extensive testing and validated that nothing else breaks as result of removing torchao.
At least on the temporal-clue examples removing torchao resolved the issue.