Python 3.11+ not yet supported for torch.compile

[bennofatius]

On MacBook Pro, Apple M1 Pro, macOS 13.3.1

Cloned the repo and successfully installed dependencies.

When trying to run generate.py using the example command provided:
python generate.py \ --load_8bit \ --base_model 'decapoda-research/llama-7b-hf' \ --lora_weights 'tloen/alpaca-lora-7b'

...I get this error:

bin /opt/homebrew/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so /opt/homebrew/lib/python3.11/site-packages/bitsandbytes/cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " CUDA SETUP: Loading binary /opt/homebrew/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so... dlopen(/opt/homebrew/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so, 0x0006): tried: '/opt/homebrew/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so' (no such file), '/opt/homebrew/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so' (not a mach-o file) The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. The class this function is called from is 'LlamaTokenizer'. Loading checkpoint shards: 100%|██████████████████████████████████████████████| 33/33 [00:09<00:00, 3.63it/s] ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /Users/benjamin.pape/Desktop/development/alpaca2/generate.py:218 in <module> │ │ │ │ 215 │ │ 216 │ │ 217 if __name__ == "__main__": │ │ ❱ 218 │ fire.Fire(main) │ │ 219 │ │ │ │ /opt/homebrew/lib/python3.11/site-packages/fire/core.py:141 in Fire │ │ │ │ 138 │ context.update(caller_globals) │ │ 139 │ context.update(caller_locals) │ │ 140 │ │ ❱ 141 component_trace = _Fire(component, args, parsed_flag_args, context, name) │ │ 142 │ │ 143 if component_trace.HasError(): │ │ 144 │ _DisplayError(component_trace) │ │ │ │ /opt/homebrew/lib/python3.11/site-packages/fire/core.py:475 in _Fire │ │ │ │ 472 │ is_class = inspect.isclass(component) │ │ 473 │ │ │ 474 │ try: │ │ ❱ 475 │ │ component, remaining_args = _CallAndUpdateTrace( │ │ 476 │ │ │ component, │ │ 477 │ │ │ remaining_args, │ │ 478 │ │ │ component_trace, │ │ │ │ /opt/homebrew/lib/python3.11/site-packages/fire/core.py:691 in _CallAndUpdateTrace │ │ │ │ 688 │ loop = asyncio.get_event_loop() │ │ 689 │ component = loop.run_until_complete(fn(*varargs, **kwargs)) │ │ 690 else: │ │ ❱ 691 │ component = fn(*varargs, **kwargs) │ │ 692 │ │ 693 if treatment == 'class': │ │ 694 │ action = trace.INSTANTIATED_CLASS │ │ │ │ /Users/benjamin.pape/Desktop/development/alpaca2/generate.py:85 in main │ │ │ │ 82 │ │ │ 83 │ model.eval() │ │ 84 │ if torch.__version__ >= "2" and sys.platform != "win32": │ │ ❱ 85 │ │ model = torch.compile(model) │ │ 86 │ │ │ 87 │ def evaluate( │ │ 88 │ │ instruction, │ │ │ │ /opt/homebrew/lib/python3.11/site-packages/torch/__init__.py:1441 in compile │ │ │ │ 1438 │ │ mode = "default" │ │ 1439 │ if backend == "inductor": │ │ 1440 │ │ backend = _TorchCompileInductorWrapper(mode, options, dynamic) │ │ ❱ 1441 │ return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, │ │ 1442 │ │ 1443 │ │ 1444 def _register_device_module(device_type, module): │ │ │ │ /opt/homebrew/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:413 in optimize │ │ │ │ 410 │ │ def toy_example(a, b): │ │ 411 │ │ │ ... │ │ 412 │ """ │ │ ❱ 413 │ check_if_dynamo_supported() │ │ 414 │ # Note: The hooks object could be global instead of passed around, *however* that wo │ │ 415 │ # for a confusing API usage and plumbing story wherein we nest multiple .optimize ca │ │ 416 │ # There is some prior art around this, w/r/t nesting backend calls are enforced to b │ │ │ │ /opt/homebrew/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:377 in │ │ check_if_dynamo_supported │ │ │ │ 374 │ if sys.platform == "win32": │ │ 375 │ │ raise RuntimeError("Windows not yet supported for torch.compile") │ │ 376 │ if sys.version_info >= (3, 11): │ │ ❱ 377 │ │ raise RuntimeError("Python 3.11+ not yet supported for torch.compile") │ │ 378 │ │ 379 │ │ 380 def optimize( │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: Python 3.11+ not yet supported for torch.compile

[Oliver-whsun]

try python 3.10.
3.11 is too new, I guess.

[whitelee]

same problem , why enter sys.platfom == "win32"?
my system is win64.