digoal
2024-03-23
PostgreSQL , PolarDB , DuckDB , Grok-1 , llama , Mac Studio
好消息, Mac Studio 可以跑马斯克开源的 Grok-1 开源大语言模型
机器: Mac Studio with an M2 Ultra and 192GB of unified ram.
Mac Studio 会不会卖疯了?
Best Mac For Large Language Models, 这里还对比了x86+GPU vs Mac Studio的各种跑大语言模型的指标: https://www.hardware-corner.net/guides/mac-for-large-language-models/
128 GB M2 Ultra (60 core) 跑70b参数大语言模型性能: Around 5 tok/s. Definitely fast enough for interactive use. (How fast did it run for you? If the 70B LLaMA 2 model was 8t/s I can't imagine the 180B Falcon model being very usable.)
In my opinion, the Apple M-series is definitely the best, most stable, quiet, and low-electricity LLM machine for testing LLM locally. Of course, it cannot be expanded as freely as nvidia, but at least it is very stable for developers who use Mac as their main weapon.
