m1/2 gpu support

[stoneLee81]

Feature request

it seems that it just support Nvidia gpu and chatglm/chatglm2 can't run under apple silicon env

Motivation

No response

Other

No response

[alew3]

+1 for this!

CPU works, but GPU only models are currently looking for NVidia GPUs which are not supported on the mac.

[aarnphm]

I can enable chatglm on CPU, but there is really no point since the inference are just way to slow to run on CPU anw

[alew3]

@aarnphm Is there any chance of GPU support for the Mac on Metal? Tensorflow and Pytorch already support Metal (not sure of how extensively though) Since the Mac shares memory with the VRam, it would be much more viable running a LLM locally as the other alternative is buying a professional NVidia Card.

[aarnphm]

Support metal is irrelevant of this. Metal won't make model inference faster unless we implement the attention layer in bare metal code

[alew3]

thanks for the feedback, much appreciated!