I got empty result while using 7b-it model

[egbertwong]

I use WSL2 on Windows 11 to run gemma_pytorch. The device is i9-13900 + RTX A6000 and I use the 7b-it model. But When I try to run the Gemma interface, I always get an empty result. What might be the reason and how can I solve it?

/opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Model loading done
======================================
PROMPT: The meaning of life is
RESULT:
======================================

[pengchongjin]

@egbertwong could you please provide the command you used to reproduce this?

[egbertwong]

@egbertwong could you please provide the command you used to reproduce this?

Hi, thanks for your reply! I just follow the steps in the README file. And I also write my commands down here:

VARIANT=7b
CKPT_PATH=/mnt/d/Code/gemma/gemma-7b-pytorch/gemma-7b-it-quant.ckpt

sudo usermod -aG docker $USER
newgrp docker

DOCKER_URI=gemma:${USER}

docker build -f docker/Dockerfile ./ -t ${DOCKER_URI}

PROMPT="The meaning of life is"

docker run -t --rm \
    --gpus all \
    -v ${CKPT_PATH}:/tmp/ckpt \
    ${DOCKER_URI} \
    python scripts/run.py \
    --device=cuda \
    --ckpt=/tmp/ckpt \
    --variant="${VARIANT}" \
    --prompt="${PROMPT}"

[pengchongjin]

It seems you are using quantized checkpoint, please make sure that you also add --quant in the command. I think it should work.

[egbertwong]

It seems you are using quantized checkpoint, please make sure that you also add --quant in the command. I think it should work.

Thanks, it worked!