stop_sequence not working

[SinanAkkoyun]

System Info

Hello! No matter what API call, when setting stop to ["."] (or something else) it doesn't work

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Install latest local version and run a LLaMa model

Expected behavior

Should stop at end of sentence.

[OlivierDehaene]

It does work:

curl https://api-inference.huggingface.co/models/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 \
        -X POST \
        -d '{"inputs": "What is your name?"}}' \
        -H "Content-Type: application/json"
        
# [{"generated_text":"What is your name?\n\nMy name is Open Assistant.\n\nWhat is your purpose?\n\nMy purpose is"}]%

curl https://api-inference.huggingface.co/models/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 \
        -X POST \
        -d '{"inputs": "What is your name?", "parameters":{"stop":["."]}}}' \
        -H "Content-Type: application/json"
        
# [{"generated_text":"What is your name?\n\nMy name is Open Assistant."}]%

I'm closing this issue. If you give us a reproducible example, I will re-open it.

[SinanAkkoyun]

Oh yes, I set the parameters wrong, my bad... Thank you!

[josephrocca]

Edit: I posted this as its own new issue here:

• parameters.stop only works if the stop sequences align with token boundaries #1039

---

@OlivierDehaene This doesn't seem to work for subwords that don't match up with tokens:

docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:1.0.3 --model-id abacaj/starcoderbase-1b-sft

Par works:

curl http://localhost:8080 \
        -X POST \
        -d '{"inputs": "The capital city of France", "parameters":{"stop":["Par"]}}}' \
        -H "Content-Type: application/json"

# [{"generated_text":"The capital city of France is Par"}]

Pari doesn't work:

curl http://localhost:8080 \
        -X POST \
        -d '{"inputs": "The capital city of France", "parameters":{"stop":["Pari"]}}}' \
        -H "Content-Type: application/json"

# [{"generated_text":"The capital city of France is Paris.\n\nYou are given a string array named cities and an integer array named cap"}]

Is this expected behavior? If so it's very counter-intuitive. IIRC, all other APIs I've used guarantee that the stop sequence strings will not occur in the generated output.