Fix Salesforce xgen inference

[jaywonchung]

Why are these changes needed?

While running Salesforce/xgen-7b-8k-inst inference, I found that its conversation template had some issues.

First, roles=("### Human: ", "###"). If you look at the last line of xgen's Hugging Face Hub example, you can see that it's expecting the model to output "Assistant:". Then why not give it to the model ourselves and save two iterations in the decoding loop? Hence the change to roles and sep_style. This results in better role formatting when using python -m fastchat.serve.cli because before the change, the assistant output looked like ### : Assistant: [RESPONSE].

Second, stop_token_ids=[50256, 0, 1, 2]. These four tokens correspond to ['<|endoftext|>', '!', '"', '#'] respectively, and due to the later three tokens, model responses stop abruptly. For instance:

### Human: How do I use pound to comment in Python?
### Assistant: To use pound (a new syntax for comments) in Python, you can simply write it as you would any other comment: as a line comment using the pound sign (#) followed by a line break.

For example, here's how you would use pound to create a comment in Python:

#
### Human: exit...

FYI this output is after fixing roles and sep_style. You can see that the model was stopped due to # and I've confirmed this with print debugging in inference.py. I've seen abrupt response stops due to ! and ", too. Just leaving 50256 in stop_token_ids fix this problem:

### Human: How do I use pound to comment in Python?
### Assistant: In Python, you can use the pound (#) character to create a comment. A comment is text that starts with the hash symbol "#" and continues until the end of the line. Anything after the hash symbol on a line is ignored by the interpreter when that line is executed. This can be a good way to add notes to your code or to make long blocks of code easier to read.

Here's an example of how to use the pound character to create a comment in a Python script:
```
# This is a comment

# This line also creates a comment

# This is the end of the comment
```
In this example, the first line is a comment because it starts with the hash symbol. The second and third lines are also comments because they also start with the hash symbol. However, the fourth line is not a comment because it does not start with the hash symbol.
### Human: exit...

Third, an unrelated bug fix in inference.py is that if you don't check whether eos_token_id is already part of stop_token_ids, which is a reference into a list inside params, you make stop_token_ids one token longer for each invocation of generate_stream.

Question: Does stop_str="<|endoftext|>" have to be specified when token ID 50256 is already "<|endoftext|>"?

[jaywonchung]

@merrymercy @infwinston Looks like other PRs are getting merged. Let me know if there's anything I can help.

[merrymercy]

waiting for @infwinston's approval

[jaywonchung]

Just a side note. I've been using this patch for two weeks to run batch inference on about 3000 prompts. It works well.

Check is failing due to formatting issue in fastchat/serve/openai_api_server.py. I won't touch that.

[merrymercy]

@jaywonchung Thanks! It has been merged. I will fix the format error later.