-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add input validation #1525
Add input validation #1525
Conversation
@@ -101,6 +101,14 @@ def special_word_token_ids(words): | |||
stop_words=special_word_token_ids(gen_config.stop_words), | |||
bad_words=special_word_token_ids(gen_config.bad_words)) | |||
|
|||
def __post_init__(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's make server return error message to users not crash
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will not crash server
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will not crash server
Hi @AllentDan Do we catch AssertionError
and handle it when validation fails?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we catch the error. Now, for the api_server, we check request before __post_init__
lmdeploy/serve/openai/api_server.py
Outdated
return ret | ||
if hasattr(request, 'model') and request.model not in get_model_list(): | ||
return create_error_response( | ||
HTTPStatus.NOT_FOUND, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The http status not found is designed to represent 404.
bad request should be ok here.
lmdeploy/messages.py
Outdated
assert type( | ||
self.n) == int and self.n > 0, 'n is not a positive integer' | ||
assert self.top_p > 0 and self.top_p <= 1 # (0, 1] | ||
assert self.top_k >= 1, 'top_k must be a positive integer' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
turbomind里面 top_k 可以是0,表示跳过top_k
ref #1324 |
lmdeploy/serve/openai/api_server.py
Outdated
if hasattr(request, 'top_k') and request.top_k < 0: | ||
return create_error_response( | ||
HTTPStatus.BAD_REQUEST, | ||
f'The top_k `{request.top_p}` cannot be a negative integer.') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@AllentDan typo request.top_p
-> request.top_k
No description provided.