Skip to content

Update client completion timeout #239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 30, 2023
Merged

Update client completion timeout #239

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
506579b
update client completion timeout
seanshi-scale Aug 30, 2023
450a1a3
isort
seanshi-scale Aug 30, 2023
5e41eb5
Merge branch 'main' into seanshi/20230830-update-client-timeout
seanshi-scale Aug 30, 2023
8761113
[skip ci]
seanshi-scale Aug 30, 2023
a8c509c
don't skip ci?
seanshi-scale Aug 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions clients/python/llmengine/completion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
CompletionSyncV1Request,
)

COMPLETION_TIMEOUT = 300
Copy link
Contributor

@saiatmakuri saiatmakuri Aug 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe most HTTPS connections break after ~60s. imo since it's also a sync endpoint, would it be better to lower this timeout and if the timeout is met, we can ask clients to try again later

Copy link
Contributor Author

@seanshi-scale seanshi-scale Aug 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed the timeout between spellbookBackend -> Launch gateway to be 5 minutes, need to see if client -> spellbookBackend has a long enough timeout

do you know what usually controls how long it takes for HTTPS connections to break? wondering if it's something we can just set



class Completion(APIEngine):
"""
Expand All @@ -31,7 +33,7 @@ async def acreate(
temperature: float = 0.2,
stop_sequences: Optional[List[str]] = None,
return_token_log_probs: Optional[bool] = False,
timeout: int = 10,
timeout: int = COMPLETION_TIMEOUT,
stream: bool = False,
) -> Union[CompletionSyncResponse, AsyncIterable[CompletionStreamResponse]]:
"""
Expand Down Expand Up @@ -193,7 +195,7 @@ def create(
temperature: float = 0.2,
stop_sequences: Optional[List[str]] = None,
return_token_log_probs: Optional[bool] = False,
timeout: int = 10,
timeout: int = COMPLETION_TIMEOUT,
stream: bool = False,
) -> Union[CompletionSyncResponse, Iterator[CompletionStreamResponse]]:
"""
Expand Down