Several Inference Endpoint fixes #66
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was generating a tricky error when calling "/health" at the server startup: this was calling prefill and returning None as the cached batch, that was failing to be serialized.
Doing that on TPU seems to slow down (due to compilation?) and takes a lot of memory.
This allows to correctly handle warmup.
This clears a potential issue when clearing TGI requests. When a client cancels a TGI request, two different methods can be called on the TGI server: - if the request is cancelled after prefill, then the router asks the server to "filter" the decoding batch from the corresponding request. This is correctly implemented, - if the request is cancelled during prefill, then the router asks the server to clear the whole prefill batch. This was not correctly implemented because in that configuration we cleared all requests, even those not included in that prefill batch. This is now fixed, basically reproducing TGI Neuron fix: huggingface/optimum-neuron#609
mfuntowicz
approved these changes
Jul 2, 2024
Comment on lines
+382
to
+383
| input_ids = torch.full((batch_size, seq_length), self.tokenizer.pad_token_id, dtype=torch.int64) | ||
| attention_mask = torch.full((batch_size, seq_length), 0, dtype=torch.int64) |
There was a problem hiding this comment.
qq: Can't we make it use int32 for input_ids and attention_mask ?
There was a problem hiding this comment.
Umh, maybe we could, but the price would be a frequent cast between int32 and int64, because PB integers are serialized in int64. I think it's not worth doing it here, we could look at the advantages/disadvantages in another PR.
| if slot.state != Slot.State.EMPTY and slot.request_id not in request_ids: | ||
| logger.debug(f"Removing request {slot.request_id}") | ||
| if slot.state != Slot.State.EMPTY and slot.id not in keep_slot_ids: | ||
| logger.info(f"Removing slot {slot.id} with request {slot.request_id}") |
There was a problem hiding this comment.
debug might be enough for this imo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?