Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Conversation

@louis-jan
Copy link
Contributor

@louis-jan louis-jan commented Feb 14, 2024

Description

  • Deprecate shared single_queue_busy state & SerialTaskQueue since it would not work with cont_batching = true and n_parallel > 1.
  • Deprecate instance's is_stopped state, better using inferenceStatus enum.
  • Introduce ConcurrentTaskQueue to handle tasks with the workers number = n_parallel.
  • No complicated state handling, put into the queue and it just work (inference & embedding).

Test results

  • n_parallel = 1 & cont_batching = false
    |        T1          |       T2           |       T3          |       T4          |
    |--------------------|--------------------|-------------------|-------------------|
    | Send chat comp.    | Send chat          | A1 completed      | A2 completed      |
    | request A1         | comp. request A2   | A2 started        |                   |
    |                    |                    |                   |                   |
    |                    |                    |                   |                   |
    
  • n_parallel = 2 & cont_batching = true
    |      T1      |      T2      |       T3       |       T4       |                T5               |
    |--------------|--------------|----------------|----------------|--------------------------------|
    | Send chat    | Send chat    | A1 & A2 are    | Send chat      | A1 & A2 are                    |         
    | completion   | completion   | running        | completion     | completed                      |                            
    | request A1   | request A2   |                | request A3     | A3 started running             |                 
    

@louis-jan louis-jan force-pushed the refactor/simplify-state-with-queue-system branch from 0f1d664 to 5a3432f Compare February 14, 2024 03:34
@louis-jan louis-jan requested a review from tikikun February 14, 2024 03:35
Copy link
Contributor

@tikikun tikikun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tikikun tikikun merged commit fb7bc74 into main Feb 14, 2024
@louis-jan louis-jan deleted the refactor/simplify-state-with-queue-system branch February 15, 2024 09:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants