Skip to content

issues Search Results · repo:OpenCSGs/llm-inference language:Python

Filter by

34 results
 (69 ms)

34 results

inOpenCSGs/llm-inference (press backspace or delete to remove)

OpenCSGs/llm-inference
Api server blocked when one request is in-process

Need more test for this issue
bug
  • SeanHH86
  • 1
  • Opened 
    on May 9, 2024
  • #137

OpenCSGs/llm-inference
Api server was blocked when LLM deployment scaling config beyond the cluster resouces

for example, ray cluster just has 12 cpus. curl -H Content-Type: application/json -H user-name: default -d [{ model_id : facebook/opt-125m , model_task : text-generation , model_revision : ...
  • SeanHH86
  • Opened 
    on May 8, 2024
  • #134

OpenCSGs/llm-inference
Auto load models from ./models for when api server start

  • SeanHH86
  • 1
  • Opened 
    on May 7, 2024
  • #132

OpenCSGs/llm-inference
Upgrade ray 2.20.0

New feature needs for deploy ray on kubernetes.
enhancement
  • SeanHH86
  • 1
  • Opened 
    on May 5, 2024
  • #130

OpenCSGs/llm-inference
Error happen when do inference for wukong dtype=bfloat16 of use default transformer pipeline load model

1:job_id:04000000 :actor_name:ServeReplica:default:opencsg--csg-wukong-1B [INFO 2024-04-30 03:46:04,636] __init__.py: 14 Import vllm related stuff failed, please make sure vllm is installed. INFO 2024-04-30 ...
  • SeanHH86
  • 1
  • Opened 
    on Apr 30, 2024
  • #128

OpenCSGs/llm-inference
GGUF implements will make duplicate copy since cannot detect config.json file in the cache folder

  • depenglee1707
  • Opened 
    on Apr 24, 2024
  • #123

OpenCSGs/llm-inference
vllm implements cannot support download model from repo besides hg

initialization: runtime_env: env_vars: HF_ENDPOINT: https://hub.opencsg.com/hf initializer: type: Vllm from_pretrained_kwargs: trust_remote_code: true pipeline: ...
  • depenglee1707
  • 2
  • Opened 
    on Apr 23, 2024
  • #120

OpenCSGs/llm-inference
avoid to ping huggingface when start serving to speed up the deployement

  • depenglee1707
  • Opened 
    on Apr 23, 2024
  • #117

OpenCSGs/llm-inference
vllm, gguf, llamacpp, these integration cannot address local path of model

  • depenglee1707
  • Opened 
    on Apr 23, 2024
  • #116

OpenCSGs/llm-inference
enable reset generate config on fly

for now the generation params is addressed in yaml files, add the ability reset these params on fly is useful: generate_kwargs: do_sample: false max_new_tokens: 512 min_new_tokens: ...
enhancement
  • depenglee1707
  • Opened 
    on Apr 17, 2024
  • #104
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue search results · GitHub