Skip to content

Conversation

@bvrockwell
Copy link
Contributor

@bvrockwell bvrockwell commented Jul 1, 2024

  • passes profiler arguments to server launch as optional
  • uses instructions provided here for how to port forward from remote to local browser

example:

python run_server.py 
--size=8b 
--batch_size=1 
--max_cache_length=128 
--quantize_weights=false  
--quantize_kv_cache=true 
--checkpoint_path=".../models" 
--tokenizer_path=".../models/tokenizer.model" 
--model_name=llama-3
--sharding_config="default_shardings/llama.yaml"
--temperature=0.5
--sampling_algorithm="weighted"
--enable_jax_profiler=True # defaults to port 9999

Test when enabled on v5e-1:

python deps/JetStream/jetstream/tools/requester.py \
--tokenizer ".../models/tokenizer.model" 
Proxy backend support is not added
Sending request to: 0.0.0.0:9000
Prompt: Today is a good day
Response:  to be
Screenshot 2024-07-01 at 10 46 45 AM

Test when disabled (defaults):

python deps/JetStream/jetstream/tools/requester.py \
--tokenizer ".../models/tokenizer.model" 
Proxy backend support is not added
Sending request to: 0.0.0.0:9000
Prompt: Today is a good day
Response:  to be

@bvrockwell bvrockwell marked this pull request as ready for review July 1, 2024 18:08
@bvrockwell bvrockwell requested review from FanhaiLu1 and qihqi July 1, 2024 18:10
Copy link
Collaborator

@FanhaiLu1 FanhaiLu1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding it!

@bvrockwell bvrockwell merged commit ec4b158 into main Jul 1, 2024
@bvrockwell bvrockwell deleted the add-server-profiler branch July 1, 2024 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants