Skip to content

[EAGLE-5408]: Add storage request inferred from tar and checkpoint size#479

Merged
patricklundquist merged 18 commits intomasterfrom
modelUploadStorageRequestSize
Feb 5, 2025
Merged

[EAGLE-5408]: Add storage request inferred from tar and checkpoint size#479
patricklundquist merged 18 commits intomasterfrom
modelUploadStorageRequestSize

Conversation

@patricklundquist
Copy link
Copy Markdown
Contributor

@patricklundquist patricklundquist commented Jan 14, 2025

Why

  • prevent build failures and poor resource allocation due to wrong storage requests.

How

Tests

  • clarifai model upload --model_path ../examples/models/model_upload/llms/llama-3-8b-instruct

Notes

Copy link
Copy Markdown
Contributor

@luv-bansal luv-bansal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I have doubts about whether the get_huggingface_checkpoint_total_size function will work for restricted models.

If it’s working, we can merge it for now. In the follow-up PR, I’ll address the comment I added.

Comment thread clarifai/runners/models/model_upload.py Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 3, 2025

Code Coverage

Package Line Rate Health
clarifai 43%
clarifai.cli 64%
clarifai.client 71%
clarifai.client.auth 74%
clarifai.constants 100%
clarifai.datasets 100%
clarifai.datasets.export 80%
clarifai.datasets.upload 75%
clarifai.datasets.upload.loaders 37%
clarifai.models 100%
clarifai.modules 0%
clarifai.rag 72%
clarifai.runners 13%
clarifai.runners.models 57%
clarifai.runners.utils 66%
clarifai.schema 100%
clarifai.urls 80%
clarifai.utils 77%
clarifai.utils.evaluation 67%
clarifai.workflows 94%
Summary 68% (4454 / 6578)

Minimum allowed line rate is 50%

@patricklundquist patricklundquist enabled auto-merge (squash) February 3, 2025 18:17
@patricklundquist patricklundquist merged commit fc99a27 into master Feb 5, 2025
@patricklundquist patricklundquist deleted the modelUploadStorageRequestSize branch February 5, 2025 05:15
@luv-bansal luv-bansal mentioned this pull request Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants