Skip to content

Feature/model pruning#1

Merged
BradleyEdelman merged 8 commits intomainfrom
feature/model_pruning
Feb 11, 2025
Merged

Feature/model pruning#1
BradleyEdelman merged 8 commits intomainfrom
feature/model_pruning

Conversation

@BradleyEdelman
Copy link
Owner

This PR introduces a refined adaptive training strategy with a constant pruning ratio. Key updates:

  • Score Calculation: This version now computes an accuracy score and a memory score based on resource usage and model performance
  • Parameter Prioritization: Accuracy and memory scores are weighted according to default or user-defined priority weighting to idenfity a priority list for parameter adjustment. Now, only the top priority paramater is adjusted in each epoch.
    • Batch size priority is weighted by memory usage.
    • Learning rate priority is inversely weighted by accuracy improvement (i.e. increases if accuracy stagnates).
  • Fixed Pruning Ratio: Pruning is constant and is stripped at the end.
  • Code Quality Improvements: Added pre-commit hooks and CI linting for consistency.

@BradleyEdelman
Copy link
Owner Author

Merge feature/model_pruning: Introducing a refined adaptive training strategy with a constant pruning ratio that utilizes a memory/accuracy scoring system and parameter priority weighting system to identify which parameters (batch size, learning rate) to adjust, how, and when.

@BradleyEdelman BradleyEdelman merged commit 23e84cb into main Feb 11, 2025
1 check passed
@BradleyEdelman BradleyEdelman deleted the feature/model_pruning branch February 11, 2025 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant