Skip to content

feat(native): expose context batch sizing controls#71

Merged
leehack merged 3 commits intomainfrom
feat/issue-68-context-batch-params
Feb 27, 2026
Merged

feat(native): expose context batch sizing controls#71
leehack merged 3 commits intomainfrom
feat/issue-68-context-batch-params

Conversation

@leehack
Copy link
Copy Markdown
Owner

@leehack leehack commented Feb 26, 2026

Summary

  • Add ModelParams.batchSize / ModelParams.microBatchSize so native n_batch and n_ubatch can be tuned independently of contextSize while preserving legacy defaults.
  • Wire resolved batch sizing into native context creation and prompt decode chunking to support smaller CPU-friendly batch settings safely.
  • Add unit coverage for batch-size resolution and update docs/changelog, including llama.cpp semantics references for n_batch and n_ubatch.

Validation

  • dart analyze
  • dart test test/unit/core/models/inference/model_params_test.dart
  • dart test test/unit/backends/llama_cpp/llama_cpp_service_test.dart

Add ModelParams batchSize/microBatchSize with legacy defaults and wire them into context creation and prompt decode chunking so users can tune memory/performance without changing existing behavior.
Document that batchSize maps to n_batch and microBatchSize maps to n_ubatch so users can tune these fields with the same mental model as upstream llama.cpp CLI options.
@leehack
Copy link
Copy Markdown
Owner Author

leehack commented Feb 26, 2026

@maiguangyang Let me know if this fulfills your request.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Feb 26, 2026

Codecov Report

❌ Patch coverage is 86.48649% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.54%. Comparing base (fd4ac69) to head (102c3bb).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
lib/src/backends/llama_cpp/llama_cpp_service.dart 91.42% 3 Missing ⚠️
lib/src/core/models/inference/model_params.dart 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #71      +/-   ##
==========================================
+ Coverage   76.51%   76.54%   +0.03%     
==========================================
  Files          66       66              
  Lines        8196     8216      +20     
==========================================
+ Hits         6271     6289      +18     
- Misses       1925     1927       +2     
Flag Coverage Δ
unittests 76.54% <86.48%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leehack leehack merged commit b39e062 into main Feb 27, 2026
6 checks passed
@leehack leehack deleted the feat/issue-68-context-batch-params branch February 27, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants