prep for mlx-swift 0.29.1 #411
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidkoski
commented
Oct 13, 2025
Package.swift
Outdated
| ], | ||
| dependencies: [ | ||
| .package(url: "https://github.com/ml-explore/mlx-swift", .upToNextMinor(from: "0.25.5")), | ||
| .package(url: "https://github.com/ml-explore/mlx-swift", branch: "mlx-0291"), |
There was a problem hiding this comment.
For now just point to the branch.
davidkoski
commented
Oct 13, 2025
| static public let gpt_oss_20b_MXFP4_Q8 = ModelConfiguration( | ||
| id: "mlx-community/gpt-oss-20b-MXFP4-Q8", | ||
| defaultPrompt: "Why is the sky blue?" | ||
| ) |
There was a problem hiding this comment.
The MXFP4 quantization is now supported. This model was used to test that and the quantized kvcache.
davidkoski
commented
Oct 13, 2025
| class QuantizedSwitchLinear: SwitchLinear, Quantized { | ||
| @ModuleInfo(key: "scales") var scales: MLXArray | ||
| @ModuleInfo(key: "biases") var biases: MLXArray | ||
| @ModuleInfo(key: "biases") var biases: MLXArray? |
There was a problem hiding this comment.
biases are now optional
davidkoski
commented
Oct 13, 2025
| public let bits: Int | ||
| public var quantMethod: String? = nil | ||
| public var linearClass: String? = nil | ||
| public var quantizationMode: String? = nil |
There was a problem hiding this comment.
These were defined so that they could be skipped (below). We can just skip them directly.
davidkoski
commented
Oct 13, 2025
|
|
||
| // additional keys that are not layer instructions, see | ||
| // mlx-community/bitnet-b1.58-2B-4T-4bit | ||
| case "quant_method", "linear_class", "quantization_mode": continue |
davidkoski
commented
Oct 13, 2025
| /// - Returns: Quantized tuples (keys, values) as ((weight, scales, biases), (weight, scales, biases)) | ||
| func updateQuantized(keys: MLXArray, values: MLXArray) -> ( | ||
| (MLXArray, MLXArray, MLXArray), (MLXArray, MLXArray, MLXArray) | ||
| (MLXArray, MLXArray, MLXArray?), (MLXArray, MLXArray, MLXArray?) |
There was a problem hiding this comment.
Support for optional biases.
davidkoski
commented
Oct 13, 2025
| case groupSize = "group_size" | ||
| case bits | ||
| } | ||
| } |
There was a problem hiding this comment.
We can just use the standard configuration for this
davidkoski
commented
Oct 13, 2025
Comment on lines
-460
to
462
| let fullPath = "language_model.\(path)" | ||
| if weights["\(fullPath).scales"] != nil && weights["\(fullPath).biases"] != nil | ||
| if weights["\(fullPath).scales"] != nil | ||
| && weights["\(fullPath).weight"]?.dtype == .uint32 |
There was a problem hiding this comment.
biases is now optional -- handle that case. I am not sure this is strictly required as the load() method handles this, though not with this exact logic.
awni
approved these changes
Oct 16, 2025
- handle changes in quantization
This was referenced Oct 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support for mlx-swift 0.29.1. In particular this is updates for changes in quantization.