Skip to content

Fix quantization P0 bugs#680

Merged
zhenchaoni merged 3 commits into
mainfrom
private/zhenni/fix_quantize_bugs
May 21, 2026
Merged

Fix quantization P0 bugs#680
zhenchaoni merged 3 commits into
mainfrom
private/zhenni/fix_quantize_bugs

Conversation

@zhenchaoni
Copy link
Copy Markdown
Member

PR Title

fix(quantize/eval): validate --precision, stream default image dataset, support pixel_mask padding

PR Description

Summary

Three independent fixes bundled together — two are user-reported bugs, one is a side-fix found while validating object detection.

Changes

1. Reject invalid --precision values (fixes #555)

Previously, winml quantize --precision banana (or any unrecognized string) silently fell through to uint8/uint8 defaults and reported "Success!", masking user typos like w4a16 or fp64. _resolve_quant_types now raises click.BadParameter for any precision that is not None, "auto" (case-insensitive), or a value accepted by is_quantized_precision. The error message lists the accepted forms.

2. Stream the default image calibration dataset (fixes #433)

The default calibration dataset (timm/mini-imagenet) was downloading the entire training split (~5 GB) just to pull 10 calibration samples. ImageDataset._get_default_dataset now sets streaming=True by default, and the shared _load_and_sample helper takes the streaming path when streaming is enabled and max_samples is set (streaming without a sample cap would be strictly worse than a bulk download, so it falls back automatically). ObjectDetectionDataset reuses the same helper, eliminating duplicated load/sample logic.

Verified: with the HF cache cleared, running winml quantize -m resnet-50 --task image-classification --model-name microsoft/resnet-50 --samples 10 leaves datasets--timm--mini-imagenet at 0 MB on disk.

3. Support pixel_mask input for DETR-family object detection

Some object detection exports (DETR, etc.) declare a pixel_mask input alongside pixel_values, which the HF image processor only populates meaningfully when padding to a fixed canvas. WinMLObjectDetectionEvaluator.prepare_pipeline now detects this case and switches the processor to aspect-preserving resize (shortest_edge/longest_edge) + pad_size + do_pad=True. Models without pixel_mask keep the previous exact-resize, no-padding behavior.

Tests

  • TestQuantizePrecisionValidation — parametrized over banana, w4a16, int4, fp64; asserts quantize_onnx is never invoked.
  • TestPreparePipeline::test_pixel_mask_path_enables_padding and test_no_pixel_mask_path_disables_padding — cover both branches with a sentinel to prove the wrong branch did not run. Uses monkeypatch for the parent-class prepare_pipeline stub so the override is reverted on teardown.

All 84 affected unit tests pass.

Manual verification

Issue Verification
#555 winml quantize -p bananaBadParameter: 'banana' is not a supported quantization precision. Expected: auto | int8 | int16 | w{x}a{y} ...
#433 Cleared HF cache → ran quantize → datasets--timm--mini-imagenet = 0.0 MB on disk; only 10 examples flowed through Map/Aligning the labels
pixel_mask Covered by new unit tests

@zhenchaoni zhenchaoni requested a review from a team as a code owner May 20, 2026 03:47
Copy link
Copy Markdown
Collaborator

@timenick timenick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Five findings — none blocking, but worth addressing before merge. The biggest one is that the new error message redirects users to --help, but --help doesn''t yet mention auto as an accepted value, so users hitting the BadParameter still won''t know how to fix it.

Comment thread src/winml/modelkit/commands/quantize.py
Comment thread src/winml/modelkit/commands/quantize.py
Comment thread src/winml/modelkit/datasets/image.py
Comment thread tests/unit/eval/test_object_detection_evaluator.py
Comment thread tests/unit/eval/test_object_detection_evaluator.py
Comment thread src/winml/modelkit/datasets/image.py
@zhenchaoni zhenchaoni merged commit 201f997 into main May 21, 2026
9 checks passed
@zhenchaoni zhenchaoni deleted the private/zhenni/fix_quantize_bugs branch May 21, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[winml quantize] [P0] --precision banana silently falls back to defaults, prints "Success!", exits 1 Bug: Quantize stage in build shows train output

3 participants