test(e2e): winml inspect E2E tests + fix --list-tasks crash#676
Merged
Conversation
Add offline and network E2E tests for `winml inspect`, covering: - CLI surface: help flags, no-args UsageError, invalid --format - --list-tasks: exits 0, non-empty task lines, known tasks present - Offline inspection via --model-type / --model-class (no weight download) - Network inspection for bert, vision models (resnet/convnext/vit), CLIP, DETR Also fix a bug in get_known_tasks(): HF_MODEL_CLASS_MAPPING contains (model_type, None) sentinel entries for per-model-type defaults; iterating over them without a None guard caused sorted() to crash with "'<' not supported between instances of 'NoneType' and 'str'".
xieofxie
reviewed
May 19, 2026
Contributor
xieofxie
left a comment
There was a problem hiding this comment.
Code review — 3 issues, posted inline so each can be resolved individually. The get_known_tasks() None-guard fix looks clean.
🤖 Generated with Claude Code
- If this code review was useful, please react with 👍. Otherwise, react with 👎.
Address PR review comments on #676: 1. Move TestInspectCliSurface (11 tests) and TestInspectListTasks (6 tests) from tests/e2e/ to tests/cli/test_inspect_cli.py per tests/CLAUDE.md: CLI surface tests belong under tests/cli/, not tests/e2e/. They now run under the default CI filter without -m e2e. 2. Replace tautological assertions in test_list_tasks_all_lines_are_strings (isinstance(str.strip(), str) is always true; len > 0 is guaranteed by the if-guard) with a regex invariant: each task line must match r"^[a-z][a-z0-9-]*$". 3. Collapse two redundant tests (test_list_tasks_no_model_flag_needed and test_list_tasks_overrides_missing_model_requirement both just asserted exit_code == 0, already covered by test_list_tasks_exits_zero) into a single test_list_tasks_is_sorted that verifies the --list-tasks output is in ascending lexicographic order.
timenick
reviewed
May 20, 2026
- Parametrize 8 test_help_documents_* into one test with a class-scoped help_output fixture (one --help call per class) - Merge test_invalid_format_exits_nonzero + test_invalid_format_names_bad_choice into single test_invalid_format - Extract shared _run helper to tests/_helpers.py with correct -> Result return type; both cli and e2e test files import from there
timenick
approved these changes
May 20, 2026
Collaborator
timenick
left a comment
There was a problem hiding this comment.
LGTM. All three review comments cleanly addressed in 41d1947:
- 8
test_help_documents_*collapsed into a single parametrized test with a class-scopedhelp_outputfixture (one--helpinvocation per class) test_invalid_format_*merged into one method_runextracted totests/_helpers.pywith the correctResultreturn type, shared by both the CLI and E2E test files
The get_known_tasks() None-sentinel fix is also correct — verified that MODEL_CLASS_MAPPING (e.g. in models/hf/sam.py) intentionally uses (model_type, None) entries as per-model-type defaults.
🤖 Generated with GitHub Copilot CLI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tests/e2e/test_inspect_e2e.py— 39 tests coveringwinml inspect.Offline tests run without downloading any model weights; network tests hit
HuggingFace Hub with real model IDs.
Also fixes a crash in
get_known_tasks():HF_MODEL_CLASS_MAPPINGcontains(model_type, None)sentinel entries for per-model-type defaults; iteratingwithout a
Noneguard causedsorted()to raiseTypeError: '<' not supported between instances of 'NoneType' and 'str'whenever
--list-taskswas invoked.CLI commands under test
Coverage
TestInspectCliSurface— CLI surface (11 tests, offline)winml inspect(no args) → exit 2 with UsageErrorwinml inspect --help→ exit 0--help:-m,-f,--model-type,--model-class,--list-tasks,-v/--verbose,-H/--hierarchy--format xml→ exit non-zero, error mentions the bad choiceTestInspectListTasks—--list-tasks(6 tests, offline)feature-extraction,mask-generation-m/--model-type/--model-classargumentwinml inspect→ exit 2;winml inspect --list-tasks→ exit 0TestInspectModelTypeOnly— offline inspection (13 tests, offline)--model-type bertand--model-type resnetresolve correctmodel_type+taskin JSON--model-class BertForMaskedLMresolves tobert/fill-maskEXPECTED_TOP_KEYSpresent in JSON outputloader,exporter,winmlsub-sections have required fieldsexporter.input_tensorsandoutput_tensorsare lists-v/--verboseand table format both exit 0--hierarchywithout-m→hierarchy: null(no crash)TestInspectBert—bert-base-uncased(3 tests, network)fill-mask,task_source == "TasksManager"feature-extractionoverride acceptednext-sentence-prediction: clean success or clean error (no raw traceback)TestInspectVision— vision models (3 tests, network)microsoft/resnet-50,facebook/convnext-tiny-224,google/vit-base-patch16-224image-classificationfor all threeTestInspectCLIP—openai/clip-vit-base-patch32(2 tests, network)feature-extractionorzero-shot-image-classificationimage-feature-extractionoverride acceptedTestInspectDETR—facebook/detr-resnet-50(1 test, network)object-detection