Skip to content

test: fix stale/wrong test assertions and CI trigger gap - #19

Merged
Adithya-Thonse merged 3 commits into
TexasInstruments:mainfrom
musicalplatypus:pr/test-ci-hygiene
Aug 6, 2026
Merged

test: fix stale/wrong test assertions and CI trigger gap#19
Adithya-Thonse merged 3 commits into
TexasInstruments:mainfrom
musicalplatypus:pr/test-ci-hygiene

Conversation

@musicalplatypus

Copy link
Copy Markdown
Contributor

Summary

Two independent test/CI hygiene fixes, confirmed unrelated to any production code change (verified via git stash — all of the below fail identically on unmodified main).

Stale/wrong test assertions

  • test_config_validation.py: KNOWN_TASK_TYPES and the model-registry check only covered timeseries + vision, so every audio modelzoo example (task_type='audio_classification', e.g. DSCNN_NPU) looked invalid. Includes the audio module's task types and routes audio configs to its own model registry.
  • test_constants.py: get_default_data_dir_for_task() now raises ValueError on unknown categories, but the test still asserted the old silent DATA_DIR_CLASSES fallback. Updated to match current behavior.
  • test_cross_device.py: TestQuantizationFlags called get_skip_normalize_and_output_int(..., partial_quantization=...), but the real parameter is auto_quantization — a wrong kwarg name, not a real API. Renamed calls; the one test that was actually about auto_quantization semantics is kept meaningful.
  • test_dataset_utils.py: test_split_factor_too_large matched a stale error-message regex ("less than 1") against the current message ("must be in the range (0.0, 1.0)"); behavior was already correct, only the assertion was stale.

One gap found and left open, not guessed at: F28E12 is listed in TARGET_DEVICES but has no entry in _DEVICE_PROFILES, so test_cross_device.py::TestCompilationProfileCorrectness still fails for it (5 tests). This needs real hardware values (cross_compiler path, target, target_c_mcpu, has_hard_npu) that aren't safe to fabricate — flagging it here rather than papering over it with placeholder values.

CI trigger gap

tinyml-modelmaker depends on tinyml-modelzoo (pip install -e tinyml-modelzoo in the Install dependencies step), but dependency-only changes to tinyml-modelzoo wouldn't trigger the test suite at all. Adds the missing path trigger.

Note: this touches .github/workflows/test-modelmaker.yml, which two of my other open PRs (pr/dataloader-pinning-fix, pr/mps-eval-fixes) also touch — each adds one path-trigger line at the same position. If merged after either, GitHub will show a trivial 3-line conflict; the correct resolution is just keeping all the added lines (a union), not choosing one over the other.

🤖 Generated with Claude Code

t5fkg8d44d-beep and others added 2 commits July 29, 2026 11:15
- test_config_validation.py: KNOWN_TASK_TYPES and the model-registry
  check only covered timeseries+vision, so every audio modelzoo
  example (task_type='audio_classification', e.g. DSCNN_NPU) looked
  invalid. Include the audio module's task types and route audio
  configs to its own model registry.
- test_constants.py: get_default_data_dir_for_task() was intentionally
  changed to raise ValueError on unknown categories in 90ce979
  (CodeRabbit fix), but the test still asserted the old silent
  DATA_DIR_CLASSES fallback. Update the test to match.
- test_cross_device.py: TestQuantizationFlags called
  get_skip_normalize_and_output_int(..., partial_quantization=...),
  but the real parameter is auto_quantization -- a wrong kwarg name,
  not a real API. Rename calls and the one test that was actually
  about auto_quantization semantics.
- test_dataset_utils.py: test_split_factor_too_large matched a stale
  error-message regex ("less than 1") against the current message
  ("must be in the range (0.0, 1.0)"); behavior was already correct.

Confirmed via git stash that all of the above failed identically
before the unrelated ConfigDict deep-merge fix, i.e. pre-existing and
unrelated to it.

One real gap found and left open per user decision: F28E12 is listed
in TARGET_DEVICES but has no entry in _DEVICE_PROFILES, so
test_cross_device.py::TestCompilationProfileCorrectness still fails
for it (5 tests). Needs real hardware values (cross_compiler path,
target, target_c_mcpu, has_hard_npu) that aren't safe to guess.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tinyml-modelmaker depends on tinyml-modelzoo (pip install -e tinyml-modelzoo
in the Install dependencies step), but dependency-only changes to it
wouldn't trigger the test suite at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@musicalplatypus

Copy link
Copy Markdown
Contributor Author

CI failure is the separately-documented F28E12 gap, already called out in this PR's description — not introduced by this change.

test_cross_device.py::TestCompilationProfileCorrectness fails on F28E12: it's listed in TARGET_DEVICES but has no entry in _DEVICE_PROFILES. Left open deliberately since it needs real hardware values (cross_compiler path, target, target_c_mcpu, has_hard_npu) that aren't safe to guess at.

Worth noting this PR's own fix does work as intended: its run shows zero google_speech_command/audio-classification failures — https://github.com/TexasInstruments/tinyml-tensorlab/actions/runs/30575996012 — unlike every other currently-open PR (#18, #20, #21, #22, #23), which all still show those same 2 pre-existing failures (confirmed present on upstream/main itself, unmodified: https://github.com/TexasInstruments/tinyml-tensorlab/actions/runs/30376872256) until this one merges.

# Conflicts:
#	.github/workflows/test-modelmaker.yml
@Adithya-Thonse
Adithya-Thonse merged commit bd96b25 into TexasInstruments:main Aug 6, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants