recipe(open-vakgyata): add CPU audio-classification recipes#1131
Conversation
Independent reviewer verdict: REQUEST_CHANGESScope reviewed: only Required owner-routable changes
Independently verified evidence
This is fixable evidence/reporting and learner-capture work, not a structural recipe failure. Keep the PR in DRAFT after the loop-back. |
|
Resolved the explainer-facing
The PR remains DRAFT with |
Independent reviewer verdict: APPROVEScope reviewed: only Remediation verification
Full gate evidence
APPROVE. Required CPU/cpu fp32 and fp16 tuples are freshly verified through Goal L2 with full coverage and no deferred tuples. Leave PR #1131 in DRAFT. |
Summary
This adds CPU fp32 and fp16 coverage recipes for
onecxi/open-vakgyata, a Wav2Vec2 audio-classification checkpoint for identifying 10 Indian speech languages. Effort L0 / Outcome L0 ships recipe-only coverage for both required tuples. The committed Goal L2 was reached withL2 PASSfor both precisions; supplementary task evaluation remainsCLI-BLOCKEDbecausewinml evalhas no registeredaudio-classificationevaluator.Model metadata
What the model does
open-vakgyatais a speech language-identification model: it accepts a normalized 16 kHz mono waveform and returns logits over 10 Indian speech-language labels (en-IN,hi-IN,or-IN,bn-IN,ta-IN,te-IN,kn-IN,ml-IN,mr-IN, andgu-IN). Confidence: verified.Wav2Vec2ForSequenceClassification, 10 labels, andid2label; the pinned preprocessor config declares 16 kHz normalization. The measured baseline interface wasinput_values float32[1,16000] -> logits float32[1,10].Primary user stories
id2labelinference; the pinned checkpoint config defines the 10 locale-code labels.Supported tasks
audio-classification, exposed by the checkpoint, Transformers, Optimum ONNX, and WinML. Confidence: verified. Evidence: the pinned checkpoint haspipeline_tag=audio-classification;Wav2Vec2Configresolves throughAutoModelForAudioClassificationtoWav2Vec2ForSequenceClassification; the Optimum probe includesaudio-classification; and WinML at the pinned baseline resolvesaudio-classification,Wav2Vec2OnnxConfig, andWinMLModelForGenericTaskand completes a recipe-free build.Model architecture
Wav2Vec2ModelandWav2Vec2ForSequenceClassificationsource, and the measured export's concrete class, 58,746,794 parameters, traced hierarchy, and 212/212 tagged nodes. Confidence: verified.Validation and support evidence
Baseline
microsoft/winml-cli@e7509b1e908c74beff0a5655b8f8d7de69c5afae,winml, version 0.2.0.PASS; current main automatically resolvedtask=audio-classification,loader.model_class=AutoModelForAudioClassification, and concreteWav2Vec2ForSequenceClassification. No task, loader, architecture, or metadata override was needed. The generated config was the starting recipe.input_values FLOAT[1,16000] -> logits FLOAT[1,10]; ONNX opset 17; 88 FLOAT and 0 FLOAT16 initializers.CPUExecutionProvider, fp32, 100 iterations; mean 34.886 ms, p50 33.872 ms, p90 38.320 ms, throughput 28.66 samples/s; RAM model-load +165.34 MB, inference +106.93 MB, total +272.28 MB. OpenVINO plugin registration emitted a missingonnxruntime_providers_shared.dllerror, but CPU benchmarking continued and completed successfully.Evaluation failed: Task 'audio-classification' is not supported.The initial invocation without--model-idexited 2 withWhen using an ONNX file, --model-id is required for preprocessor and config resolution.This task-registry blocker occurs independently of dataset selection. The checkpoint supplies no evaluation dataset revision/config/split, audio-column mapping, label mapping, or task metric.VENDOR-ONLY. Vendor and post-WinML task lists were identical:audio-classification,audio-frame-classification,audio-xvector,automatic-speech-recognition, andfeature-extraction; WinML added no task.Goal
audio-classificationeval registry and absent checkpoint dataset metadata are non-blocking L3/eval limitations rather than a blocking question.Outcome
L2 PASS.examples/recipes/onecxi_open-vakgyata/cpu/cpu/audio-classification_fp32_config.jsonexamples/recipes/onecxi_open-vakgyata/cpu/cpu/audio-classification_fp16_config.jsonwav2vec2-003was captured separately: the standard stable-LayerNormWav2Vec2ForSequenceClassificationcheckpoint resolves without overrides, reaches CPU L2 parity at fp32 and fp16, and hasAdd=35,Mul=24, andTranspose=23as the top-three operator counts in both analyzed artifacts.CLI-BLOCKEDfor both tuples becauseaudio-classificationhas no registered WinML evaluator; the requested capability is an audio-classification schema, dataset adapter, preprocessing path, and task metric evaluator.Per-EP/device/precision results — including perf and eval data
Goal ladder
Wav2Vec2FeatureExtractor-normalized named input from a valid 16 kHz mono 16-bit PCM WAV.L0 structural results
onnx.checkerfull-check PASS; IR 8; ai.onnx opset 17;input_values FLOAT[1,16000] -> logits FLOAT[1,10]; initializers: FLOAT 88, INT64 6; external data 234,981,376 bytes and colocated; fresh recipe/config semantic match true; random-head warning false.onnx.checkerfull-check PASS; IR 8; ai.onnx opset 17;input_values FLOAT[1,16000] -> logits FLOAT[1,10]; initializers: FLOAT16 88, INT64 6; external data 117,490,688 bytes and colocated; fresh recipe/config semantic match true; random-head warning false.Perf results
The provider snapshot contained
DmlExecutionProviderandCPUExecutionProvider; each measured run selectedCPUExecutionProvider.L2 numeric results
The reference was the real
Wav2Vec2ForSequenceClassificationcheckpoint pinned atf2754058e485dfc65cc62589b8d0e21c1d328399. Input was namedinput_values, shape[1,16000], dtypetensor(float), from a normalized valid 16 kHz mono 16-bit PCM WAV containing 16,000 samples.Eval results
Schema probe verdict:
CLI-FEATURE-GAP, exit 2.winml evalreports thataudio-classificationis unsupported and rejects the task before dataset selection.audio-classificationhas no registeredwinml evalevaluator; rejection occurs before dataset selection. The CLI also warns that--precision fp32is ignored for pre-built ONNX inputs because precision is already baked into the model.audio-classificationhas no registeredwinml evalevaluator; rejection occurs before dataset selection. The CLI also warns that--precision fp16is ignored for pre-built ONNX inputs because precision is already baked into the model.Delta
_notemembers.replaceat/quant, fromnullto the following existing WinML-generated precision configuration:{"mode":"fp16","samples":10,"calibration_method":"minmax","weight_type":"uint8","activation_type":"uint8","per_channel":false,"symmetric":false,"weight_symmetric":null,"activation_symmetric":null,"save_calibration":false,"distribution":"uniform","seed":null,"calibration_load_path":null,"calibration_save_path":null,"op_types_to_quantize":null,"nodes_to_exclude":null,"task":"audio-classification","model_id":"onecxi/open-vakgyata","model_type":"wav2vec2","fp16_keep_io_types":true,"fp16_op_block_list":null}NOT-REQUIRED. Current main already resolves architecture, task, loader, input, and output without overrides, and no generalized source behavior was added or needed.recipe; no safe missing metadata/config/architecture-derived class rule exists to implement.examples/recipes/README.mdremains untouched.Analyze summary — component level and op level
Tester status: PASS for the complete emitted JSON and retained classifications; this is static rule compatibility analysis, not runtime execution. A fresh source checkout contains only the runtime-rule README, so public reproduction requires the versioned
rules-v0.2.0.ziparchive, SHA-256 check, 1,746-parquet extraction check, andWINMLCLI_RULES_DIRsetup shown below. On this host the analyzer process exits 1 because optional provider registration fails, but still emits complete 11-row JSON whose metadata and classifications exactly match the retained internal rule-backed evidence. This is an honest partial process success, not proof of runtime execution; no component-mapping gaps remain.Component-level summary
mappedDiv,Erf,Add,Mul; unsupported nonemappedDiv,Erf,Add,Mul; unsupported noneOp-level summary
Div,Erf,Add,Mul; unsupported none.Div,Erf,Add,Mul; unsupported none.Rule-less EPs — CUDA GPU, MIGraphX GPU, DirectML GPU, CPUExecutionProvider CPU, and Vitis AI NPU — classify all operator types as unknown for both artifacts.
Reproduce commands
The block below is the tester-supplied portable reproduction. It includes the exact version/HEAD and detached-baseline checks, recipe-free baseline build, fp32/fp16
winml configcommands, versioned analyzer-rule prerequisite, complete L2 harness, perf runs, and the expected eval-blocker probes.Show complete portable PowerShell reproduction