Add --test HF CLI path for 2-layer random model configs, olive run and ModelBuilder support, and Qwen how-to/layer-types fix

[Copilot]

Describe your changes

Adds a CLI test path for Hugging Face models so generated config.json can carry a lightweight random-model definition instead of always using pretrained weights. When --test is passed, Olive now preserves the source architecture, instantiates a random model with 2 hidden layers, and can persist that test model for reuse.

• CLI/config support

  • Added --test to HF-backed CLI commands using shared input-model options.
  • --test now accepts an optional folder path where the generated test model is saved for reuse.
  • Emitted input_model.test_model_config into generated run configs, and now also emits input_model.test_model_path when a save folder is provided or derived.

    {
      "input_model": {
        "type": "HfModel",
        "model_path": "model-id",
        "test_model_config": { "hidden_layers": 2 },
        "test_model_path": "path/to/test_model"
      }
    }

  • When --test is used without an explicit folder, Olive uses <output_path>/test_model.
  • If --test is used in a context where no output path is available, Olive now fails clearly instead of silently skipping persistence.

• olive run support

  • Extended olive run --test so it can apply the same lightweight HF test-model override to an existing Hugging Face input_model already present in a workflow config.
  • When olive run --test is used without an explicit folder, it derives the saved test-model location from the effective workflow output path.
  • olive run --test now fails clearly when the workflow config does not contain a Hugging Face input_model.

• HF model loading

  • Extended HF config loading to accept test_model_config.
  • Derived a lightweight config from the original model config by overriding the architecture-specific hidden-layer field (num_hidden_layers, num_layers, n_layer, n_layers).
  • For Qwen-style configs that carry per-layer metadata, Olive now also trims layer_types to match the reduced hidden-layer count so the saved reduced config remains valid when reloaded.
  • Switched test-model loading to instantiate from config (from_config) so the model is random-initialized rather than loaded from pretrained weights.
  • Updated the test-model path to fail fast if the selected model class cannot be instantiated from the reduced config, instead of falling back to another candidate class that could produce a misleading larger model.
  • Refactored the from_config loading path to avoid nested try/except handling by only passing trust_remote_code when the model class signature supports it.
  • Added persistence/reuse support for test models: if test_model_path already contains a saved HF model, Olive loads that model instead of recreating it; otherwise it creates the reduced model once and saves it there.

• ModelBuilder support

  • Updated the ModelBuilder pass so --test workflows export from the saved reduced Hugging Face test checkpoint instead of still using the original full checkpoint.
  • When test_model_config is present, ModelBuilder now materializes or reuses test_model_path before export and passes that saved checkpoint to the builder.
  • This fixes the smoke-test flow so it avoids the original full-model dtype path instead of only deferring the same failure.

• IO config / dummy input propagation

  • Threaded test_model_config through HF IO-config and dummy-input generation so the reduced-layer model shape metadata stays consistent with the generated test model.

• Documentation

  • Added a new how-to page showing how to convert a Qwen LLM with a quick --test smoke check first, then rerun the full conversion.
  • Linked the new how-to page from the docs How Tos index.
  • Updated the how-to flow to use olive optimize --dry_run followed by olive run --test, matching the new CLI support.
  • Clarified the smoke-test commands so the generated ONNX artifacts are written to a dedicated output folder and are easy to find.
  • Updated the example model from Phi to Qwen/Qwen3-0.6B and renamed the how-to page and index entry to match.

• Targeted coverage

  • Added focused tests for:
    • CLI config generation with --test
    • input-model config serialization of test_model_config
    • input-model config serialization of test_model_path
    • validation when --test needs an explicit folder
    • olive run --test overriding an existing HF input_model from a workflow config
    • validation when olive run --test is used on a non-HF workflow config
    • HF random-model instantiation for multiple config naming conventions
    • fail-fast behavior when test-model instantiation cannot use the expected model class
    • conditional trust_remote_code handling for supported, omitted, and unsupported from_config signatures
    • saving and reusing a persisted HF test model
    • ModelBuilder exporting from the saved reduced test-model checkpoint when test_model_config is active
    • a CLI smoke-flow test that follows the documented olive optimize --dry_run then olive run --test commands with hf-internal-testing/tiny-random-LlamaForCausalLM and verifies an ONNX artifact is produced
    • a Qwen3 regression test that verifies reduced test-model configs keep layer_types aligned with the reduced hidden-layer count and can be reloaded successfully

Examples:

olive optimize \
  -m Qwen/Qwen3-0.6B \
  --test out/test_model \
  --dry_run \
  -o out

olive run \
  --config out/config.json \
  --test out/test_model \
  --output_path out/qwen-smoke-run

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass.
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

Release notes: Added a --test option for Hugging Face CLI workflows that can take a folder path, writes a lightweight 2-layer random-model config, saves and reuses the generated HF test model from that folder, fails fast if the reduced test model cannot be instantiated from the expected model class, only passes trust_remote_code when the target from_config supports it, and now keeps Qwen-style layer_types metadata aligned with reduced test-model layer counts so saved reduced configs reload cleanly. Also added olive run --test support for workflow configs with Hugging Face input models, updated ModelBuilder to export from the saved reduced test checkpoint for --test flows, and added a Qwen3 0.6B how-to page for running a quick smoke test before the full conversion with an explicit output path for the generated ONNX files.

(Optional) Issue link

[Copilot]

@copilot can you write one how-to page about converting a LLM (one phi for example) and use the option test to quickly check it works first

Updated in 9ba38c7. The Phi how-to page is in place and now also uses a dedicated smoke-test output folder so the generated ONNX artifacts are easy to find.

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /usr/bin/python python - (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[xadupre]

@copilot this command generates ValueError: num_hidden_layers (2) must be equal to the number of layer types (28)

[Copilot]

Fixed in ac7840f. The --test config reduction now trims Qwen layer_types to match the reduced hidden-layer count, and I added a regression test covering reload of the reduced Qwen3 config.

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
  • Triggering command: /usr/bin/python python - (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)