Qualcomm AI Engine Direct - [GenAI Pipeline] PR4: Adapter interfaces, default implementations & dataset providers

[qti-horodnic]

Summary

This PR adds the adapter layer that wraps external APIs (ExecuTorch, QNN SDK, HuggingFace) behind injectable Protocol interfaces for testability. Each strategy implementation (in subsequent PRs) delegates to these adapters rather than calling external APIs directly, enabling unit tests with mocked dependencies.

What's included

Adapter Protocols (6 files):

• QuantizerAdapter: Protocol wrapping make_quantizer, prepare_pt2e, calibrate, convert_pt2e
• CompilerAdapter: Protocol wrapping ExportSession compilation flow + CompilationResult dataclass
• DeviceRunnerAdapter: Protocol wrapping SimpleADB push/execute/pull + InferenceResult dataclass
• ModelLoaderAdapter: Protocol wrapping HuggingFace model/tokenizer loading
• CalibrationDataAdapter: Protocol for calibration dataset construction
• TrainingDataAdapter: Protocol for QAT training data (yields (features, labels) pairs)

Default Implementations (6 files):

• DefaultQuantizerAdapter: Delegates to export_utils.make_quantizer + torchao.quantization.pt2e
• DefaultCompilerAdapter: Placeholder for recipe-based compilation (depends on ExportRecipe/ExportSession APIs not yet available). Raises NotImplementedError with guidance to inject a custom CompilerAdapter using to_edge_transform_and_lower_to_qnn.
• DefaultDeviceRunnerAdapter: Delegates to SimpleADB for on-device execution
• DefaultModelLoaderAdapter: Delegates to HuggingFace AutoModelForCausalLM + AutoTokenizer
• DefaultCalibrationDataAdapter: Random token sequences; accepts a caller-supplied dataset or DataLoader via extra_options["dataset"]
• DefaultTrainingDataAdapter: Pass-through for caller-supplied QAT data; raises ValueError if absent (labelled data can't be synthesized)

Configuration:

• .coveragerc updated to omit default_*_adapter.py files (integration-test-only, require real SDK/hardware)
• __init__.py files updated to export new adapter types

New datasets/ package

Dataset providers are a cross-stage concern, not a model-preparation detail — the same corpus feeds PTQ calibration during quantization and
on-device result evaluation during inference (including pre-built .pte flows where model preparation never runs).
They therefore live in a top-level datasets/ package rather than under strategies/model_preparation/.

PR Review Checklist

• All new classes follow single responsibility (one class per file) - Yes.
• All dependencies are injected via constructor with sensible defaults - Yes.
• All external calls are behind injectable interfaces - Yes (Protocol pattern).
• Unit tests cover every public method - Yes.
• No existing files are modified (Phase 1 constraint) - Yes (only existing files modified are those added in previous GenAI prs).
• Docstrings on all public classes and methods - Yes.
• Type annotations on all function signatures - Yes.
• Logging follows the strategy in the LLD - Yes (lazy imports, debug-level logging in defaults).

Related PRs

• PR 1: Core data model, engine routing & exceptions: Qualcomm AI Engine Direct - [GenAI Pipeline] PR1: Core Data Model, Engine Routing & Exceptions #20409
• PR 2: Strategy interfaces & stage wrappers: Qualcomm AI Engine Direct - [GenAI Pipeline] PR2: Strategy Interfaces & Stage Wrappers #20795
• PR 3: Pipeline orchestrator: Qualcomm AI Engine Direct - [GenAI Pipeline] PR3: Pipeline Orchestrator #21149
• PR 4: Adapter interfaces, default implementations & dataset providers: this pr.
• PR 5: Model preparation & quantization strategies: pending
• PR 6: Compilation & inference strategy implementations: pending.
• PR 7: Integration & E2E tests: pending.

Test plan

python -m pytest \
  backends/qualcomm/genai_pipeline/tests/ \
  -v

All existing tests continue to pass (no regressions).

Test Coverage

Command to run:

python -m pytest backends/qualcomm/genai_pipeline/tests/ \
  --cov=backends/qualcomm/genai_pipeline \
  --cov-config=backends/qualcomm/.coveragerc \
  --cov-report=term-missing

Result:

Name                                                                                                     Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------------------------------------------------------------------------------
backends/qualcomm/genai_pipeline/configs/compilation_input_config.py                                        11      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/compilation_output_config.py                                        8      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/inference_input_config.py                                          12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/inference_output_config.py                                          9      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py                                   7      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py                                 11      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/quantization_input_config.py                                       12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/quantization_output_config.py                                       5      0      0      0   100%
backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py                                        5      0      0      0   100%
backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py                               26      0      6      0   100%
backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py                                  14      0      2      0   100%
backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py                                           5      0      0      0   100%
backends/qualcomm/genai_pipeline/engine_proxy.py                                                            20      0      4      0   100%
backends/qualcomm/genai_pipeline/exceptions.py                                                              20      0      6      0   100%
backends/qualcomm/genai_pipeline/genai_pipeline.py                                                          99      7     12      1    93%   190-201
backends/qualcomm/genai_pipeline/pipeline_context.py                                                        52      0     14      0   100%
backends/qualcomm/genai_pipeline/pipeline_stage.py                                                           5      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/compilation_stage.py                                                14      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/inference_stage.py                                                  14      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py                                          14      2      0      0    86%   30, 37
backends/qualcomm/genai_pipeline/stages/quantization_stage.py                                               14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py                              7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py                                 11      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py                   7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py                              14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py                       7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py                                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py       7      1      0      0    86%   40
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py                        8      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py                 7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py                            7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py                                9      0      0      0   100%
----------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                      475     10     44      1    98%                                                                                                    417     10     30      1    98%

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21751

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 3aac96f with merge base c56e6bf ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[qti-horodnic]

@pytorchbot label "release notes: qualcomm"

[psiddh]

@qti-horodnic Firstly, this is a great refactor, modular, composable, and creating abstractions for all external
dependencies. This will also help a lot with unit testing things. In general +1 to this direction. I have a few high-level thoughts, nothing blocking, mostly about what gets frozen here vs. what lands in PRs 5-7.

[psiddh]

@qti-horodnic Firstly, this is a great refactor, modular, composable, and creating abstractions for all external dependencies. This will also help a lot with unit testing things. In general +1 to this direction. I have a few high-level thoughts, nothing blocking, mostly about what gets frozen here vs. what lands in PRs 5-7.

1. Per-model transforms and the registry. The loader docstring says per-model graph/weight transforms will be "declared as data on the model registry entry" rather than via adapter subclasses, which is great imo. One thing that might be worth doing before the interfaces settle: today those transforms aren't referenceable yet — convert_linear_to_conv2d is a shared function, but things like the Gemma RMSNorm +1 offset and the partial-RoPE permute are inline in _prepare_model. Could we extract them into named transforms first, then add the transform-list column? Also worth noting one of them is the HF-module to static flat-KV-module swap, which means a transform is Module -> Module rather than an in-place mutation, that may constrain the signature.

2. Related: LLMModelConfig / @register_llm_model in examples/qualcomm/oss_scripts/llama/ is already a registry (carries repo_id, quant_recipe, num_sharding, convert_weights). Is the plan to extend that one, or introduce a new one? Just want to avoid ending up with two sources of per-model truth.

3. No "graph set" concept : One weight set can expand into N graphs, and N is a strategy decision: static_llama yields
   prefill / decode / token-embedding / calibrate-only; a single multi-token graph collapses to one. Every adapter here assumes N=1 structurally, and the fan-out , plus the cross-graph encoding copy that's required whenever N>1 — is punted to strategies that don't exist yet, so each will re-derive the same set. A graph-set type (like build_qnn_llm_graphset) is the missing abstraction; N=1 is just its degenerate case. Wdyt ?

4. Are we replacing llama.py with this new GenAI pipeline, or will both paths exist?

5. DefaultCompilerAdapter. Small one, the default currently raises NotImplementedError and the working path is "inject your own." Would it make sense to have the proven to_edge_transform_and_lower_to_qnn path be the default (or) following PRs will fill it ?

[qti-horodnic]

@psiddh
Thanks for taking the time to review and give feedback, I appreciate it. I'll address each one of your comments in a follow-up comment below, but before that let me just provide a brief overview of the project's structure.

Phase 1: The current work, includes PRs 1-7 as outlined in the PR description. Includes only the addition of the skeleton of the new GenAI infrastructure. No existing code paths change behavior, and the skeleton isn't wired into any current entry point. This phase is purely additive and inert by construction, which is why several interfaces here are single-graph / stub-bodied: they're the N=1 degenerate case, with the general form landing in Phase 2.

Phase 2: The next phase will include 4 PRs, divided between me and @DannyYuyang-quic into 2 (roughly) parallel work streams. This phase will include moving legacy code (e.g. llama.py) behind the adapters with a compat shim. The bulk of the logic implementation happens in this phase: multi-graph export, encoding reconciliation, the working compiler adapter, and the device runner.
Note that the existing code will stay exactly where it is in this phase, no code is removed, in order to maintain backwards compatibility.

Phase 3: Cleanup of old code. This will be done after only phase 2 has been completed and all critical code paths have been validated successfully. Nothing is deleted before this phase.

[qti-horodnic]

1. Agreed. I looked at _prepare_model more closely and there are actually four kinds: state_dict mutations pre-load, one that needs the constructed module (the RoPE permute reads head counts), in-place module mutations, and Module -> Module. Ordering between them is load bearing, so it'll be two lists rather than a flat transforms column. Extracting them as named functions in Phase 2, before anything depends on the shape.

On the HF -> static flat-KV swap: I'd keep that out of the transform list since it's "which class to construct" and is already registry data. Separating the two keeps the Module -> Module constraint from spreading to everything else.

2. I plan to extend LLMModelConfig. This will be done as part of phase 2 as well.

3. Agreed, and it's already the frozen interface between the phase 2 work streams.

4. As mentioned in my comment above, llama.py stays as is through phase 2. It will be removed in phase 3, gated on parity testing the functionality.

5. Good catch, the docstring is inaccurate. The real implementation against those APIs is in PR6 in this phase, so the default is functional there rather than "inject your own". Fixed the docstring here.

[psiddh]

Merging it now (inert for now) , as it unblocks the next few PRs