SmolVLM encoder not getting delegated to XNNPack

[jackzhxng]

🐛 Describe the bug

SmolVLM from Optimum (PR - huggingface/optimum-executorch#163) is not getting fully delegated to XNNPack in the encoder subgraph.

The pte dump shows this - P1983888427, you can see undelegated addmms still in the encoder.

Printed EP right after torch.export: P1984977202
Printed EP right after to_edge_transform_and_lower: P1985017274

Versions

N/A

cc @GregoryComer @digantdesai @cbilgin @mergennachin

[digantdesai]

Thanks @jackzhxng for this. First, does this happen on v1.0 or only on main?

Looking at all three pastes, seems like the addmm nodes do indeed come from linear decomposition. Spot checked (transformers/models/idefics3/modeling_idefics3.py:231) for example.

This makes me question (without digging too deep) - to_edge_transform_and_lower do-no-decompose feature. Can you dig into that a bit more or @GregoryComer or I can take over.

[jackzhxng]

@digantdesai happening on this fairly recent ET nightly pin.

This is where to_edge_transform_and_lower is called - https://github.com/huggingface/optimum-executorch/blob/main/optimum/exporters/executorch/recipes/xnnpack.py#L84. It is called for both decoder and encoder, but the encoder is not getting fully delegated.