Help needed to export Phi3v in ONNX

[ladanisavan]

Hi there,

I'm seeking guidance on exporting a custom fine-tuned Phi-3 Vision model to ONNX. I've followed the ONNX build model guide from this link.

The build command I used was:
python3 -m onnxruntime_genai.models.builder -i ep_2_grad_32_lr_3e-5/ -o onnx_output/ -p int4 -e cuda --extra_options int4_block_size=32 int4_accuracy_level=4

The build process was successful and generated the following files:

• genai_config.json
• model.onnx
• model.onnx.data
• special_tokens_map.json
• tokenizer.json
• tokenizer_config.json

However, the number of files generated doesn't match the file count in the official HF repo for ONNX microsoft/Phi-3-vision-128k-instruct-onnx-cuda

Files highlighted in red below are missing:

Additionally, while loading the model using ONNX Runtime, the following error occurs:
OrtException: Load model from onnx_output failed: Protobuf parsing failed.

I have also noticed that sections for "embedding" and "vision" are missing from the genai_config.json

Can someone help me identify if I'm missing anything? Thanks

[kunal-vaishnavi]

The Phi-3 vision ONNX models are created as follows.

• The vision component (phi-3-v-128k-instruct-vision.onnx) is created using torch.onnx.export with some modifications to the original PyTorch source code.
• The text embedding component (phi-3-v-128k-instruct-text-embedding.onnx) is created using the ONNX helper APIs.
• The text component (phi-3-v-128k-instruct-text.onnx) is created using the model builder with --extra_options exclude_embeds=true enabled. The model builder prints a warning that only the text component is created.
  

  onnxruntime-genai/src/python/py/models/builder.py

  Lines 2387 to 2388 in 00ceb80

  	elif config.architectures[0] == "Phi3VForCausalLM":
  	print("WARNING: This is only generating the text component of the model. Setting `--extra_options exclude_embeds=true` by default.")

• The genai_config.json and processor_config.json are created manually.

I can open-source the scripts used to create these ONNX models and run them with ONNX Runtime GenAI.

[2U1]

@kunal-vaishnavi If you open-source it, I would really appreciate it!

[ladanisavan]

@kunal-vaishnavi open-source scripts would be really helpful to the Phi community.

[kunal-vaishnavi]

I have uploaded the necessary files in each of the Hugging Face repos and created this PR to show how to use them.

• microsoft/Phi-3-vision-128k-instruct-onnx-cpu
• microsoft/Phi-3-vision-128k-instruct-onnx-cuda
• microsoft/Phi-3-vision-128k-instruct-onnx-directml