Jni layer use multimodal runner

[kirklandsign]

No description provided.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13825

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 62 Pending

As of commit bb2bfe0 with merge base b759ae8 ():

NEW FAILURES - The following jobs have failed:

• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 0c95af926040f46d479f8da996a97fc99899cfe85bd4bd44ff1decf749a704a1 /exec failed with exit code 127
• trunk / test-huggingface-transformers-macos (olmo-1b|coreml_fp32_gpu|--quantize) / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5_1_5b, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t b2bb345fd39a4d68d1308aa9c9fc9d654b4887b9d4a838875ad53ba60acdf5c6 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[jackzhxng]

Nice, how did you test?

[kirklandsign]

Nice, how did you test?

Honestly not sure how to test multimodal. I can just check for llama regression for now.

Maybe llava? Do you have idea?

[jackzhxng]

Yeah, I think just checking Llama for regression would be good for now

[kirklandsign]

Yeah, I think just checking Llama for regression would be good for now

Validated

[mergennachin]

@kirklandsign Is there a test for jni layer?

[kirklandsign]

@kirklandsign Is there a test for jni layer?

No. We use E2E CI.

However, I have trouble with validating this. Multimodal runner requires method get_max_seq_len but when I tried to export llava, seems this method is missing.

[kirklandsign]

I have some difficulty validating this. Getting error

Attempted to resize a static tensor. Expected shape (1), but received (4).
Error resizing tensor at input 0

executorch/extension/llm/runner/multimodal_prefiller.cpp

Lines 110 to 111 in 41c299f

	auto prefill_result = module_->execute(
	kTextModelMethod, {cache_position_tensor, encoder_output});

[jackzhxng]

@kirklandsign is this for text prefill or image prefill? If it's text prefill, then you should check if the encoder embeddings has a batch dimension

[kirklandsign]

@kirklandsign is this for text prefill or image prefill? If it's text prefill, then you should check if the encoder embeddings has a batch dimension

text.

[kirklandsign]

Now the part I'm not sure is

executorch/extension/llm/runner/multimodal_prefiller.cpp

Line 108 in 41c299f

{static_cast<int>(seq_len)},

which is different from llava runner

executorch/examples/models/llava/runner/llava_text_decoder_runner.h

Line 34 in 19d22ab

&start_pos, {1}, executorch::aten::ScalarType::Long);

Otherwise we can merge this PR