[iOS] support for multimodal

[Kathryn-cat]

This PR introduces multimodality for iOS. Specifically, below is a demo of running MiniGPT on iOS.

minigpt_ios.mov

Changes:

• standalone image_embed.cc and image_module.py for image-module-related functionalities
• support uploading images or photo taking in iOS
• prefillImage function in LLMChat.mm handling conversion from UIImage* to void* to tvm::runtime::NDArray
• add a image pre-processing module in relax_model

Update:

• did not add minigpt model to the app-config.json file cuz it would affect users. let's add it in a followup pr after we upload the tuned minigpt model to HF

[tqchen]

cc @MasterJH5574 @yzh119 @cyx-6 , would be great if you can help review

[Kathryn-cat]

Just a heads up, I didn't change the app-config.json file because I would like to formally add minigpt once we upload the tuned module to HF, so that it won't affect users right now.
If you would like some local testing, I can share the commands.

[Kathryn-cat]

I'm planning to introduce chat.generate() python high-level API in a followup PR, because the API might need some further discussions. Let me know if everything is good in this PR.

[dylanbeadle]

@Kathryn-cat when (or where) can we expect to find minigpt4-7b-q3f16_0? I would like to test it if possible.

edit: found this, so I'll keep an eye on it there: #679

[Kathryn-cat]

Hey @dylanbeadle , minigpt4-7b-q3f16_0 is just for testing purpose. You can build minigpt4 with q3f16_0 (might be a slightly outdated quantization method) and test it in iOS. I'll work on testing the image-multimodal part on iOS soon too.

[dylanbeadle]

Hi @Kathryn-cat , I tried to build minigpt4 from https://huggingface.co/wangrongsheng/MiniGPT-4-LLaMA-7B using python3 build.py --model ../../mlc/MiniGPT-4-LLaMA-7B --quantization q3f16_0 --target metal but got "Unsupported model: {model_name}" (presumably because the model name does not start with 'minigpt' (from relax_model/minigpt.py", line 509). ANy suggestions or pointers on the correct(?) base MiniGPT4 model I should start with?

[Lurrobert]

Great work! 🎉

[kumar-abhi]

Hi @Kathryn-cat, I tried to build miniGPT-4 from https://huggingface.co/wangrongsheng/MiniGPT-4-LLaMA-7B but getting error as attached screenshot. Any suggestions or pointers to resolving this please.

[amol-prakash]

Hey @Kathryn-cat, I tried building MiniGPT-4 using the below URLs:

Url1 - https://huggingface.co/Vision-CAIR/MiniGPT-4
Url2 - https://huggingface.co/wangrongsheng/MiniGPT-4-LLaMA-7B

I'm getting below errors while doing this process. Any suggestions or pointers to resolving this please.

<username>@USER mlc-llm % python3 -m mlc_llm.build --hf-path /Vision-CAIR/MiniGPT-4 --target iphone --max-seq-len 768 --quantization q4f16_1
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
	log
Cloning into 'dist/models/MiniGPT-4'...
fatal: repository 'https://huggingface.co//Vision-CAIR/MiniGPT-4/' not found
Downloaded weights to dist/models/MiniGPT-4
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/build.py", line 47, in <module>
    main()
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/build.py", line 41, in main
    parsed_args = core._parse_args(parsed_args)  # pylint: disable=protected-access
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/core.py", line 429, in _parse_args
    parsed = _setup_model_path(parsed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/core.py", line 471, in _setup_model_path
    validate_config(args.model_path)
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/core.py", line 514, in validate_config
    assert os.path.exists(
AssertionError: Expecting HuggingFace config, but file not found: dist/models/MiniGPT-4/config.json.
<username>@USER mlc-llm % python3 -m mlc_llm.build --hf-path /wangrongsheng/MiniGPT-4-LLaMA-7B --target iphone --max-seq-len 768 --quantization q4f16_1
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
	log
Cloning into 'dist/models/MiniGPT-4-LLaMA-7B'...
fatal: repository 'https://huggingface.co//wangrongsheng/MiniGPT-4-LLaMA-7B/' not found
Downloaded weights to dist/models/MiniGPT-4-LLaMA-7B
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/build.py", line 47, in <module>
    main()
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/build.py", line 41, in main
    parsed_args = core._parse_args(parsed_args)  # pylint: disable=protected-access
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/core.py", line 429, in _parse_args
    parsed = _setup_model_path(parsed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/core.py", line 471, in _setup_model_path
    validate_config(args.model_path)
  File "/Users/<username>/Projects/Training Projects/mlc-llm/mlc-llm/mlc_llm/core.py", line 514, in validate_config
    assert os.path.exists(
AssertionError: Expecting HuggingFace config, but file not found: dist/models/MiniGPT-4-LLaMA-7B/config.json.

[dylanbeadle]

@amol-prakash seems you have an extra starting “/“ in your —hg-path parameter and the repos are not found. But even after fixing that, you may run into a similar issue as the one I posted above (#524 (comment)).