Should we filter some useful models? Hard to generate any image or find one useful model.

[quizD]

config:

inference_mode: huggingface
local_deployment: minimal

If I run command:

python3 awesome_chat.py --config config.yaml --mode cli

I got these results:

[ User ]: show me a joke and a image about cat
[ Jarvis ]:  I understand your request. Firstly, I used the GPT-2 model to generate a joke for you. Unfortunately, there was an inference error and I could not generate a joke. Secondly, I used the Google/VIT-Base-Patch16-224 model to classify an image of a cat. However, there was an error in the path or URL, so I could not generate an image. Lastly, I used the PromptHero/OpenJourney model to generate an image of a cat. Unfortunately, there was an error in the path or URL, so I could not generate an image. I apologize for not being able to fulfill your request. Is there anything else I can help you with?

Always got error message like this

Traceback (most recent call last):
  File "/JARVIS/server/awesome_chat.py", line 570, in model_inference
    inference_result = huggingface_model_inference(model_id, data, task)
  File "/JARVIS/server/awesome_chat.py", line 411, in huggingface_model_inference
    img_data = image_to_bytes(img_url)
  File "/JARVIS/server/awesome_chat.py", line 213, in image_to_bytes
    load_image(img_url).save(img_byte, format="png")
  File "/anaconda3/envs/jarvis/lib/python3.8/site-packages/diffusers/utils/testing_utils.py", line 268, in load_image
    raise ValueError(
ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and public/<GENERATED>-1 is not a valid path
2023-04-04 18:18:56,609 - __mp_main__ - WARNING - Inference error: {'message': 'Incorrect path or url, URLs must start with `http://` or `https://`, and public/<GENERATED>-1 is not a valid path'}
2023-04-04 18:18:58,304 - __mp_main__ - WARNING - Inference error: unknown error

or error file path like this

2023-04-04 18:18:58,304 - __mp_main__ - WARNING - Inference error: unknown error
[Errno 2] No such file or directory: 'public/images/1788.png'
Traceback (most recent call last):
  File "/JARVIS/server/awesome_chat.py", line 570, in model_inference
    inference_result = huggingface_model_inference(model_id, data, task)
  File "/JARVIS/server/awesome_chat.py", line 346, in huggingface_model_inference
    img.save(f"public/images/{name}.png")
  File "/anaconda3/envs/jarvis/lib/python3.8/site-packages/PIL/Image.py", line 2429, in save
    fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: 'public/images/1788.png'
2023-04-04 18:19:02,531 - __mp_main__ - WARNING - Inference error: {'message': "[Errno 2] No such file or directory: 'public/images/1788.png'"}

[seshubonam]

+1

[tricktreat]

Hi, there may be several reasons for this: 1. some huggingface inference endpoints did not work in the minimal setting, such as the text-to-image model WarriorMama777/OrangeMixs in your case. 2. the task planning capability of LLM is still insufficient. For example, in your case, LLM redundantly parses out the task of image classification.

For reason#1, we recommend a local_deployment: full setting. Deploy more stable models locally for each task for Jarvis to call. For reason#2, we will perform task-specific fine-tuning of open-source LLMs in the future to further enhance their planning capabilities (currently relying on few-shot prompting).

BTW, you can see the whole process in debug.log. Jarvis is still being refined, and we welcome you to fix some bugs and submit PRs .