[Question]: How to use img2text with Ollama?

[andrealesani]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

I am trying to use vision models on Ollama such as llava or llama3.2-vision but I cannot get it working. I have a knowledge base with only one PDF document that contains 3 images. The parsing will always return a single, empty chunk.

This is what I have tried to set:

• /knowledge/dataset?id=... > Configuration > Layout recognition & OCR > LLava
• /knowledge/dataset?id=... > Action (on the specific document) > Chunk method > Layout recognition & OCR > LLava
• /user-setting/model > System model settings > Img2text model > LLava

Am I missing something?

[writinwaters]

It doesn't seem about right that you can select Layout recognition & OCR in chunk method. Layout recognition & OCR is a step before chunking. What happens if you select DeepDoc in Layout recognition & OCR and General in chunk method?

[andrealesani]

Sorry, that might be a misunderstanding. When you want to change a document's parsing settings from Ragflow's UI, you have to go to:
/knowledge/dataset?id=... > Action (on the specific document) > Chunk method > Layout recognition & OCR > LLava

Actually, now in version 17 (I was using 16, but even 17 now behaves in the same way) the name has changed to:
/knowledge/dataset?id=... > Action (on the specific document) > Chunk method > Document parser > LLava

If you wanted to change the actual chunking method, you would need to go to:
/knowledge/dataset?id=... > Action (on the specific document) > Chunk method > Chunk method (again !!) > Select General/Book/etc.

So, I am trying to specify in every possible field (general settings, knowledge base settings, document settings) to use LLava (or LLama3.2 vision) to interpretate images, and I am not sure it is working correctly. I am also not sure of what to expect from the outcome, should it just behave like an "enhanced OCR" or can it also output a general description of the image? Because that would be quite useful for my use case.

[KevinHuSh]

You can pull the latest nightly version of docker image the next day your time.
The feature that utilizes CV models to describe figures in PDF will be fullfilled soon.