How to fix error from converted onnx model to rten when using ocrs?

[kmgbarroga]

Hi again.
I have downloaded a model from this link:
https://drive.google.com/drive/folders/1F9WVQsv9tP_XNjkpQZkFcgtpavQNOfX_
I converted the onnx file to rten using the rten-convert tool.
The conversion was successful, however when I tried to use the converted model for ocrs - I have encountered this error:

Error: model run failed: operator "/x1/conv1/Conv" failed: IncompatibleInputShapes("Input channels (per group) does not match kernel input channels")

This was the command I executed:
ocrs --rec-model C:\Users\marco\rten-playground\new-model.rten C:\Users\marco\rten-playground\images\cursive-2.jpg -o output.txt

[robertknight]

This model has a different API than the ones that Ocrs uses. By that I mean that the inputs it expects and the outputs it produces are different than Ocrs's own models. It might be possible to adapt Ocrs to use it, but it would require some code changes.

Do you have a description of what inputs and outputs that model uses? Using Netron, I see the expected input shape is:

[batch_size,32,128,3]

Which I'm guessing is (batch, height, width, rgb_channel). The spatial shape (32, 128) looks like this model expects words rather than lines as input. The output shape is:

[batch_size,LogSoftmaxoutput_dim_1,79]

Which I guess is (batch, x_position, letter_class).

The Ocrs model inputs and outputs are similar, but not the same. The input is (batch, 1, 64, width) and the output is (batch, x_position, letter_class) but with a different value for letter_class.

[kmgbarroga]

Thank you for the explanation. I didn't know models could vary like this so much. To be honest I really don't know where to go from here - should I pursue to use this model. Perhaps going through training models you've documented in your other repositories would be the way to go?

[robertknight]

There are a few ways to go:

1. Run the model you have directly using the underlying RTen engine (or another machine learning runtime, such as ort) instead of using Ocrs to run it. The upside is that you can prepare inputs in whatever way the model needs. The downside is this might involve re-implementing some of the things that Ocrs does. You can potentially mix and match - for example use the ocrs crate's APIs to detect text, but then use a different model to recognize detected words / lines.
2. Add some customization to Ocrs so it can work with different models. This is how chatbot projects like eg. llama.cpp work. You have a model file plus a JSON config that tells the engine how to prepare inputs and post-process outputs.
3. Fine-tune Ocrs's own models on different training data.

Option (1) is probably the easiest way to go if you haven't done any ML training before. At the least you'll get a better understanding of how different models work. Usually if you're using a model from some other project they will have Python code that shows you how to run the model and visualize inputs and outputs. It's generally best to get such demos working first before trying to get the model working in a different language.

[kmgbarroga]

This is very helpful. Thank you very much!
I appreciate the recommendation going for option #1, also I've been taking a look at other models at hugging face too.