OCR and r.search() in both Japanese and English at the same time? - some ideas to try

[ManaHime]

Hello!
First of all thank you for this project!
I've been able to successfully get Japanese to work with Sikulix using the instruction here: #405
But I am in a situation where where I may need to recognize strings that have both Japanese and English characters mixed in (for example item names or item numbers)
I was wondering if that was something that was possible to setup?

Also even though the r.read() function seems to work, I don't seem to be able to succesfully get something like:
r.click('クリックしたいの何か.png")?

I'm sorry if my questions aren't asked in a better, but I was wondering if anyone has any idea of how to do those 2 things?

[marcelocecin]

Hello, we could help you more effectively if you could share a print of a practical case.

[ManaHime]

So if I write this in say paint or anywhere really:

グーグル
BCA300-AFC4+BEC415ｾｯﾄ

and then I do something simple like:

import rpa as r
r.init(visual_automation = True, chrome_browser=False)

r.keyboard('[alt][tab]')
r.snap('page.png', 'results.png')

r.vision('import org.sikuli.script.TextRecognizer as TR')
r.vision('Settings.OcrReadText = True')
r.vision('Settings.OcrLanguage = "jpn"')
r.vision('TR.reset()')

print(r.read('results.png'))

r.close()

I end up getting this as an output (I'm going to ommit the UI of whatever app for simplicity):
ダ一グル

BCA300-〈FC4+BEC4ー5セツ卜
this is not great for alpha numeric values, but I guess it does get セット (set) correctly.
if I used "eng" instead of jpg (r.vision('Settings.OcrLanguage = "eng"') ) I get this instead:
BCA300--AFC4+BEC4151?V f
this is much better for alphanumerical values but it completely scrambles the セット (set) part of the string.
Also if I try to do something like this it will actually click on it's own ABC:

import rpa as r
r.init(visual_automation = True, chrome_browser=False)

print("start")
r.snap('page.png', 'results.png')
r.read('results.png')
r.click('ABC.png')

r.close()

But if I try this instead it will not work and not do anything:

import rpa as r
r.init(visual_automation = True, chrome_browser=False)

print("start")
r.snap('page.png', 'results.png')
r.read('results.png')

r.vision('import org.sikuli.script.TextRecognizer as TR')
r.vision('Settings.OcrReadText = True')
r.vision('Settings.OcrLanguage = "jpn"')
r.vision('TR.reset()')
r.click('セット.png')

r.close()

[marcelocecin]

Hi, try to see if this other issue can help you
aisingapore/TagUI#1299
One of the solutions could be to use regions as a limiter for extracting data from each language (English/Japanese)

[kensoh]

Adding on, I don't think SikuliX engine can read both Japanese and English at the same time. I'm assuming you cannot use web web element to read from webpage because this isn't a website. No idea on this, especially that the text is beside each other.

For clicking, can you share the log files in print(r.tagui_location()) folder --> there is a tagui or .tagui folder depending on your OS, then --> src --> tagui.sikulix folder.