UTF-8 Encoding Arabic Support #40

AmrSubZero · 2016-07-07T04:56:25Z

As i know The API sends and receives the raw bytes

But when i try to display the arabic words correctly, it displays like "��" which seems to be "ISO-8859-1" or maybe "windows-1252" i don't know.

Is there an option allows me to change the API Charset Encoding to UTF-8? like the Pear2/RouterOS PHP API? (see the link).

or maybe convert the (diamond question marks) to "UTF-8" in Java? if so, please post an example!

I really need to correctly display arabic words, thanks.

GideonLeGrange · 2016-07-09T11:48:26Z

Can you please provide me with an example RouterOS config so I can reproduce the problem?

GideonLeGrange · 2016-07-11T14:21:44Z

@AmrSubZero I'm looking into this. It should be possible to add a setCharSet() call that will allow you to get strings constructed with the right charset. I need to change code deep in the implementation which will cause some refactoring. I don't want to just patch and release code without being able to test, so I have two questions for you:

Can you please provide me with a sample Mikrotik configuration with Arabic characters in?
Can you tell me what character encoding is the correct one? I'm thinking cp1256 but is it?

Let me know please, I'd like to fix this.

AmrSubZero · 2016-07-17T02:10:49Z

First, i' have created a sample configuration working on v5.25 of routerOS with arabic characters (comments column) for "ip/hotspot/users & ip bindings" so you can connect, get and display the "comment" column for those, and test with them.

In the configuration the user is admin and there's no password
the IN or WAN Ethernet IP is 192.168.1.1 and the OUT or LAN or LOCAL IP is 10.0.0.1

I'm getting connected via API by connecting to 10.0.0.1 with the user admin and no password as i mentioned above, if there's a connection problem, please let me know.

Here's the configuration file

Second, the correct character encoding is "cp1256".

I'm truly glad that you're ready to help, waiting you, thanks.

GideonLeGrange · 2016-10-08T17:09:54Z

@AmrSubZero I think I have a fix for this. If you want to test the fix, please pull the current master branch from github and build and try that version.

As an experiment, I've changed the internal encoding for transmission between the mikrotik-java API and RouterOS to be UTF-8 and it seems to solve the problem. I'll release version 3.0.3 shortly to make this change available.

Thanks to @boen_robot - your comments above pointed me in the right direction, which lead to this StackOverflow article http://stackoverflow.com/questions/5729806/encode-string-to-utf-8

A Java String is internally always encoded in UTF-16 - but you really should think about it like this: an encoding is a way to translate between Strings and bytes.
So if you have an encoding problem, by the time you have String, it's too late to fix. You need to fix the place where you create that String from a file, DB or network connection.

GideonLeGrange · 2016-10-08T18:40:07Z

@AmrSubZero

This works for me. Can you please see if it works for you as well.

Example java code that creates users with Arabic, Japanese and Cyrillic text in comments

    private static final String JAPANESE = "事報ハヤ久送とゅ歳用ト候新放すルドう二5春園ユヲロ然納レ部悲と被状クヘ芸一ーあぽだ野健い産隊ず";
    private static final String CRYLLIC = "Лорем ипсум долор сит амет, легере елояуентиам хис ид. Елигенди нолуиссе вих ут. Нихил";
    private static final String ARABIC = "تجهيز والمانيا تم قام. وحتّى المتاخمة ما وقد. أسر أمدها تكبّد عل. فقد بسبب ترتيب استدعى أم, مما مع غرّة، لأداء. الشتاء، عسكرياً";

    private void test() throws MikrotikApiException {
        con.execute("/ip/hotspot/user/add name=userJ comment='" + JAPANESE + "'");
        con.execute("/ip/hotspot/user/add name=userC comment='" + CRYLLIC + "'");
        con.execute("/ip/hotspot/user/add name=userA comment='" + ARABIC + "'");

        for (Map<String, String> res : con.execute("/ip/hotspot/user/print return name,comment")) {
            System.out.printf("%s : %s\n", res.get("name"), res.get("comment"));
        }
    }

_Output from the that code_

default-trial : counters and limits for trial users
userJ : 事報ハヤ久送とゅ歳用ト候新放すルドう二5春園ユヲロ然納レ部悲と被状クヘ芸一ーあぽだ野健い産隊ず
userC : Лорем ипсум долор сит амет, легере елояуентиам хис ид. Елигенди нолуиссе вих ут. Нихил
userA : تجهيز والمانيا تم قام. وحتّى المتاخمة ما وقد. أسر أمدها تكبّد عل. فقد بسبب ترتيب استدعى أم, مما مع غرّة، لأداء. الشتاء، عسكرياً

_Screen shot of ssh login to Mikrotik_

AmrSubZero · 2016-10-09T05:58:20Z

@GideonLeGrange

I've updated the maven repository in my android project to : compile 'me.legrange:mikrotik:3.0.3'

cleaned & rebuilt the project and tried the following examples :

private static final String ARABIC = "تجربة نص عربى";
con.execute("/ip/hotspot/user/add name=userA comment='" + ARABIC + "'");

Results :

and :

for (Map<String, String> res : con.execute("/ip/hotspot/user/print return name,comment")) { testArray.add(res.get("name") + " / " + res.get("comment"));
}

Results :

Which correctly displayed the userA arabic string in the App, but not the other users.

So we have Two problems here :

First problem : when i send a command to add user with comment it's not adding the comment in the right arabic condition it's displaying in Winbox like :

This is not the correct arabic string that i passed in the add user command.

i think sending arabic from api to mikrotik should not be in "UTF-8" encoding, or maybe it should be a specific encoding which i don't know, to be displayed correctly in Winbox.

Second problem : in that case i will need to Set all the users comments to that weird character encoding to display them correctly in the App, but i won't be able to read arabic in Winbox i will only read arabic correctly in the App which is a problem not a feature.
and still the same thing, the correct arabic comments in winbox, displaying as diamond question mark in the App.

Sending characters from API to MikroTik should be done with a specific encoding.
Reading characters from MikroTik through API should be a different encoding (UTF-8 i think).

If there anything you don't understand just let me know.

GideonLeGrange · 2016-10-09T17:28:39Z

On my setup the API and ssh command line show the same information, but Winbox and the web interface don't display them correctly, they look like in your pictures (I even enbled Arabic in my Chrome settings to see if the web interface will display correctly).

So I have a question. If you ssh in to your mikrotik and do a /ip/hotspot/user print command, how are the users shown?

GideonLeGrange · 2016-10-09T19:17:57Z

I've experimented some more and modified the API code to explicitly convert from cp1256 before sending data over the network and to convert to cp1256 when receiving data from the network (and before passing it to the user) and it makes no difference at all. I tried the same with iso-8859-6, no difference.

The Arabic text set using the API looks correct from the router (ssh) command line, and it is read back correctly. But Winbox and the web interface do not show correctly.

I suspect that Winbox and the web interface do not encode and decode the Arabic characters. I set Chrome's encoding settings (under View/Encoding) to "Arabic (Windows-1256)" and it does not make any difference either. I tried with "Arabic (iso-8859-6)" as well, no difference. This leads me to believe that it is already wrong by the time it reaches the browser.

I fixed the encoding problem in the API in 3.0.3 and at this point the API will send non-English characters correctly and read them correctly. Other software is not working and I can't fix that. I think you're going to have to decide on a workaround for this problem. The options I see are:

Trust Winbox and don't show the comments in your app.
Trust your app and ignore the fact that Winbox formats it badly.
Restrict your app and your users to English characters.

AmrSubZero · 2016-10-10T00:33:38Z

I understand that you want to fix this, and i appreciate your help.

But let's discuss it.

As i asked (this question) boen_robot told me that WinBox displays the bytes using the Windows ANSI charset which is different for every locale.

And as i know, boen_robot is the owner of the Pear2/RouterOS PHP API for Mikrotik, he has a Charset Configurable that works correctly for each user configuration, as i mentioned in this Issue the setCharset() function, the user choosing the REMOTE and LOCAL charset, to Write to MikroTik with the correct charset, and Read from MikroTik also with the correct charset.

//Here's where we specify the charset pair.
$client->setCharset(
array(
RouterOS\Communicator::CHARSET_REMOTE => 'windows-1256',
RouterOS\Communicator::CHARSET_LOCAL => 'UTF-8'
)
);

The cool thing is that i tested his API using setCharset() and it is working like a charm!

Writing to MikroTik from API, displayed the Arabic correctly in the WinBox :

Reading from MikroTik through API, also displayed the Arabic correctly in the Web :

boen_robot has recommended you to create a Charset Configurable to fix this problem :

Can that be achieved? is it possible to fix this by doing a Charset Configurable setCharset() method?

I'll be happy if that can be achieved, i have a huge data in my RouterOS the only way that i can distinction between each item and the other is the comment column and it will be too much easier to read it in arabic.

Let me know any step you take, please!

Thanks alot.

GideonLeGrange · 2016-10-10T07:08:26Z

I tried manipulating the the charset and it does not solve it. Maybe I'm doing it wrong, but
unfortunately I cannot spend more time on this.

The library is open source and you can easily implement a setCharset() method yourself. If you do so and it fixes the problem, please submit a pull request for your changes.

AmrSubZero · 2016-10-10T09:35:11Z

That is a bad news, but anyway, thanks Mr Gideon for your help.

AbdelsalamHaa · 2018-05-02T08:54:06Z

Hi , I'm using tesseract 4 with vs 2017 . i have used with English characters first , now i started to include arabic as well . the thing is got weird characters even when i change the eng.traineddata to ara.traineddata.

i found out that it's the characters when the UTF8 code is treated as Hex code .

this is the image i want to recognize .
the result is

i convert the letters in arabic to UTF8 code using this website
https://r12a.github.io/app-conversion/
and when i take this code and converted as a hex code to character i get the same characters that tesseract showed me the first time.
"Ø¹Ø¨Ø¯Ø§Ù„Ø³Ù„Ø§Ù… Ù…Ø¯ÙŠ Ø¹Ø¨Ø¯Ø§Ù„Ø¹Ø²ÙŠØ²"

I think the problem is with UTF8 or might be my visual studio can't recognize Arabic letters or what

GideonLeGrange · 2018-05-02T13:13:10Z

Hi You replied to a thread concerning the support for Arabic characters in an API used to program Mikrotik routers. I cannot help you with problems with Visual Studio. Gideon

…

On 02 May 2018, at 10:54, AbdelsalamHaa ***@***.***> wrote: Hi , I'm using tesseract 4 with vs 2017 . i have used with English characters first , now i started to include arabic as well . the thing is got weird characters even when i change the eng.traineddata to ara.traineddata. i found out that it's the characters when the UTF8 code is treated as Hex code . <https://user-images.githubusercontent.com/35866217/39512903-1cb50738-4e25-11e8-8d0b-eba47c163d8b.png> this is the image i want to recognize . the result is <https://user-images.githubusercontent.com/35866217/39513059-9a6a2424-4e25-11e8-9bcd-2eb56d3437b6.png> i convert the letters in arabic to UTF8 code using this website https://r12a.github.io/app-conversion/ <https://r12a.github.io/app-conversion/> and when i take this code and converted as a hex code to character i get the same characters that tesseract showed me the first time. "Ø¹Ø¨Ø¯Ø§Ù„Ø³Ù„Ø§Ù… Ù…Ø¯ÙŠ Ø¹Ø¨Ø¯Ø§Ù„Ø¹Ø²ÙŠØ²" I think the problem is with UTF8 or might be my visual studio can't recognize Arabic letters or what — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABFCFMclw-T_iyq1UVP_o0PdqD87yXJoks5tuXQzgaJpZM4JGvMx>.

AbdelsalamHaa · 2018-05-03T01:05:05Z

@GideonLeGrange okay thanks for your reply

mohamadtik · 2020-10-12T00:55:31Z

how i can used this code via class.php.api ????

//Here's where we specify the charset pair.
$client->setCharset(
array(
RouterOS\Communicator::CHARSET_REMOTE => 'windows-1256',
RouterOS\Communicator::CHARSET_LOCAL => 'UTF-8'
)
);

GideonLeGrange · 2020-10-12T05:51:41Z

how i can used this code via class.php.api ????

//Here's where we specify the charset pair.
$client->setCharset(
array(
RouterOS\Communicator::CHARSET_REMOTE => 'windows-1256',
RouterOS\Communicator::CHARSET_LOCAL => 'UTF-8'
)
);

I wrote and maintain a Java API. I cannot help you with a PHP API that somebody else wrote. Ask the author of that class.

GideonLeGrange closed this as completed in a6230a8 Oct 8, 2016

GideonLeGrange added a commit that referenced this issue Oct 8, 2016

Added CharSets example with reference to #40

40ebed6

GideonLeGrange reopened this Oct 9, 2016

GideonLeGrange added awaiting-feedback not a bug labels Oct 9, 2016

GideonLeGrange closed this as completed Oct 9, 2016

GideonLeGrange removed awaiting-feedback not a bug labels Oct 10, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UTF-8 Encoding Arabic Support #40

UTF-8 Encoding Arabic Support #40

AmrSubZero commented Jul 7, 2016 •

edited

GideonLeGrange commented Jul 9, 2016

GideonLeGrange commented Jul 11, 2016

AmrSubZero commented Jul 17, 2016 •

edited

GideonLeGrange commented Oct 8, 2016

GideonLeGrange commented Oct 8, 2016 •

edited

AmrSubZero commented Oct 9, 2016 •

edited

GideonLeGrange commented Oct 9, 2016

GideonLeGrange commented Oct 9, 2016 •

edited

AmrSubZero commented Oct 10, 2016 •

edited

GideonLeGrange commented Oct 10, 2016

AmrSubZero commented Oct 10, 2016

AbdelsalamHaa commented May 2, 2018 •

edited

GideonLeGrange commented May 2, 2018 via email

AbdelsalamHaa commented May 3, 2018

mohamadtik commented Oct 12, 2020

GideonLeGrange commented Oct 12, 2020

UTF-8 Encoding Arabic Support #40

UTF-8 Encoding Arabic Support #40

Comments

AmrSubZero commented Jul 7, 2016 • edited

GideonLeGrange commented Jul 9, 2016

GideonLeGrange commented Jul 11, 2016

AmrSubZero commented Jul 17, 2016 • edited

GideonLeGrange commented Oct 8, 2016

GideonLeGrange commented Oct 8, 2016 • edited

AmrSubZero commented Oct 9, 2016 • edited

GideonLeGrange commented Oct 9, 2016

GideonLeGrange commented Oct 9, 2016 • edited

AmrSubZero commented Oct 10, 2016 • edited

GideonLeGrange commented Oct 10, 2016

AmrSubZero commented Oct 10, 2016

AbdelsalamHaa commented May 2, 2018 • edited

GideonLeGrange commented May 2, 2018 via email

AbdelsalamHaa commented May 3, 2018

mohamadtik commented Oct 12, 2020

GideonLeGrange commented Oct 12, 2020

AmrSubZero commented Jul 7, 2016 •

edited

AmrSubZero commented Jul 17, 2016 •

edited

GideonLeGrange commented Oct 8, 2016 •

edited

AmrSubZero commented Oct 9, 2016 •

edited

GideonLeGrange commented Oct 9, 2016 •

edited

AmrSubZero commented Oct 10, 2016 •

edited

AbdelsalamHaa commented May 2, 2018 •

edited