[Bug]: Arabic text is showing wrongly

[ObaydaAlesawi]

Short bug description

When I debug an Arabic text, it shows wrong

Extended bug description

No response

Your project is not the problem

• I tried using a fresh laravel install - I'm fairly certain it has nothing to do with my project.

How to reproduce?

Debug an Arabic text let's say "مرحبا" and see what is showing.

Expected behavior

Tinker should display the word as it is

Screenshots

IDE & Version

PHPstorm2023.1.1

PHP Version

8.1

OS and version

Windows 10

Laravel Tinker plugin version

2.6

Interpreter type

Local

Laravel Version

9.52

Tinker version

2.7

Psysh version

0.11.2

Additional Context

No response

[Roboroads]

Might be a windows specific issue.. unsure yet..

[ObaydaAlesawi]

@Roboroads may I do anything that would help you?

[Roboroads]

@Roboroads may I do anything that would help you?

I'm not entirely sure yet what the problem is. I can't reproduce it on my linux system, the rest being the same. So, since the only difference the OS is, I have to test using Windows to flag this as unreproducible.

I did however remove something that was buggy, could you confirm when 6.2.1 comes out if the bug still persists? (Or, download the plugin from the release page and install manually to test)

[ObaydaAlesawi]

@Roboroads I've downloaded the last release https://github.com/Roboroads/laravel-tinker/releases/tag/v2.6.1 but it still has the same issue

[Roboroads]

@Algeneralo Auch. Thanks for testing tho. I'll see when I can spin up a windows to test it :)

[ObaydaAlesawi]

@Roboroads any updates on this?

[Roboroads]

@Algeneralo I was on vacation for a week. Just got back.

I don't have all the time in the world since I have 2 other jobs that are paying the bills, so those get priority, so sorry for the wait on my end.

If you want, you can also look at it yourself and send a PR this way 😊

[ObaydaAlesawi]

@Roboroads I can handle it no problem, do you have a starting point or an idea that may help me start debugging?

[kleczewsky]

I can confirm that polish letters are also being changed. Tinker in terminal works correct, but Laravel Tinker in phpstorm has this problem.

Win 10
PhpStorm 2022.3.2
Plugin ver 2.6.1

terminal:

plugin:

[ZBrettonYe]

same with chinese letters

[stale]

This issue has been automatically marked as stale because it has not had recent activity. This means this issue is resolved or nobody else has had this issue or feature request. It will be closed if no further activity occurs within 30 days, so if this is still relevant, please let us know!

[ObaydaAlesawi]

Still need to be fixed

[2rkmen]

same with cyrillic

[stale]

This issue has been automatically marked as stale because it has not had recent activity. This means this issue is resolved or nobody else has had this issue or feature request. It will be closed if no further activity occurs within 30 days, so if this is still relevant, please let us know!

[Roboroads]

Applies magic freshness

[Roboroads]

Update about this - I've currently spent about 20 hours just trying to figure this out. I'm pretty sure it's an encoding issue but have no idea how to change it. It's a windows-only issue, that's what I do know (working on a windows VM is not working in favor of speed :P).

I've posted in the JB Slack already - haven't had any useful response yet.

Just know that I'm not ignoring this issue or anything, it's actually really hard to fix for some reason.

[x777o]

Hi guys. I encountered the same problem in Russian. Like echo "Привет"; I found out that downgrading to 2.5.1 helps. I think this is related to update 2.6 - specifically "Meta tag to set charset to UTF8 in toolwindow HTML."
Thanks to the author for his work!

[ObaydaAlesawi]

Update about this - I've currently spent about 20 hours just trying to figure this out. I'm pretty sure it's an encoding issue but have no idea how to change it. It's a windows-only issue, that's what I do know (working on a windows VM is not working in favor of speed :P).

I've posted in the JB Slack already - haven't had any useful response yet.

Just know that I'm not ignoring this issue or anything, it's actually really hard to fix for some reason.

@Roboroads I'd be delighted to assist you with this matter, especially since I'm a Windows user and have encountered the same issue. Let's tackle this together

[Roboroads]

Hi guys. I encountered the same problem in Russian. Like echo "Привет"; I found out that downgrading to 2.5.1 helps. I think this is related to update 2.6 - specifically "Meta tag to set charset to UTF8 in toolwindow HTML." Thanks to the author for his work!

@x777o Yea the 2.6 update I changed the way I'm reading from the console - applying colors on the Kotlin side of the plugin. It has nothing to do with the UFT8 marker, that was a way to try and fix this bug.

Between 2.5 and 2.6 I changed the how the syntax highlighting is produced, from pattern matching the output to just using
how tinker returns colors (ANSI) and translating those for more accurate highlighting. This is where the problem lies: the JB-side of things gives me the output encoded in windows-1252 (from the windows terminal, ofc) and it's really annoying to debug why and what settings I can change.

I'd be delighted to assist you with this matter

@Algeneralo Hows your Java/Kotlin - because it's how JB is giving the plugin the output back.

EDIT: Typing this out actually gave me a new idea to try out - maybe on the PHP side, before outputting, I can encode all output into something unicode-only, and translate it back in the plugin.. Work in progress again!

[ObaydaAlesawi]

Hi guys. I encountered the same problem in Russian. Like echo "Привет"; I found out that downgrading to 2.5.1 helps. I think this is related to update 2.6 - specifically "Meta tag to set charset to UTF8 in toolwindow HTML." Thanks to the author for his work!

@x777o Yea the 2.6 update I changed the way I'm reading from the console - applying colors on the Kotlin side of the plugin. It has nothing to do with the UFT8 marker, that was a way to try and fix this bug.

Between 2.5 and 2.6 I changed the how the syntax highlighting is produced, from pattern matching the output to just using how tinker returns colors (ANSI) and translating those for more accurate highlighting. This is where the problem lies: the JB-side of things gives me the output encoded in windows-1252 (from the windows terminal, ofc) and it's really annoying to debug why and what settings I can change.

I'd be delighted to assist you with this matter

@Algeneralo Hows your Java/Kotlin - because it's how JB is giving the plugin the output back.

EDIT: Typing this out actually gave me a new idea to try out - maybe on the PHP side, before outputting, I can encode all output into something unicode-only, and translate it back in the plugin.. Work in progress again!

I hope you've managed to find a solution?

[Roboroads]

@ObaydaAlesawi Still a work in progress - I wish the JB plugin exosystem was a little more open :P
Also been sick the past 3 weeks, just got a litttle better and pikcing things back up slowly.

[Roboroads]

This new idea is bearing fruits!

[Roboroads]

Update: GOT IT.

[Roboroads]

Laravel Tinker-2.6.2-rc1.zip

Who wants to test it? Here'a a build that should fix this issue.
plugins > [Cog-icon] > Install plugin from Disk.. > find the zip

[2rkmen]

windows 10 PhpStorm 2024.1 this feature works in cyrillic. Roboroads, thank you!!
what another features should i test?

[Roboroads]

windows 10 PhpStorm 2024.1 this feature works in cyrillic.

Awesome, I had to fix dd and dump still but it's all fixed in 3da4bef :)

The solution might suprise you: instead of juist plaintext output, the output now gets b64 encoded and decoded in the plugin again. This way we're not losing characters in the encoding from UTF-8 to windows-1252 to UTF-8 (which is what causes the cyrillic and arabic characters to show up weird since they're not present in windows-1252). Now it's encoded to b64, which is windows-1252 safe, then in the plugin decoded back to UTF-8 :D

Roboroads, thank you!!
You're welcome ^^

Expect this to be released in 2.7.0 - which is releasing now. Expect JB to check it in the coming 2 days.