Weird input cursor positioning problem

[kingdom84521]

Purpose of the issue

Bug report

Version Information

ConEmu Version: 210912 stable
Cmder Version: 1.3.18.1106

Windows Version: Windows Enterprise 20H2

Description of the issue

Step to reproduce the error:

1. Open a normal cmder window somewhere ( the place doesn't matter )
2. Type some words until the length pass over the window's width
3. Hold backspace key until there has no characters been delete anymore.

More Description:

This error has been occured for a long long time, a least over one year. I expected it will be fixed someday in the newer version of cmder or ConEmu, but it did not. Although this looks pretty like a ConEmu error for me, but I still want to check if it really is. If there can be some developers confirm the error is a pure ConEmu error, then I will close the issue and go to ConEmu's repo and try to find more help there.

[chrisant996]

1. See if copying the latest Clink into the %CMDER_ROOT%\vendor\clink directory changes what happens.

2. What does set TERM report? Maybe the terminal is misconfigured. That could explain strange terminal output problems.

3. What font is being used?

4. What is the system code page, and what language is Windows installed using?

[chrisant996]

Also, does running clink set terminal.emulation emulate change what happens?

[chrisant996]

Also, you can test whether the problem happens in plain CMD:

1. Start a stand-alone CMD window, without using ConEmu or Cmder.
2. Type %cmder_root%\vendor\clink inject to inject Clink.
3. Type %cmder_root%\vendor\init.bat to start up the Cmder prompt, without using ConEmu.

Does the problem still occur?

(If you don't have a %cmder_root% environment variable defined, then you will need to substitute it with the path to your Cmder installation.)

I am unable to reproduce the reported problem. I'm using Clink v1.2.48 on Windows 10 21H1 in US English, with the font Fira Code NF. I've tried changing many different settings related to terminal behavior and ANSI codes. I can't make the problem happen.

[kingdom84521]

First, thank you @chrisant996 for the quick response for this issue, this problem has bothered me for a couple of while.
Second, one thing I didn't notice is--I didn't really change any settings of cmder, I even delete the whole cmder directory and re-install it, then open the application straight up, but the problem is still occuring.

I have tried those step you have mentioned, the answers or responses should all at down below:

1. Copying the latest version of Clink into the %CMDER_ROOT%\vendor\clink -> didn't change things, problem still remained
2. What does set TERM report? -> it report TERM=cygwin
3. What font is being using? -> As the defaults of cmder, consolas
4. What is the system code page, and what language is Windows installed using? -> I'm not really sure about how to check this one, but I think the code page is ANSI, and the language I am using is Traditional Chinese (zh_tw)
5. Does running clink set terminal.emulation emulate change what happens? -> didn't change things, problem still remained
6. Test whether the problem happens in plain CMD -> By following the step you have given, cmder opened successfully, and the problem didn't occur on plain CMD.

[kingdom84521]

And by noticing your windows version is not same as mine, I have update the version to 21H2, but sadly the problem is still occuring.

---

Appending Informations after trying:

ConEmu Version: 210912 stable
Cmder Version: 1.3.18.1106
Clink Version: 1.2.48
Windows Version: 21H2

[chrisant996]

The code page can be checked by running chcp. I expect it will probably report Active code page: 950 (Traditional Chinese).

I searched in the ConEmu repo issues page for "wrap" and for "same line". There are several reports of similar things, but the situations sounds a bit different.

The first thing to do is check the Real Console by pressing Win+Ctrl+Alt+Space and then try to reproduce the problem. Does the Real Console window show the same problem, or does the problem only appear in the ConEmu window?

[kingdom84521]

I have tried the real console, and yes, the problem will also show on the smaller real console window.

[chrisant996]

I have tried the real console, and yes, the problem will also show on the smaller real console window.

Interesting. Let's see if Clink can help diagnose what's happening:

1. Start a new Cmder window.
2. Run clink set debug.log_terminal true to turn on logging of all output.
3. Reproduce the problem.
4. Run clink set debug.log_terminal false to turn off the extra logging.

Then run clink info to find the log file, and attach the file to your reply. (If GitHub refuses to attach a .log file, then rename it to clink.txt.)

Or if you prefer, you can email the file privately to the email address in my GitHub profile.

The extra logging will make it possible to see the exact output of both the prompt and the input text. One of the most common reasons listed for this kind of problem is apparently incorrect escape sequences in the prompt string. The logging will make it possible to see the exact characters being output.

[kingdom84521]

Here's the log file, feel free to share to other developers.
My clink.log

The content is kinda long so I'll just put a download link here, if there are any problem while downloading, please tell me.

[kingdom84521]

For some additional information of this situation, I have asked some friend to help me test out if there will have same problem on their pc, and the result turns out to be yes. Like I said, the problem has been existed for a long long time, so plenty of my friends just gave up waiting and go to use powershell or wsl instead.

[chrisant996]

Here's the log file content, feel free to share to other developers. My clink.log

The content is kinda long so I'll just put a download link here, if there are any problem while downloading, please tell me.

Perfect, thank you! The log shows what happened.

First Problem: At column 0 of line 6, it prints two characters: the lambda λ, and then a space. But the console APIs are reporting that it took THREE columns to display the two characters.

Question 1: Does the smaller Real Console window use three columns to display the two characters? I expect it does.

Second Problem: When the "A" inputs reach column 118, then Readline prints a space and a carriage return. This is the same as what it does on my computer where there is no problem. But what is different is that on my computer the space character is printed at column 0 of line 7 -- and on your computer the space is printed at column 118 of line 6.

This is probably happening because the lambda character is a different width in the ConEmu window's font vs in the Real Console window's font. That kind of discrepancy throws off the arithmetic for figuring out columns and lines, and then Readline and ConEmu and the Real Console window get out of sync with each other.

Question 2: Can you please share a screen shot of the Fonts page in the ConEmu Settings window?

Question 3: What code page number is reported when you run the chcp command?

Question 4: What is the font used in the Real Console window? Press Win+Alt+Ctrl+Space to show the Real Console window, and then right click on its title bar and choose Properties, then find the font it's using.

I will use the answers to try to reproduce the issue on my computer.

[chrisant996]

Also:

• One way to work around the problem is to use a different font.
• Another option is to replace the lambda character with an ASCII character such as > or $ or %. If you want to try that, then it's best to not edit the Cmder scripts -- instead use the script below.

Here is a script you could copy into the %cmder_root%\config directory, to replace the lambda character:

replace_lambda.lua

-- Change this to whatever you want to replace the lambda with:
local replace_with = ">"

local p = clink.promptfilter(99)
function p:filter(prompt)
    local escaped = replace_with:gsub("%%", "%%%%") or replace_with
    prompt = prompt:gsub("m\xce\xbb ", "m"..escaped.." ")
    return prompt
end

[daxgames]

Or download the unreleased build that allows configuring the prompt and lamba char.

[kingdom84521]

Question 1: Does the smaller Real Console window use three columns to display the two characters? I expect it does.
Question 2: Can you please share a screen shot of the Fonts page in the ConEmu Settings window?
Question 3: What code page number is reported when you run the chcp command?
Question 4: What is the font used in the Real Console window?

And here is the answers of questions above:
Answer 1: Because the font size of the real console is too small, so I can't really tell you if there is that kind of problem you just told me.
Answer 2: Here is the screenshot:

Answer 3: As you thought, 950
Answer 4: This is the screenshot of the "Properties -> Fonts" window:

Since I am using Traditional Chinese as my display language, to avoid the situation from the difficulty of reading, so here is a official english name from Microsoft of my font name -- "MingLiU", if you need more explanations of the output on the window, ask me anytime.

Next. here are the solutions I have tried after last post:
(Note that before trying every solution, I have set the settings back to defaults)

1. Change cmder's font to match real console window's font(MingLiU) -> didn't work, problem still remained
2. Change real console window's font to match cmder's font(Consolas) -> didn't work, problem still remained
3. Change real console window's font to a totally new one(MS Mincho): -> didn't work, problem still remained
4. Change cmder's font to a totally new one(Roboto): -> didn't work, problem still remained
5. Change cmder's font size to match reall console window's font size: -> didn't work, problem still remained
6. Put replace_lambda.lua file into %cmder_root%\config and replace the lambda character to >: Worked well !

Sorry about I can't try the solution mentioned by @daxgames cause I don't really know how to build your project to a executable .exe file from the newest state.
I am now using the replace_lambda.lua file as a temporary solution, cause this solution is kind of a brute force to me.
If it was possible, please don't close this issue too soon, I'm still looking forward about using the original settings but can perfectly execute without any bug on Cmder!!

At last, thank you for all developers who try to solve this problem, this problem really bother me for a long time, this could be a big rescue to me, so big thank to you guys!!

[kingdom84521]

And I am still here, if you still cannot reproduce the bug and need more support, feel free to tag me anytime!

[daxgames]

@kingdom84521 no need to build ut yourself.

https://ci.appveyor.com/project/MartiUK/cmder/build/artifacts

[chrisant996]

Question 1: Does the smaller Real Console window use three columns to display the two characters? I expect it does.

Answer 1: Because the font size of the real console is too small, so I can't really tell you if there is that kind of problem you just told me.

@kingdom84521 You can change the size of the real console's font in its Properties dialog.

Here is what I see when using code page 950 on my computer:

Indeed the real console shows 3 cells used for the 2 characters, but the ConEmu window shows only 2 cells used. But on my computer the ConEmu window still successfully knows that the widths were different, and it still wraps the cursor after the right number of characters as though it had used 3 cells.

The problem about wrapping onto the same line seems like a ConEmu problem.
The problem about the width of the lambda character is due to mismatched fonts, but it's also a ConEmu problem that the lambda is rendered differently.

There is unfortunately ambiguity about the "proper" width of a range of characters in Unicode, and various CJK code pages behave differently than other code pages. I think that my US-EN installation of Windows may have different fonts for code page 950 than your ZH-TW installation does. I don't know how to force my computer to behave like yours, without installing Chinese Windows (which I don't want to do 🙃).

Since I am using Traditional Chinese as my display language, to avoid the situation from the difficulty of reading, so here is a official english name from Microsoft of my font name -- "MingLiU", if you need more explanations of the output on the window, ask me anytime.

I have a "MingLiU_HKSCS-ExtB" font and a "MingLiU-ExtB" font installed (because I installed a Chinese language pack), but it won't let me use either of them in the Real Console window.

6. Put replace_lambda.lua file into %cmder_root%\config and replace the lambda character to >: Worked well !

... I am now using the replace_lambda.lua file as a temporary solution, cause this solution is kind of a brute force to me.

At this point, we have gathered enough information to know that the issue is between ConEmu and Windows.

The upcoming version of Cmder has a built-in way to change the lambda character, because of exactly this problem between ConEmu and Windows.

The strange part is that for some reason, on your computer the width issue causes text to wrap onto the same line. That isn't what most reports about the font mismatch issues have stated, so it wasn't possible at first to recognize that you're encountering a known issue, and that the next Cmder already has a workaround.

If it was possible, please don't close this issue too soon, I'm still looking forward about using the original settings but can perfectly execute without any bug on Cmder!!

At last, thank you for all developers who try to solve this problem, this problem really bother me for a long time, this could be a big rescue to me, so big thank to you guys!!

This is not a Cmder issue, nor a Clink issue. It is a ConEmu issue. Your best option for being able to someday use the lambda character in the prompt, is if you report the issue in the ConEmu repo, and try to work with Maximus for how to be able to reproduce the problem.

[kingdom84521]

Got it! I thought this is a Cmder issue after all these try, sorry about I don't know how all these output thing are really worked down below the application, so since this is a ConEmu issue, I will then close it and try to contact to ConEmu's developers, but still I want to say thank you to you guys.

(Maybe I will re-open the issue someday later, to mention the issue is solved on ConEmu)

[chrisant996]

Also, the ConEmu site has some articles about troubleshooting and configuring Unicode and code page issues.

For example: https://conemu.github.io/en/UnicodeSupport.html#font-charset

Maybe using 65001 code page (UTF8) may help.

[chrisant996]

This appears to be the same problem as Maximus5/ConEmu#2404.
Detailed analysis is available there.

[bitdream]

Also, the ConEmu site has some articles about troubleshooting and configuring Unicode and code page issues.

For example: https://conemu.github.io/en/UnicodeSupport.html#font-charset

Maybe using 65001 code page (UTF8) may help.

Using 65001 code page doesn't help!!!
The bug is still here even if the conEmu is updated to the latest version.

[chrisant996]

@bitdream Specifically what are you seeing? Share a screen shot.
Specifically what do you do leading up to whatever problem you're encountering?
What version of Clink are you using?

Lambda issues should have been solved quite a while ago, so I wonder if you're using an old version of Clink.

[bitdream]

When I type 12345 and then backspace, it deletes from 4 rather than 5. The cursor position looks messed.
After some search, I end up with changing this line in cmder_prompt_config.lua and the problem is solved.
prompt_lambSymbol = "$"
The version of ConEmu is 230724 and I can not tell the version of Cmder. It's downloaded at 2023/12/31.
I have lived with it for almost half a year and finally I digged into it today🤣

[chrisant996]

@bitdream If you want to use the default lambda prompt character, then you could revert the prompt_lambSymbol change, and try the following instead:

clink set terminal.east_asian_ambiguous two

(You haven't shared a screen shot yet, but I'm guessing you need two based on some things you mentioned. Or you might need clink set terminal.east_asian_ambiguous one instead.)

Also, if you press Ctrl-x, Ctrl-z, then Clink will print some information about itself (including the version), and near the end will be a section something like this:

ambiguous width characters in prompt:
  CJK ambiguous characters:
        Unicode: 0x03BB, UTF8 0xCE 0xBB, reported width 1, text "λ"
    Running 'chcp 65001' can often fix width problems with these characters.
    Or you can use a different character.

Can you press Ctrl-x, Ctrl-z and then copy/paste the "ambiguous width characters in prompt" section into a reply here?

Depending on what system font your computer is using, the system font width for "λ" might be different than the font that the ConEmu terminal is actually using. That's why you might need to manually tell Clink one or two for the width.

[bitdream]

OK, here is the output. Neither clink set terminal.east_asian_ambiguous two nor one seems having any effect on my side.
version:
version 1.4.4.7fe1ca
binaries D:\tools\cmder_mini\vendor\clink
session:
session 12364
profile D:\tools\cmder_mini\config
scripts D:\tools\cmder_mini\vendor ; D:\tools\cmder_mini\vendor\clink ; D:\tools\cmder_mini\config

(Use a numeric argument for additional diagnostics; e.g. press Alt+1 first.)

[chrisant996]

Oh! You're using a very old version of Clink. How did that happen?

You said you downloaded Cmder on 2023/12/31, but then I'd expect you to have Cmder v1.3.24 which comes with Clink v1.5.1.

Maybe there was some confusion about when it was downloaded, or maybe it was downloaded from somewhere that had an old version.

East Asian Ambiguous Width Character support should be working in v1.4.12 and higher.

You can probably update Clink simply by typing clink update.

[bitdream]

The date of the unzipped files are of 2022/12/19. That's possible since this is the time I started to use this laptop.
So I have lived with it for more than one year🤣
You are correct after the update everything wokrs.
Thanks!

[chrisant996]

The date of the unzipped files are of 2022/12/19.

Ah, it was a typo in the cited date.
As of 2023/12/31, Cmder includes a version of Clink that solves the lambda problems.
But 2022/12/19 is much older and doesn't have the fixes.

Mystery solved. 🙂

It's a good idea to check for updates when there are problems. That can sometimes save a lot of time and pain.

[bitdream]

Yes...but I remembered at some earlier time I tried update once but without luck.🤦‍♂️
Anyway thanks for your active maintenance for this great project.

[DRSDavidSoft]

@bitdream I strongly suggest downloading the latest version of cmder_mini and extracting it over your (very old) installation. In addition to chris' amazing work to support λ properly, we have also had applied many improvements, fixes, and general tweaks overall to Cmder since the version you have, it's a good idea to update other parts of Cmder as well!
Always keep your Clink updated as well to enjoy all the latest and greatest available features!