Overwrite previous character in Japanese IME #115

tac0x2a · 2019-05-15T16:52:18Z

Hello.

In writing multi-byte character text by Japanese IME (Google IME), some time overwrote previous character.

This likely to happen in poor cpu machine.
This example is captured in SurfaceGo(Pentium Gold 4415Y).
But, This is hard to occur in Thinkpad X1 Extreme(Core i7-8850H).

When disable Awsome Emacs Keymap extension, this issue become not occurd.

Thanks.

whitphx · 2019-05-21T03:57:08Z

Thank you for reporting the issue. I will tackle this.

memo

Related info:

~~issue with Chinese IME VSCodeVim/Vim#341~~ -> This is issue not in insert mode
- PR: Fix #341 CJK Problem. VSCodeVim/Vim#446
Korean characters overlap each other VSCodeVim/Vim#1287
- PR: Fix #1287: CJK characters(korean) overlap each other in insert mode VSCodeVim/Vim#3214
- This seems not helpful. This modification is just to make composition event handlers disabled during insert mode. In our case, on the other hand, this issue occurs even though these composition handlers are not bound.
Fix Chinese input through IME aioutecism/amVim-for-VSCode#111
- PR: aioutecism/amVim-for-VSCode@f38d5c6

whitphx · 2019-08-27T16:06:39Z

Background

Type/Composition events with Japanese IME

In case the user types "あい" then press Enter:

When user types "あ"
1. "compositionStart" event is invoked
2. "type" event is invoked with {"text": "あ"}
When user types "い"
1. "replacePreviousChar" event is invoked with {"text": "あい", "replaceCharCnt" :1}
When user types Enter
1. "compositionEnd" event is invoked.

This plugin's event binding

This plugin binds its original event handler to type event.
- This is for Feature/new line with prefix argument #78
- In the event handler, after prefix arguments are processed, the process is delegated to the normal default:type command.

Hypothesis

If processing prefix arguments takes too much time in type event handler somehow, replacePreviousChar can be invoked before default:type command.
This can make the previous character deleted.

Possible solution

To ensure that ALL replacePreviousChar events between compositionStart and compositionEnd are invoked AFTER default:type.

whitphx · 2019-12-10T06:18:18Z

Memo

Env

Version: 1.40.2
Commit: f359dd69833dd8800b54d458f6d37ab7c78df520
Date: 2019-11-25T14:52:45.129Z
Electron: 6.1.5
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Darwin x64 18.7.0

Set the log level to debug

Action

Create new tab (command+n)
Type * w/o IME
Switch the IME on.
Input AL with IME (type shift-a, shift-l)
The resultant text was *AAL, whereas it should be * AL.

Log at the bug case:

(The resultant text was *AAL)


console.ts:137 [Extension Host] debug: [command]	 Command executed: "type"
console.ts:137 [Extension Host] debug: [type command]	 args.text = "*"
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.handleType]	 Not in prefix argument mode. exit.
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.cancel]
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Single char (text: "*", prefix argument: undefined).
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Execute "default:type" (text: "*")
console.ts:137 [Extension Host] debug: [command]	 Command executed: "type"
console.ts:137 [Extension Host] debug: [type command]	 args.text = " "
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.handleType]	 Not in prefix argument mode. exit.
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.cancel]
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Single char (text: " ", prefix argument: undefined).
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Execute "default:type" (text: " ")
console.ts:137 [Extension Host] debug: [command]	 Command executed: "type"
console.ts:137 [Extension Host] debug: [type command]	 args.text = "A"
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.handleType]	 Not in prefix argument mode. exit.
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.cancel]
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Single char (text: "A", prefix argument: undefined).
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Execute "default:type" (text: "A")

(This log was complete. There was no type command for "L")

Log at the successful case:

The resultant text was * AL.

console.ts:137 [Extension Host] debug: [command]	 Command executed: "type"
console.ts:137 [Extension Host] debug: [type command]	 args.text = "*"
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.handleType]	 Not in prefix argument mode. exit.
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.cancel]
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Single char (text: "*", prefix argument: undefined).
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Execute "default:type" (text: "*")
console.ts:137 [Extension Host] debug: [command]	 Command executed: "type"
console.ts:137 [Extension Host] debug: [type command]	 args.text = " "
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.handleType]	 Not in prefix argument mode. exit.
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.cancel]
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Single char (text: " ", prefix argument: undefined).
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Execute "default:type" (text: " ")
console.ts:137 [Extension Host] debug: [command]	 Command executed: "type"
console.ts:137 [Extension Host] debug: [type command]	 args.text = "A"
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.handleType]	 Not in prefix argument mode. exit.
console.ts:137 [Extension Host] debug: [PrefixArgumentHandler.cancel]
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Single char (text: "A", prefix argument: undefined).
console.ts:137 [Extension Host] debug: [EmacsEmulator.type]	 Execute "default:type" (text: "A")

Next action

Set logger to compositionStart and compositionEnd events.

whitphx · 2020-05-21T16:47:35Z

↓こちらの.vsixファイルから本拡張をインストールして、本issueが解決されているかチェックしてみてください。
↓Please check if this issue is resolved by installing a new version of this extension from the link below:
https://github.com/tuttieee/vscode-emacs-mcx/releases/download/v0.20.0-dev.0/emacs-mcx-0.20.0-dev.0.vsix

こちらのissueの解決の候補となる実装を書きました。
しかし、このバグは毎回再現するわけでなく、確信を持って確認するのが難しい状況です。
さらにVSCodeの文字入力機能に関する低レベルなコマンドをフックする修正となっており、文字入力全般に関する安定性も心配です。
そこで、まずはVS Marketにはリリースせず、.vsixファイル（extensionをバンドルしたファイル）をGitHubのリリースページで配布することにしました。
この.vsixファイルで本拡張をインストールし、検証に協力していただけないでしょうか？
↓こちらから.vsixファイルをDLしてインストールしてください（Command paletteの"Extensions: Install from VSIX..."からできます）
https://github.com/tuttieee/vscode-emacs-mcx/releases/tag/v0.20.0-dev.0
しばらく（１週間程度？）使ってみて問題がなさそうなら報告していただけると助かります。
私自身も使っています。
安定していそうならmasterに取り込んでMarketにリリースします。

注: macOSでことえりのライブ変換と併用すると、長めの文字列を入力した際に変換結果が二重に表示されるバグに遭遇するかもしれません。
こちらはVSCode本体のrepositoryにissueを投げてあります: microsoft/vscode#98328
最終的にリリースする際には、本修正を有効にするか無効にするかをConfigで選べるようにする予定です（どちらも一長一短で完全な解決ではないですが、ワークアラウンドとして）

I made a new implementation that can be a candidate of a fix to this issue, but it's not very confirmed because it's difficult to reproduce the bug frequently and check the fix.
In addition, low-level VSCode commands related to character input has been modified and I worry about the stability.
Then, I published the .vsix file (the bundle file of the extension) on GitHub release page, instead of releasing it in the VS Market.
So, can you please join checking the fix using the .vsix file?
You can download the file from the link below and install it via "Extensions: Install from VSIX..." in the Command Palette.
https://github.com/tuttieee/vscode-emacs-mcx/releases/tag/v0.20.0-dev.0
I'm also using it now of course.
If it looks stable when you used it for about a week, please report it.
I will merge it into the master branch and release it to the Market if it looks good.

NOTE: If you are using the Japanese IME of macOS with Live Conversion, you can find a bug where the converted input looks double.
I created an issue on the VSCode repository: microsoft/vscode#98328
I'm planning to make it possible to turn off the fix here to avoid it in the public release. It will not be a complete solution, but a workaround though.

suchi · 2020-05-24T09:49:01Z

残念ながらまだ解決しないようです。確実な再現パターンとしては jsファイルで

foo() ; // valueがうまく入力できない

これを入力時に "value/が/うまく/入力/できない" のスラッシュで変換確定すると
"vvaluがうまく入入できない" となります。コメントキャラクタの後ろでのみ再現します。

whitphx · 2020-05-24T13:40:20Z

@suchi レポートありがとうございます。以下の情報もいただけるでしょうか？

VSCode version
OS
IME

suchi · 2020-05-24T14:48:01Z

失礼、基本情報が抜けていました

VSCode 1.45.0
OS: Windows 10 Pro (10.0.18363)
IME: ATOK 2015(28.0.7) (ATOK 2017(30.0.8)の環境での再現も確認しています)

whitphx · 2020-06-29T15:13:18Z

上記の開発版では問題は解決していませんでした（さらに悪いことにSKKとの相性は悪化しました）。別の解決を試みたので、こちらのバージョンを以下のリンク先の.vsixファイルからをインストールして、本issueが解決されているかチェックしてみてください。
↓The previous development version did not resolve the problem (and it worked worse with SKK). I tried another solution and please check if this issue is resolved by installing the new version of this extension from the link below:
https://github.com/tuttieee/vscode-emacs-mcx/releases/download/v0.20.0-dev.1/emacs-mcx-0.20.0-dev.1.vsix

@suchi 前バージョンに関してご協力ありがとうございました。こちらのバージョンも試していただけると幸いです。

whitphx · 2020-06-30T16:10:25Z

すみません、上記バージョンは不完全でした。↓こちらでお願いします。
Sorry, in the previous release the fix was not fully applied. Please try this version.
https://github.com/tuttieee/vscode-emacs-mcx/releases/download/v0.20.0-dev.2/emacs-mcx-0.20.0-dev.2.vsix

whitphx · 2020-07-12T13:18:30Z

It may be a bug derived from VSCode itself. I created an issue in the VSCode repository that is linked above.

suchi · 2020-07-12T13:55:59Z

foo(); // vvaluがうまく入入できない 0.20.0-dev.0
foo(); // valueがうまく入力できない 0.20.0-dev.2

申し訳ありません、dev.1のメンション見落としていました。今試したところ、dev.2では正常に入力できました!

VSCode version: 1.46.1
OS: Windows 10 Pro (10.0.18363)
IME: ATOK 2015(28.0.7)

whitphx · 2020-07-12T15:39:19Z

@suchi どうもありがとうございます 🙇
dev.2の変更はすでにMarketに0.20.xとして出ておりますので、そちらを使い続けていただければと思います（もし本件の解決にはならなくても、他にも意味のある変更だったので、本Issueの状況とは独立にリリースしてしまいました）。

一方で、最近私の環境で、dev.2でも似たような現象が再現してしまいました。
この現象を起こす要因が複数ありうる気がしており、そのうち一つがdev.2で潰したもの、また別の一つが microsoft/vscode#102291 かと思っています。
こちらも回避するワークアラウンドを考えています。

なんにせよ、まずはそちらの環境で解決したとのこと、よかったです。ご報告どうもありがとうございました。

AquiTCD · 2020-07-13T01:47:38Z

AquaSKKと使用中ですが1週間ほど使用して特に問題なく動いてる様子です。
大変たすかります、ありがとうございます。ご報告まで。

VSCode version: 1.47.0
OS: macOS 10.14.6
IME: AquaSKK 4.7.0

whitphx · 2020-07-13T09:01:10Z

@AquiTCD どうもありがとうございます。よかったです。

tac0x2a mentioned this issue Jul 6, 2019

Visual Studio Code である程度の速さでタイプすると確定時に何文字か消える codefirst/aquaskk#88

Closed

This was referenced May 19, 2020

[WIP] CompositionHandler to manage invokation orders of type() and replacePreviousChars() #308

Closed

CompositionHandler to manage invokation orders of type() and replacePreviousChars() #309

Merged

whitphx mentioned this issue Jul 3, 2020

[WIP] Fix/wait type command promise #341

Closed

whitphx mentioned this issue Jul 12, 2020

Overridden "type" command does not work properly with heavy processing inside it microsoft/vscode#102291

Closed

whitphx mentioned this issue Jul 23, 2020

Fix to simply override replacePreviousChar command without no effects in order for type command to work correctly #383

Closed

This was referenced Sep 13, 2020

Make it possible to disable prefix argument #135

Closed

characters chaos when inputting chinese characters after updated newest vscode #425

Closed

This was referenced Oct 12, 2020

Feature/prefix argument without type command #457

Closed

Feature/universal argument without type command #478

Merged

whitphx closed this as completed in #478 Oct 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overwrite previous character in Japanese IME #115

Overwrite previous character in Japanese IME #115

tac0x2a commented May 15, 2019

whitphx commented May 21, 2019 •

edited

Loading

whitphx commented Aug 27, 2019

whitphx commented Dec 10, 2019

whitphx commented May 21, 2020

suchi commented May 24, 2020

whitphx commented May 24, 2020

suchi commented May 24, 2020 •

edited

Loading

whitphx commented Jun 29, 2020

whitphx commented Jun 30, 2020

whitphx commented Jul 12, 2020

suchi commented Jul 12, 2020

whitphx commented Jul 12, 2020

AquiTCD commented Jul 13, 2020

whitphx commented Jul 13, 2020

Overwrite previous character in Japanese IME #115

Overwrite previous character in Japanese IME #115

Comments

tac0x2a commented May 15, 2019

whitphx commented May 21, 2019 • edited Loading

memo

whitphx commented Aug 27, 2019

Background

Type/Composition events with Japanese IME

This plugin's event binding

Hypothesis

Possible solution

whitphx commented Dec 10, 2019

Env

Action

Log at the bug case:

Log at the successful case:

Next action

whitphx commented May 21, 2020

suchi commented May 24, 2020

whitphx commented May 24, 2020

suchi commented May 24, 2020 • edited Loading

whitphx commented Jun 29, 2020

whitphx commented Jun 30, 2020

whitphx commented Jul 12, 2020

suchi commented Jul 12, 2020

whitphx commented Jul 12, 2020

AquiTCD commented Jul 13, 2020

whitphx commented Jul 13, 2020

whitphx commented May 21, 2019 •

edited

Loading

suchi commented May 24, 2020 •

edited

Loading