Fix input multibyte characters #183

siefca · 2013-11-23T16:40:19Z

…e characters

…objects

Details: - changed Ncurses::CharCode to work with wide characters, - removed select wrapper when reading input, - moved input reading into Ncurses::CharCode class, - implemented passing Ncurses::CharCode objects as far as possible.

… returned by get_wch

…osition)

…d for form_driver workaroud

siefca · 2013-11-23T17:17:22Z

Wait, I'm moving some portions to other files and refactoring CharCode interface a bit.

gauteh · 2013-11-23T18:41:56Z

lib/sup/buffer.rb

+      def form_driver_w form, status, c
+        form_driver form, c
+      end
+      module_function :form_driver_w


There should be a:

warn "Your Ncursesw does not have a form_driver_w function (wide char aware), non-ASCII chars may not work on your system."

here I think (so that it is shown at start up).

gauteh · 2013-11-23T18:46:15Z

Nice, we need a test which enters stuff using form_driver_w and checks whether the buffer that is returned is sane.

…y singleton class

…pared directly

…m_driver and called after initscr

…code or a character)

…_driver

gauteh · 2013-11-24T00:51:07Z

lib/sup/util/ncurses.rb

+            "non-ASCII chars may not work on your system."
+      warn  msg
+      print msg
+      sleep 3


Only warn (most users won't have a form_driver_w), if you set SUP_LOG_LEVEL=debug output is flushed when logging functions are used and it will be displayed immediately, print should not be necessary.

Right. Done.

siefca · 2013-11-24T02:06:33Z

Ok, changes for #180 seem quite usable.
AFAIR we need some Asian environments to test ltr and charsets.

gauteh · 2013-11-24T08:30:18Z

Yes, definitely. CJK letters should all be covered by UTF-8, but we might have problems with input in other CJK-charsets. We should, however, be able to work internally in UTF-8.

It appears that Ncurses (and Sup) doesn't really support RTL at all.. and users have relied on terminals that have swapped text for them. Anyway, we use REQ_NEXT_CHAR in form_driver_w and not REQ_RIGHT_CHAR.

siefca · 2013-11-24T19:45:15Z

It's ugly but it's working. Is it working on your system?

gauteh · 2013-11-24T19:49:16Z

Testing with patched ncurses, it works, partially 😬 when multiple sigwinches happen the window is redrawn for every sigwinch. Have you checked with the existing code? It used to handle that...

(I know, you have discovered some parts of Sup that were not supposed to see daylight again)

…appened

siefca · 2013-11-24T20:42:06Z

How about now?

I placed it (ungetch) within a mutex synchronizer with condition. From what I saw using other console software (mc, vim), the multiple refreshes are happening sometimes too; to avoid that totally we should implement some delay in sigwinch_happened! just after it sets the flag.

gauteh · 2013-11-24T20:45:52Z

Pretty good, sometimes a few refreshes happen, but I can't remember if it was any different before :) (also; #111 might be of interest)

gauteh · 2013-11-24T20:52:42Z

Yup, works on both original ncurses and patched, the slight refreshing is the same as before.

siefca · 2013-11-24T22:37:52Z

Just out of curiosity: where error method comes from?

gauteh · 2013-11-25T07:40:04Z

https://github.com/sup-heliotrope/sup/blob/develop/lib/sup/logger.rb#L74 ?

gauteh · 2013-11-26T08:13:10Z

If, in Sup, I do:

m # new mail
To: asdfadfasdfad
Subject: asdfasdfad <ctrl-c> # abort new mail

Sup (one thread) spins up to 100% CPU usage, seems to work fine otherwise though.

siefca · 2013-11-26T17:05:08Z

Here is the problem with CPU. If there wasn't empty singleton returned by generate there would also be a problem with memory. :]

    def self.get handle_interrupt=true
      begin
        status, code = nonblocking_getwch        # <---- returns [Ncurses::ERR, 0]
        generate code, status
      rescue Interrupt => e
        raise e unless handle_interrupt
        keycode Ncurses::KEY_CANCEL
      end
    end

Ncurses.get_wch returns Ncurses::Err after CTRL+C had been pressed and it returns it continuously.
I'm not sure but the problem might be in ncurses itself or in a wrapper.

It looks like the internal select() or something is not waiting for socket to be ready or some internal flag is never cleared.

[Ncurses::ERR, 0] pair means that there is no data to read, but it behaves like it needed a select wrapper again.

gauteh · 2013-11-26T17:27:12Z

Hm, ok. As far as I can understand, get_wch or getch can never return
ERR. Perhaps we still need the IO.select ?

gaute

siefca · 2013-11-26T17:34:55Z

IO.select helps but there is another possibility that something on our level stops sleeping for a while.

This code runs like crazy after CTRL+C is hit:

Redwood::exceptions.nonempty?
[…]
  if c.empty?
    […]
    next  # loops back since empty

Before CTRL+C is pressed it ticks in some intervals.

siefca · 2013-11-26T17:46:54Z

Non-blocking get_wch returns immediatelly as designed. But why it worked before? Probably because there was select making 0.5 second waits to check if there even is data to be read. Our code does the same thing but is not going to sleep if there is no data (instead it enters the loop that utilizes 100% CPU). I'll try conditional select.

…roneous CharCode spawning

siefca · 2013-11-26T18:50:25Z

Fixed.

gauteh · 2013-11-26T21:17:39Z

Nice catch.

gauteh · 2013-12-08T12:31:01Z

The form_driver_w routine has now been included in the latest ncurses-patch version: http://lists.gnu.org/archive/html/bug-ncurses/2013-12/msg00010.html.

siefca · 2013-12-08T17:09:29Z

😄 Thx Gaute!

gauteh · 2013-12-09T08:23:30Z

lib/sup/textfield.rb

@@ -97,39 +101,26 @@ def handle_input c
    reset_completion_state
    @value = nil

-    d =
-      case c
+    ctrl_c =


What does ctrl_c mean here (and below)? Not C-c? Maybe add some comment (and change var name) so that it is understandable for the next guy :) It is pretty messy stuff (kudos for figuring it out)!

Yeah, I'll change it since it only has meaning for me (control character) and may mislead others.

form_driver does not work correctly on all systems (specifically Mac OS X), it is difficult or impossible to determine whether the input char is a char or key code resulting from a char sequence in a multibyte char. If no form_driver_w is present we fall back to form_driver. form_driver_w is included in ncursesw-1.4.2 with the Ncurses 5.9 20131207 patchlevel. http://lists.gnu.org/archive/html/bug-ncurses/2013-12/msg00010.html original patch: http://lists.gnu.org/archive/html/bug-ncurses/2013-11/msg00015.html Author: Paweł Wilk <siefca@gnu.org> Squashed commit of the following: commit 7a513f2 Author: Paweł Wilk <siefca@gnu.org> Date: Tue Nov 26 19:49:30 2013 +0100 Added conditional select in Ncurses::nonblocking_getwch to prevent erroneous CharCode spawning commit 0b6859e Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 21:14:52 2013 +0100 BufferManager.sigwinch_happened! modified to run once if it already happened commit 857cb60 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 20:44:21 2013 +0100 Added workaround for screen redrawing when sigwinch happened commit d72d72f Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 20:43:46 2013 +0100 Optimizations in Ncurses::CharCode.get commit c2139b6 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 18:19:57 2013 +0100 Fixed uninitialized instance variable warning commit 72fb7e4 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:43:16 2013 +0100 Added critical patches for Ncurses::Form::DriverHelpers#form_driver and Ncurses.get_wch commit 340fb7f Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:42:24 2013 +0100 Ncurses::Form::DriverHelpers improved; now all wrappers are reusing core wrapper commit ca362c2 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:40:38 2013 +0100 Ncurses::CharCode#enc_char improved commit 380079e Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:40:01 2013 +0100 Added Ncurses::CharCode::Empty.empty override commit 8da83d9 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:39:28 2013 +0100 Cleanups commit de9f7bc Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:39:02 2013 +0100 Added Ncurses::CharCode.dumb! switch to control behavior when CharCode cannot detect keycodes commit ffad7e5 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 17:35:30 2013 +0100 Added Ncurses::CharCode.generate to save objects from spawning when ERR is returned by get(w)ch commit ad6f6f2 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 03:02:01 2013 +0100 Removed printing of warning about a missing form_driver_w() function commit ea277c1 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 02:55:23 2013 +0100 Cleanups commit 505532e Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:24:08 2013 +0100 TextField now uses form_driver helpers from Ncurses::Form::DriverHelpers commit 9103729 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:23:22 2013 +0100 Added Form::DriverHelpers module with methods that ease usage of form_driver commit 05c2b56 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:22:08 2013 +0100 Improved Ncurses::CharCode#replace method (detects if input is a charcode or a character) commit a2f0dac Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:21:11 2013 +0100 Removed previous wrapper for form_driver_w() commit 3a5cbdc Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:20:12 2013 +0100 Cleanups commit a958553 Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:19:51 2013 +0100 Replacement wrapper for form_driver_w() moved to Ncurses::prepare_form_driver and called after initscr commit 7766cbf Author: Paweł Wilk <siefca@gnu.org> Date: Sun Nov 24 01:17:37 2013 +0100 Multiple typos fixed commit 8bf705c Author: Paweł Wilk <siefca@gnu.org> Date: Sat Nov 23 20:58:54 2013 +0100 Key for continuing search in buffer in Buffer#handle_input is now compared directly commit 1dcfabd Author: Paweł Wilk <siefca@gnu.org> Date: Sat Nov 23 20:57:22 2013 +0100 Moved ncurses tweaks to lib/sup/util/ncurses.rb, added CharCode::Empty singleton class commit ed85b52 Author: Paweł Wilk <siefca@gnu.org> Date: Sat Nov 23 17:37:50 2013 +0100 TextField class is now making use of Ncurses::Form.form_driver_w commit 9ce6134 Author: Paweł Wilk <siefca@gnu.org> Date: Sat Nov 23 17:37:10 2013 +0100 Cleanups in buffer.rb commit f8f7116 Author: Paweł Wilk <siefca@gnu.org> Date: Sat Nov 23 17:36:49 2013 +0100 Added compat proxy for the Ncurses::Form.form_driver_w module method commit 171eada Author: Paweł Wilk <siefca@gnu.org> Date: Fri Nov 22 02:36:27 2013 +0100 Fixed TextField#handle_input (keysyms taken into account) and prepared for form_driver workaroud commit 3217a65 Author: Paweł Wilk <siefca@gnu.org> Date: Fri Nov 22 02:32:28 2013 +0100 Added Ncurses::curyx module method (helper for reading whole cursor position) commit 60b02d6 Author: Paweł Wilk <siefca@gnu.org> Date: Thu Nov 21 19:08:51 2013 +0100 Ncurses::CharCode::get now uses chr to create character based on code returned by get_wch commit 58088c4 Author: Paweł Wilk <siefca@gnu.org> Date: Thu Nov 21 17:55:45 2013 +0100 Typo fixed commit ca8e984 Author: Paweł Wilk <siefca@gnu.org> Date: Thu Nov 21 17:21:49 2013 +0100 Ncurses::CharCode class refactored, code adapted to use its instances Details: - changed Ncurses::CharCode to work with wide characters, - removed select wrapper when reading input, - moved input reading into Ncurses::CharCode class, - implemented passing Ncurses::CharCode objects as far as possible. commit f0168be Author: Paweł Wilk <siefca@gnu.org> Date: Tue Nov 19 23:20:13 2013 +0100 Adapted Keymap, Buffer and TextField classes to make use of CharCode objects commit dfea936 Author: Paweł Wilk <siefca@gnu.org> Date: Tue Nov 19 23:17:11 2013 +0100 Added Ncurses::CharCode helper class for storing and transferring wide characters

gauteh · 2013-12-16T14:43:08Z

Merged.

…put" This reverts commit 7e7858c.

…put" This reverts commit 7e7858c. Conflicts: bin/sup

siefca added 10 commits November 19, 2013 23:17

Added Ncurses::CharCode helper class for storing and transferring wid…

dfea936

…e characters

Adapted Keymap, Buffer and TextField classes to make use of CharCode …

f0168be

…objects

Typo fixed

58088c4

Ncurses::CharCode::get now uses chr to create character based on code…

60b02d6

… returned by get_wch

Added Ncurses::curyx module method (helper for reading whole cursor p…

3217a65

…osition)

Fixed TextField#handle_input (keysyms taken into account) and prepare…

171eada

…d for form_driver workaroud

Added compat proxy for the Ncurses::Form.form_driver_w module method

f8f7116

Cleanups in buffer.rb

9ce6134

TextField class is now making use of Ncurses::Form.form_driver_w

ed85b52

gauteh reviewed Nov 23, 2013
View reviewed changes

siefca added 9 commits November 23, 2013 20:57

Moved ncurses tweaks to lib/sup/util/ncurses.rb, added CharCode::Empt…

1dcfabd

…y singleton class

Key for continuing search in buffer in Buffer#handle_input is now com…

8bf705c

…pared directly

Multiple typos fixed

7766cbf

Replacement wrapper for form_driver_w() moved to Ncurses::prepare_for…

a958553

…m_driver and called after initscr

Cleanups

3a5cbdc

Removed previous wrapper for form_driver_w()

a2f0dac

Improved Ncurses::CharCode#replace method (detects if input is a char…

05c2b56

…code or a character)

Added Form::DriverHelpers module with methods that ease usage of form…

9103729

…_driver

TextField now uses form_driver helpers from Ncurses::Form::DriverHelpers

505532e

gauteh reviewed Nov 24, 2013
View reviewed changes

siefca added 2 commits November 24, 2013 02:55

Cleanups

ea277c1

Removed printing of warning about a missing form_driver_w() function

ad6f6f2

siefca added 2 commits November 24, 2013 20:43

Optimizations in Ncurses::CharCode.get

d72d72f

Added workaround for screen redrawing when sigwinch happened

857cb60

BufferManager.sigwinch_happened! modified to run once if it already h…

0b6859e

…appened

Added conditional select in Ncurses::nonblocking_getwch to prevent er…

7a513f2

…roneous CharCode spawning

gauteh reviewed Dec 9, 2013
View reviewed changes

gauteh closed this Dec 16, 2013

This was referenced Mar 12, 2014

output stalls unless user refreshes (ctrl-l) #248

Closed

Tag by label #260

Closed

gauteh mentioned this pull request Apr 21, 2014

Incorrect screen rendering after poll #292

Closed

eMBee added a commit to eMBee/sup that referenced this pull request Feb 14, 2015

Revert "Merge sup-heliotrope#183: Wrap form_driver_w for wide char in…

6dcdb7d

…put" This reverts commit 7e7858c.

eMBee added a commit to eMBee/sup that referenced this pull request Feb 14, 2015

Revert "Merge sup-heliotrope#183: Wrap form_driver_w for wide char in…

e5ba3eb

…put" This reverts commit 7e7858c. Conflicts: bin/sup

eMBee added a commit to eMBee/sup that referenced this pull request Feb 16, 2015

Revert "Merge sup-heliotrope#183: Wrap form_driver_w for wide char in…

9ed734b

…put" This reverts commit 7e7858c. Conflicts: bin/sup

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix input multibyte characters #183

Fix input multibyte characters #183

siefca commented Nov 23, 2013

siefca commented Nov 23, 2013

gauteh Nov 23, 2013

siefca Nov 23, 2013

gauteh commented Nov 23, 2013

gauteh Nov 24, 2013

siefca Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 24, 2013

gauteh commented Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 25, 2013

gauteh commented Nov 26, 2013

siefca commented Nov 26, 2013

gauteh commented Nov 26, 2013

siefca commented Nov 26, 2013

siefca commented Nov 26, 2013

siefca commented Nov 26, 2013

gauteh commented Nov 26, 2013

gauteh commented Dec 8, 2013

siefca commented Dec 8, 2013

gauteh Dec 9, 2013

siefca Dec 10, 2013

gauteh commented Dec 16, 2013

Fix input multibyte characters #183

Fix input multibyte characters #183

Conversation

siefca commented Nov 23, 2013

siefca commented Nov 23, 2013

gauteh Nov 23, 2013

Choose a reason for hiding this comment

siefca Nov 23, 2013

Choose a reason for hiding this comment

gauteh commented Nov 23, 2013

gauteh Nov 24, 2013

Choose a reason for hiding this comment

siefca Nov 24, 2013

Choose a reason for hiding this comment

siefca commented Nov 24, 2013

gauteh commented Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 24, 2013

gauteh commented Nov 24, 2013

siefca commented Nov 24, 2013

gauteh commented Nov 25, 2013

gauteh commented Nov 26, 2013

siefca commented Nov 26, 2013

gauteh commented Nov 26, 2013

siefca commented Nov 26, 2013

siefca commented Nov 26, 2013

siefca commented Nov 26, 2013

gauteh commented Nov 26, 2013

gauteh commented Dec 8, 2013

siefca commented Dec 8, 2013

gauteh Dec 9, 2013

Choose a reason for hiding this comment

siefca Dec 10, 2013

Choose a reason for hiding this comment

gauteh commented Dec 16, 2013