Better console #236

Closed
ccoupe opened this Issue Apr 6, 2016 · 42 comments

Comments

Projects
None yet
3 participants
@ccoupe
Contributor

ccoupe commented Apr 6, 2016

The new console in Shoes 3.2 is kind of miserable - particularly on OSX but the tesi/gtk is not a shining example of perfection either. Windows is OK. The nix's use a pty that talks to a secondary event poll in Shoes. What if we could create a subprocess that runs a terminal and communicates over a PTY. Turns out there is a Ruby sdtlib for this.

$ irb
2.2.4 :001 > require 'pty'
 => true 
2.2.4 :002 > PTY.spawn('gnome-terminal')
 => [#<File:/dev/pts/4>, #<File:/dev/pts/4>, 16575] 
2.2.4 :003 > ^C
2.2.4 :003 > 

That doesn't wire up stdout<->stdin between processes but the API and we don't really want bash in the subprocess but lets wave our hands and pretend that is something we can control (probably)

Shoes goes odd if you require 'pty'

Shoes.app do
  stack do
    para "Demonstrate Pty.spawn with new Terminal"
    button "Try It" do
      #require 'pty'
      if RUBY_PLATFORM =~ /darwin/
        puts 'darwin'
        `/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal`
      else
        puts "linux"
        system('gnome-terminal')
      end
    end
  end
end

Yet if you use the Shoes irb (alt-=) you can type the same IRB commands above without error.

OSX is more like Windows. If you start a terminal from an app and you have a terminal on screen it uses that one. If you have no terminals open then it does create a new one (with a error message on it) I don't understand why you can't `PTY,spawn(...) in a Shoes script. Some sort of scope/method missing magic?

@ccoupe ccoupe added the Normal label Apr 6, 2016

@ccoupe ccoupe added this to the 3.3.2 milestone Apr 6, 2016

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 6, 2016

Contributor

Nice idea !
PTY,spawn works for me (in your example) in linux, though... well, i don't have a good understanding of all this term/tty/pty thing
Don't we need a IO.pipe like in the official doc of PTY to interact with gnome_terminal ?

otherwise, i can send a 'ls' command, for example, that way and get back the result in a para

PTY.spawn('ls') do |output, input, pid|
        input.write "hi\n"
        output.each { |line| @ret.text = @ret.text + line }
end

There is also this, don't know if it's useful, (for the present case , i mean)
http://ruby-doc.org/stdlib-2.0.0/libdoc/io/console/rdoc/IO.html

Shell library too

Contributor

passenger94 commented Apr 6, 2016

Nice idea !
PTY,spawn works for me (in your example) in linux, though... well, i don't have a good understanding of all this term/tty/pty thing
Don't we need a IO.pipe like in the official doc of PTY to interact with gnome_terminal ?

otherwise, i can send a 'ls' command, for example, that way and get back the result in a para

PTY.spawn('ls') do |output, input, pid|
        input.write "hi\n"
        output.each { |line| @ret.text = @ret.text + line }
end

There is also this, don't know if it's useful, (for the present case , i mean)
http://ruby-doc.org/stdlib-2.0.0/libdoc/io/console/rdoc/IO.html

Shell library too

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Apr 6, 2016

Collaborator

I had many attempts using such techniques on Windows to integrate the real IRB in Shoes but without success.

Collaborator

BackOrder commented Apr 6, 2016

I had many attempts using such techniques on Windows to integrate the real IRB in Shoes but without success.

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 6, 2016

Contributor

I've done something maybe similar, connecting Shoes, IRB and some Gimp input/output :
https://github.com/passenger94/Gimp-Ruby/blob/master/lib/plug-ins/irbconsole.rb
https://github.com/passenger94/Gimp-Ruby/blob/master/lib/plug-ins/shoes_console.rb.
Works nice on my system

Contributor

passenger94 commented Apr 6, 2016

I've done something maybe similar, connecting Shoes, IRB and some Gimp input/output :
https://github.com/passenger94/Gimp-Ruby/blob/master/lib/plug-ins/irbconsole.rb
https://github.com/passenger94/Gimp-Ruby/blob/master/lib/plug-ins/shoes_console.rb.
Works nice on my system

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 7, 2016

Contributor

A bit long winded.

Actually, the Windows code for the new console is pretty nice as is - one could re write the Shoes.irb to use it (but read on before doing that)

Remember that the console works by replacing the low level file_descriptors for stdin/stdout/stderr so they point one side of pty (think one side some pipes) and the other side of the pty(pipes) is a terminal emulator. For nix/osx, that is a poorly written emulator in the same process. Clever but still a hack.

Updated sample:

Shoes.app do
  stack do
    para "Demonstrate Pty.spawn with new Terminal"
    button "Try It" do
      if RUBY_PLATFORM =~ /darwin/
        puts 'darwin'
        `/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal`
      else
        require 'pty.so'
        puts "linux stdin = #{$stdin.fileno} stdout = #{$stdout.fileno}"
        rd, wr, pid = PTY.spawn("gnome-terminal")
        wr.puts "new #{pid}  pty?"
      end
    end
  end
end

Sadly, gnome-terminal has some problems used this way. It runs a shell (and .bashrc.) and changes it's stdin/out/err to that shell process so it's no longer talking to our pty. OSX would have the same problem (probably) What would be better for the nix's is when shoes.show_console is called that the code does a fork() and the new (subprocess) runs a terminal emulator. (which is pretty close to the code above, if gnome-terminal worked here) There's also an issue that gnome-terminal is not in all Linux installs.

conceptually in C,

long pty[3] = create_pty(); 
char *[] command_line(args)
 if (pid = fork()) {
     // in child subprocess
     start_terminal(pty[0], pty[1], pty[2], command_line)  // something we wrote or the user already has
    // never returns
}

For now that's the tesi based 'thing' but now it's running in a different process and doesn't have to
call back to Shoes/ruby to display things. It couldn't because its a different process We'd have to manage the subprocess pid at quit time.

There are several use case to consider: (1) test-unit/rspec, (2) byebug/remote, (3) other gems that think they have a terminal on stdin/out/err, (4) Shoes/irb

1 and 3 are satisfied by this proposal. (2) Byebug/remote is too complicated. Let's create a 'shoesbug' method (which starts the byebug remote in the Shoes process and then creates the console with the byebug command to connect to the (parent process). Details left undiscovered and they are important. We do have Shoes & ruby in the console subprocess so we can execute the ruby code to make the server connection. Irb(4) use case: Unless it has some sort of server mode, we can't poke inside the parent process BUT the existing implementation could just read/write stdin/stdout (if irb uses readline) after creating a console. No need to manage keystrokes in Shoes or call Shoes to display things. Again, details matter. Also, we don't have to do anything for Irb.

There are many terminal emulators in the OSS world we so can get better code than tesi. (search terminal in syntapic) For linux , we also have gtk's vte. On OSX, there are not as many OSS emulators (lots of binary only shareware) and they seem designed to replace Apples terminal which is a lot more code than we need. There are other OSS state machines for handling the vt102/vt220 escapes instead of tesi and we do have OSX code to create a window, set the font, draw characters arriving from the pty and send key strokes to it. Don't have the color stuff or bold or ...

That said, the difference between what we have now and this proposal is a lot of C/Obj-C code. The difference between that start_terminal and a full app with menus .... is

main(argc,argv)  {
  start_terminal(stdin, stdout, stderr, "/bin/sh")
}

Not much different except we don't need to create or handle lots of menus or font selection, and we control what the command line args are.

Contributor

ccoupe commented Apr 7, 2016

A bit long winded.

Actually, the Windows code for the new console is pretty nice as is - one could re write the Shoes.irb to use it (but read on before doing that)

Remember that the console works by replacing the low level file_descriptors for stdin/stdout/stderr so they point one side of pty (think one side some pipes) and the other side of the pty(pipes) is a terminal emulator. For nix/osx, that is a poorly written emulator in the same process. Clever but still a hack.

Updated sample:

Shoes.app do
  stack do
    para "Demonstrate Pty.spawn with new Terminal"
    button "Try It" do
      if RUBY_PLATFORM =~ /darwin/
        puts 'darwin'
        `/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal`
      else
        require 'pty.so'
        puts "linux stdin = #{$stdin.fileno} stdout = #{$stdout.fileno}"
        rd, wr, pid = PTY.spawn("gnome-terminal")
        wr.puts "new #{pid}  pty?"
      end
    end
  end
end

Sadly, gnome-terminal has some problems used this way. It runs a shell (and .bashrc.) and changes it's stdin/out/err to that shell process so it's no longer talking to our pty. OSX would have the same problem (probably) What would be better for the nix's is when shoes.show_console is called that the code does a fork() and the new (subprocess) runs a terminal emulator. (which is pretty close to the code above, if gnome-terminal worked here) There's also an issue that gnome-terminal is not in all Linux installs.

conceptually in C,

long pty[3] = create_pty(); 
char *[] command_line(args)
 if (pid = fork()) {
     // in child subprocess
     start_terminal(pty[0], pty[1], pty[2], command_line)  // something we wrote or the user already has
    // never returns
}

For now that's the tesi based 'thing' but now it's running in a different process and doesn't have to
call back to Shoes/ruby to display things. It couldn't because its a different process We'd have to manage the subprocess pid at quit time.

There are several use case to consider: (1) test-unit/rspec, (2) byebug/remote, (3) other gems that think they have a terminal on stdin/out/err, (4) Shoes/irb

1 and 3 are satisfied by this proposal. (2) Byebug/remote is too complicated. Let's create a 'shoesbug' method (which starts the byebug remote in the Shoes process and then creates the console with the byebug command to connect to the (parent process). Details left undiscovered and they are important. We do have Shoes & ruby in the console subprocess so we can execute the ruby code to make the server connection. Irb(4) use case: Unless it has some sort of server mode, we can't poke inside the parent process BUT the existing implementation could just read/write stdin/stdout (if irb uses readline) after creating a console. No need to manage keystrokes in Shoes or call Shoes to display things. Again, details matter. Also, we don't have to do anything for Irb.

There are many terminal emulators in the OSS world we so can get better code than tesi. (search terminal in syntapic) For linux , we also have gtk's vte. On OSX, there are not as many OSS emulators (lots of binary only shareware) and they seem designed to replace Apples terminal which is a lot more code than we need. There are other OSS state machines for handling the vt102/vt220 escapes instead of tesi and we do have OSX code to create a window, set the font, draw characters arriving from the pty and send key strokes to it. Don't have the color stuff or bold or ...

That said, the difference between what we have now and this proposal is a lot of C/Obj-C code. The difference between that start_terminal and a full app with menus .... is

main(argc,argv)  {
  start_terminal(stdin, stdout, stderr, "/bin/sh")
}

Not much different except we don't need to create or handle lots of menus or font selection, and we control what the command line args are.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 7, 2016

Contributor

I'm leaning towards using https://github.com/kika/libtsm because it could/should work with OSX as well as Linux and it has a gtk example we can start from. It's not documented in a way most of us would like -- aka misleading and confusing - pty/tty is already a deep dive for the young folk ;^). It's much better than the tesi crap.

Clone it and cd in

./autoconf.sh
./configure --enable-gtktsm
make
Contributor

ccoupe commented Apr 7, 2016

I'm leaning towards using https://github.com/kika/libtsm because it could/should work with OSX as well as Linux and it has a gtk example we can start from. It's not documented in a way most of us would like -- aka misleading and confusing - pty/tty is already a deep dive for the young folk ;^). It's much better than the tesi crap.

Clone it and cd in

./autoconf.sh
./configure --enable-gtktsm
make
@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 7, 2016

Contributor

The Kids Are Alright !! 😛

indeed more complete library !

Contributor

passenger94 commented Apr 7, 2016

The Kids Are Alright !! 😛

indeed more complete library !

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 7, 2016

Contributor

Actually the way to build libtsm is

./autoconf.sh
./configure
./make 
./make install
./make gtktsm

gtktsm is a properly written program. Much can be learned. I suspect 90% can be copied for Shoes purposes. libtsm (in /usr/local here) or /usr if installed from apt-get) is a new dependency and a new lib to copy into Tight Shoes. Getting gtktsm/libtsm working in OSX will be the larger challenge. I've created a remote branch 'tsm_con' since it's going to change things a lot and we don't know if it's worth it.

Contributor

ccoupe commented Apr 7, 2016

Actually the way to build libtsm is

./autoconf.sh
./configure
./make 
./make install
./make gtktsm

gtktsm is a properly written program. Much can be learned. I suspect 90% can be copied for Shoes purposes. libtsm (in /usr/local here) or /usr if installed from apt-get) is a new dependency and a new lib to copy into Tight Shoes. Getting gtktsm/libtsm working in OSX will be the larger challenge. I've created a remote branch 'tsm_con' since it's going to change things a lot and we don't know if it's worth it.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 8, 2016

Contributor

Now for the problems. There are many. Deal Breakers? The gtktsm sample was not written to handle pty the way we need. I knew that from the beginning but it runs deep in the sample. Secondly, on OSX, the rules for fork() are pretty strict - its needs to be followed by an exec..(path_to_executable) before the child does anything to confuse cocoa. This is true of Gtk apps as well. We need a real executable - not some functions in the duped code. This is a big issue for OSX (new app and build and all the rake files). It also turns out that the gtktsm sample doesn't handle scrolling (and the backing buffer may not be handled). That's a lot of work just to discover it might not work well in a separate process for all the use cases.

Perhaps the current design isn't all that bad, just incomplete (can't handle multi char escapes for setting colors...) There is a lot to like about using NSTextView and GtkTextView if we could handle insert/delete char/line/range which is suspiciously like issue #144 requirements only called from the terminal state machine(tesi or libtsm) -> shoes_native_textedit_ set_font_color(). That would be a lot of methods to add to text_edit but that's the point, text_edit is not edit_box and new methods are desired. And, it wouldn't change the windows implementation of show_console since Windows will never call the pty state machine - no ptys on Windows. (and that's OK in this scheme).

The tesi/llibtsm/vt102/xterm escape sequences can help define what the Shoes native methods need to be for the text_edit widget. Tonight, there's a lot to like about this approach.

Contributor

ccoupe commented Apr 8, 2016

Now for the problems. There are many. Deal Breakers? The gtktsm sample was not written to handle pty the way we need. I knew that from the beginning but it runs deep in the sample. Secondly, on OSX, the rules for fork() are pretty strict - its needs to be followed by an exec..(path_to_executable) before the child does anything to confuse cocoa. This is true of Gtk apps as well. We need a real executable - not some functions in the duped code. This is a big issue for OSX (new app and build and all the rake files). It also turns out that the gtktsm sample doesn't handle scrolling (and the backing buffer may not be handled). That's a lot of work just to discover it might not work well in a separate process for all the use cases.

Perhaps the current design isn't all that bad, just incomplete (can't handle multi char escapes for setting colors...) There is a lot to like about using NSTextView and GtkTextView if we could handle insert/delete char/line/range which is suspiciously like issue #144 requirements only called from the terminal state machine(tesi or libtsm) -> shoes_native_textedit_ set_font_color(). That would be a lot of methods to add to text_edit but that's the point, text_edit is not edit_box and new methods are desired. And, it wouldn't change the windows implementation of show_console since Windows will never call the pty state machine - no ptys on Windows. (and that's OK in this scheme).

The tesi/llibtsm/vt102/xterm escape sequences can help define what the Shoes native methods need to be for the text_edit widget. Tonight, there's a lot to like about this approach.

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 9, 2016

Contributor

too much problems ?

Contributor

passenger94 commented Apr 9, 2016

too much problems ?

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 10, 2016

Contributor

What the shoes console code does is backwards to what gtktsm and tesi was writtten for - their assumption is they a collecting key strokes and pass that to /bin/sh is a subprocess. and output from the subprocess is routed to gtk display methods. What Shoes wants is different.

Tesi is not impossible to use or to understand. It does needs as serious clean up and better comments plus function names that fit the purpose (libtsm has that problem too when used in the Shoes way).

Parsing all and responding to all escape sequences in the xterm-256 definition would be very difficult and useless since most will never be used. I short circuited tesi handling at tesi.c:54 and duped some processing into haveChar in gtk-terminal/cocoa-terminal. That hack needs to be undone. Use the 'tesi', Luke!

Contributor

ccoupe commented Apr 10, 2016

What the shoes console code does is backwards to what gtktsm and tesi was writtten for - their assumption is they a collecting key strokes and pass that to /bin/sh is a subprocess. and output from the subprocess is routed to gtk display methods. What Shoes wants is different.

Tesi is not impossible to use or to understand. It does needs as serious clean up and better comments plus function names that fit the purpose (libtsm has that problem too when used in the Shoes way).

Parsing all and responding to all escape sequences in the xterm-256 definition would be very difficult and useless since most will never be used. I short circuited tesi handling at tesi.c:54 and duped some processing into haveChar in gtk-terminal/cocoa-terminal. That hack needs to be undone. Use the 'tesi', Luke!

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 10, 2016

Contributor

i've made some basic escape sequence parsing for a ri based Ruby doc Shoes app
https://github.com/passenger94/Dance_floor/blob/master/ruby_doc.rb#L87

A Jedi craves not these things !

Contributor

passenger94 commented Apr 10, 2016

i've made some basic escape sequence parsing for a ri based Ruby doc Shoes app
https://github.com/passenger94/Dance_floor/blob/master/ruby_doc.rb#L87

A Jedi craves not these things !

ccoupe added a commit that referenced this issue Apr 12, 2016

for #236. Many lexical/syntatical to tesi and gtk-terminal.c
* uses tesi escape parsing instead of haveCharacter shortcut
* pass testObject* instead of void p* - be explicit.
* advance escape sequences are not written or tested -- just trying
  to get back to what haveCharacter did. Close but
* backspace delete from buffer but not on screen.

ccoupe added a commit that referenced this issue Apr 16, 2016

on branch 'terminal' for #236
* fixed tesi handling of escapes for attributes. Big change
* gtk: handle attributes (poorly) - need to use Text Tags. groan.
* added many callbacks from tesi parser - haveCharacter not required
* osx is not even close to working.
* will change names from console to terminal in future.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 16, 2016

Contributor

This is even harder than I thought. Tesi had bugs and never could parse xterm-color escapes in a useful manner. On OSX, with it's missing stdout, pty related things are going to be much different. Tesi may not be used for OSX - too soon to say.

gtk-terminal.c (only used on Linux) may switch to using Gtk vte instead of gtk_text_view (because VTE does all the missing stuff and seems to be part of every Linux except Raspbian which is a low value target when downloads are counted and they are used to do sudo apt-get install libvte-2.90

One last thing. I'm going to call it Shoes Terminal, visually, verbally and in the code to avoid confusion with Shoes Console in log.rb

Contributor

ccoupe commented Apr 16, 2016

This is even harder than I thought. Tesi had bugs and never could parse xterm-color escapes in a useful manner. On OSX, with it's missing stdout, pty related things are going to be much different. Tesi may not be used for OSX - too soon to say.

gtk-terminal.c (only used on Linux) may switch to using Gtk vte instead of gtk_text_view (because VTE does all the missing stuff and seems to be part of every Linux except Raspbian which is a low value target when downloads are counted and they are used to do sudo apt-get install libvte-2.90

One last thing. I'm going to call it Shoes Terminal, visually, verbally and in the code to avoid confusion with Shoes Console in log.rb

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 16, 2016

Contributor

Gtk terminal has an icon, just like osx! Windows can't play that game. Bonus points to use the packaged app name instead of "Shoes". Here's samples/expert-console.rb
screenshot from 2016-04-16 00 45 54

Contributor

ccoupe commented Apr 16, 2016

Gtk terminal has an icon, just like osx! Windows can't play that game. Bonus points to use the packaged app name instead of "Shoes". Here's samples/expert-console.rb
screenshot from 2016-04-16 00 45 54

ccoupe added a commit that referenced this issue Apr 16, 2016

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 17, 2016

Contributor

There is a lot to like with VTE - except it's a moving target. The api changes and Shoes has no idea if the destination has the API Shoe was compiled against and the internet doc is pathetic. The /usr/share/gtk-doc is better but version specific. I'll commit the gtk-terminal.c code for VTE (version 0.34 api)- who knows what your libvte-2.90 implements. Maybe someone else wants to deal with it.

screenshot from 2016-04-16 22 17 57

Contributor

ccoupe commented Apr 17, 2016

There is a lot to like with VTE - except it's a moving target. The api changes and Shoes has no idea if the destination has the API Shoe was compiled against and the internet doc is pathetic. The /usr/share/gtk-doc is better but version specific. I'll commit the gtk-terminal.c code for VTE (version 0.34 api)- who knows what your libvte-2.90 implements. Maybe someone else wants to deal with it.

screenshot from 2016-04-16 22 17 57

ccoupe added a commit that referenced this issue Apr 17, 2016

on #236 branch 'terminal' both gtk_text_view and vte widget
* some errors (scrolling, bg/fg colors) exist.
* vte is a moving API - not good for Shoes.
@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 17, 2016

Contributor

term

vte 2.90.34

but no luck with tests in colorized output (like tests for vlc, here testing Shoes::Color)

term_test

Contributor

passenger94 commented Apr 17, 2016

term

vte 2.90.34

but no luck with tests in colorized output (like tests for vlc, here testing Shoes::Color)

term_test

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 17, 2016

Contributor

There are some important things missing in the VTE version. (like scrolling and keyboard input). VTE is the core of gnome-terminal so it parses all the escape sequences of an xterm and responds as an xterm. That also means every Linux out there has a slightly different version of 2.90 with slightly different incompatible .so

Contributor

ccoupe commented Apr 17, 2016

There are some important things missing in the VTE version. (like scrolling and keyboard input). VTE is the core of gnome-terminal so it parses all the escape sequences of an xterm and responds as an xterm. That also means every Linux out there has a slightly different version of 2.90 with slightly different incompatible .so

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 17, 2016

Contributor

is it a lost cause then ? looks better ...

Contributor

passenger94 commented Apr 17, 2016

is it a lost cause then ? looks better ...

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 17, 2016

Contributor

We would have to include libvte.so with Shoes (might work) and doesn't screw up gtk3 on the running system. Odds are high the x86 linux build machine has an older vte than either of us and much older than the guy running 15.10 and the color picker.

Font's and colors can be set in the older terminal and I believe we can support attributes like color, bold, an so on. Just a matter of predefining text tags for them and then creative use of iterators to mark the start and end. Then apply the tag to the text buffer. And then something similar but different on osx.

Contributor

ccoupe commented Apr 17, 2016

We would have to include libvte.so with Shoes (might work) and doesn't screw up gtk3 on the running system. Odds are high the x86 linux build machine has an older vte than either of us and much older than the guy running 15.10 and the color picker.

Font's and colors can be set in the older terminal and I believe we can support attributes like color, bold, an so on. Just a matter of predefining text tags for them and then creative use of iterators to mark the start and end. Then apply the tag to the text buffer. And then something similar but different on osx.

ccoupe added a commit that referenced this issue Apr 18, 2016

for #236 we can change colors in Gtk_TextView console
* can't handle nested attributes - like setting background and foreground
  and then the text and then changing the background but not the foreground.
* GtkTextMarks and iters may not behave if your doc/coding is different from
  the gtk you compile against.
* doesn't suck too much and I learned a lot which will be useful for #144
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 18, 2016

Contributor

Here's the Gtk_TextView version of the console:
screenshot from 2016-04-17 22 25 33

There's much to be done but if we're clever I think we can work in all the attributes and quite likely the very odd scroll regions and xterm cursor movement (aka ncurses)

Contributor

ccoupe commented Apr 18, 2016

Here's the Gtk_TextView version of the console:
screenshot from 2016-04-17 22 25 33

There's much to be done but if we're clever I think we can work in all the attributes and quite likely the very odd scroll regions and xterm cursor movement (aka ncurses)

ccoupe added a commit that referenced this issue Apr 19, 2016

on #236, for Linux, we have colors, bold and underline
* do not combine with ; in the escape sequence.  tesi bug.
* colors are for character cells - not the window.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 19, 2016

Contributor

The gtk_text_view terminal continues to improve.
screenshot from 2016-04-18 21 47 00

As noted in the commit, there is a tesi bug dealing with ; in combined sequences (set bold and red for example) and they don't span properly - like underline, green,text,red, text,end_underline. That will take a more clever data structure - I'm not sure its worth the effort.

Contributor

ccoupe commented Apr 19, 2016

The gtk_text_view terminal continues to improve.
screenshot from 2016-04-18 21 47 00

As noted in the commit, there is a tesi bug dealing with ; in combined sequences (set bold and red for example) and they don't span properly - like underline, green,text,red, text,end_underline. That will take a more clever data structure - I'm not sure its worth the effort.

ccoupe added a commit that referenced this issue Apr 19, 2016

for #236, Linux - parse multiple attributes in one esc sequence
* much to be done with tesi. It's just mostly wrong. This is better. IMO

ccoupe added a commit that referenced this issue Apr 19, 2016

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 19, 2016

Contributor

👠 👠 👠
Yes, sir !
(with some intended failure)
term_test3

Contributor

passenger94 commented Apr 19, 2016

👠 👠 👠
Yes, sir !
(with some intended failure)
term_test3

@ccoupe ccoupe self-assigned this Apr 20, 2016

ccoupe added a commit that referenced this issue Apr 20, 2016

ccoupe added a commit that referenced this issue Apr 21, 2016

getting closer to parse xterm sequence (#236)
* even though the chance of backend implementions (gtk/cocoa) are
  near zero. tesi.c and test.h should suck less
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 23, 2016

Contributor

After much thinking and tinkering, It's hard to do xterm cursor movement in gkt_text_view. Not impossible, just difficult (and probably not very useful when working). Lines in gtk_text_buffers are variable length, null terminated. Asking for a character position past the end of a line segfaults (or returns the last character at best) -- not a lot of flexibility for moving the insertion point on the X axis.
Worst of all there's no way to detect that you aren't really pointing to where you asked.

One path I started down was to convert the first 24/25 lines to have 80 characters each when involved in a cursor based escape sequence but that's difficult too, However, we could have two gtk_text_buffers and switch them in the gtk_text_view, the log_buffer that @passenger94 shows as working above this comment, and a 'legacy_buffer' which is just 24|25 lines of 80 spaces to be moved to and replaced. Since there can be only one Shoes Terminal because there's only one stdin/stdout/stderr device, these buffers can be global vars (they are just C pointers) and gtk_text_view * could be stored a global too since it will never change. The terminal would start in log mode by default and attempts to move the cursor with with escape sequences (and clear screen, erase line. etc) would be ignored.

We should have a way for the Shoes scriptwriter to select which mode to start in. Might as well also specifying the terminal size (row, columns), background and foreground colors - because some people like black backgrounds and the window title (like we for dialogs). The redundant 'Shoes Terminal' in the 'menu panel' could be replaced with 'legacy_mode' checkbox that could switch the buffers and mode setting. If Windows can't do all those changes (it probably can't), so be it. If OSX can't all of if, thats OK too. Please advise.

Contributor

ccoupe commented Apr 23, 2016

After much thinking and tinkering, It's hard to do xterm cursor movement in gkt_text_view. Not impossible, just difficult (and probably not very useful when working). Lines in gtk_text_buffers are variable length, null terminated. Asking for a character position past the end of a line segfaults (or returns the last character at best) -- not a lot of flexibility for moving the insertion point on the X axis.
Worst of all there's no way to detect that you aren't really pointing to where you asked.

One path I started down was to convert the first 24/25 lines to have 80 characters each when involved in a cursor based escape sequence but that's difficult too, However, we could have two gtk_text_buffers and switch them in the gtk_text_view, the log_buffer that @passenger94 shows as working above this comment, and a 'legacy_buffer' which is just 24|25 lines of 80 spaces to be moved to and replaced. Since there can be only one Shoes Terminal because there's only one stdin/stdout/stderr device, these buffers can be global vars (they are just C pointers) and gtk_text_view * could be stored a global too since it will never change. The terminal would start in log mode by default and attempts to move the cursor with with escape sequences (and clear screen, erase line. etc) would be ignored.

We should have a way for the Shoes scriptwriter to select which mode to start in. Might as well also specifying the terminal size (row, columns), background and foreground colors - because some people like black backgrounds and the window title (like we for dialogs). The redundant 'Shoes Terminal' in the 'menu panel' could be replaced with 'legacy_mode' checkbox that could switch the buffers and mode setting. If Windows can't do all those changes (it probably can't), so be it. If OSX can't all of if, thats OK too. Please advise.

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 23, 2016

Contributor

From your comments i understand that gtk_text_view don't play well with escape sequence related to cursor movement (they require a well defined buffer?), so looks like a good plan to separate both ...

Looks like gtk_text_view is doing his royal picky huge kludge Princess, separation of concern is a mantra in development area, i've been told !

Contributor

passenger94 commented Apr 23, 2016

From your comments i understand that gtk_text_view don't play well with escape sequence related to cursor movement (they require a well defined buffer?), so looks like a good plan to separate both ...

Looks like gtk_text_view is doing his royal picky huge kludge Princess, separation of concern is a mantra in development area, i've been told !

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 24, 2016

Contributor
Shoes.app do 
  stack do
    para "Terminal test"
    button "do it" do
      #Shoes.show_console
      Shoes.terminal columns: 64, rows: 12, fontsize: 9, title: "Bug236",
        fg: "yellow", bg: "black"
      #if RUBY_PLATFORM =~ /darwin/
      #  $stderr.puts "Filenums #{STDOUT.fileno} #{STDERR.fileno}"
      #  $stdout = $stderr
      #end
      $stderr.puts "STDERR OK"
      $stdout.puts "STDOUT OK" 
      puts "Way to go!"
      puts "\033[32mGood in green?\033[00m or is \033\[35mthis better\033\[00m"
      puts "\033[31m\033[40mRed on Black\033[0m OK? \033[01mBold?\033[0m"
      puts "And \033[04;33;46mUnderline joy?\033\[0m"
    end
  end
end

Produces
screenshot from 2016-04-24 00 40 18

Contributor

ccoupe commented Apr 24, 2016

Shoes.app do 
  stack do
    para "Terminal test"
    button "do it" do
      #Shoes.show_console
      Shoes.terminal columns: 64, rows: 12, fontsize: 9, title: "Bug236",
        fg: "yellow", bg: "black"
      #if RUBY_PLATFORM =~ /darwin/
      #  $stderr.puts "Filenums #{STDOUT.fileno} #{STDERR.fileno}"
      #  $stdout = $stderr
      #end
      $stderr.puts "STDERR OK"
      $stdout.puts "STDOUT OK" 
      puts "Way to go!"
      puts "\033[32mGood in green?\033[00m or is \033\[35mthis better\033\[00m"
      puts "\033[31m\033[40mRed on Black\033[0m OK? \033[01mBold?\033[0m"
      puts "And \033[04;33;46mUnderline joy?\033\[0m"
    end
  end
end

Produces
screenshot from 2016-04-24 00 40 18

ccoupe referenced this issue Apr 24, 2016

big commit on branch 'terminal' for issue 236
* Shoes.terminal can have a hash arg see bugs/bug236.rb
  Show.show_console uses backward compatiblity defaults but
  ends up calling the same code in gtk-terminal.
* Windows is totally oblivious to all of the Linux fun. Thats Good.
* does NOT handle the log/game mode switch - GUI first.
* expect pain on OSX.
@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 24, 2016

Contributor

👠 👠 👠 👠 👠 👠
Nice

Windows is totally oblivious to all of the Linux fun

haha

we now have Shoes.terminal in lieu of Shoes.console, right ?

Contributor

passenger94 commented Apr 24, 2016

👠 👠 👠 👠 👠 👠
Nice

Windows is totally oblivious to all of the Linux fun

haha

we now have Shoes.terminal in lieu of Shoes.console, right ?

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Apr 25, 2016

Contributor

ok, fixed ! :-)

Contributor

passenger94 commented Apr 25, 2016

ok, fixed ! :-)

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 26, 2016

Contributor

I'm working on the 'game' mode for the terminal (cursor placement) and as you should expect from me, I may have gone down the wrong rabbit hole and missed some warning signs like there is one more line in the gtk_text_buffer than you create. The newline at the end is part of the line width, I could have set 'overwrite' mode and I should use text_view instead of text_buffer for some things, but not others. It's a good learning experience and doesn't affect the log mode which does continue to work.

Contributor

ccoupe commented Apr 26, 2016

I'm working on the 'game' mode for the terminal (cursor placement) and as you should expect from me, I may have gone down the wrong rabbit hole and missed some warning signs like there is one more line in the gtk_text_buffer than you create. The newline at the end is part of the line width, I could have set 'overwrite' mode and I should use text_view instead of text_buffer for some things, but not others. It's a good learning experience and doesn't affect the log mode which does continue to work.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 27, 2016

Contributor

Gaining ground (gtk linux)

Shoes.app do 
  stack do
    para "Game Mode Terminal test"
    button "do it" do
      columns = 80
      rows = 24
      Shoes.terminal columns: columns, rows: rows, fontsize: 12, title: "Test Cursor Movements",
        mode: 'game'
      ruler1 = ''
      ruler2 = ''
      (1..columns).each do |i|
        tens = i/10
        ones = i%10
        ruler1 << (tens == 0 ? ' ' : tens.to_s)
        ruler2 << ones.to_s
      end
      puts ruler1
      puts ruler2
      (3..rows+2).each {|row| puts row }
      (3..rows).each { |ln| $stdout.write "\033[#{ln};1H#{ln}"; $stdout.flush }
      $stdout.write "\033[10;10H10,10\033[20;20H20,20"
      $stdout.flush
      $stdout.write "\033[24;79HEND\033[1;1H"
      $stdout.flush
    end
  end
end

You might notice that line numbers 3..2x are drawn twice in two different ways. Part of the test. Last one wins, because that's how old school terminals work.
screenshot from 2016-04-26 22 01 07

Contributor

ccoupe commented Apr 27, 2016

Gaining ground (gtk linux)

Shoes.app do 
  stack do
    para "Game Mode Terminal test"
    button "do it" do
      columns = 80
      rows = 24
      Shoes.terminal columns: columns, rows: rows, fontsize: 12, title: "Test Cursor Movements",
        mode: 'game'
      ruler1 = ''
      ruler2 = ''
      (1..columns).each do |i|
        tens = i/10
        ones = i%10
        ruler1 << (tens == 0 ? ' ' : tens.to_s)
        ruler2 << ones.to_s
      end
      puts ruler1
      puts ruler2
      (3..rows+2).each {|row| puts row }
      (3..rows).each { |ln| $stdout.write "\033[#{ln};1H#{ln}"; $stdout.flush }
      $stdout.write "\033[10;10H10,10\033[20;20H20,20"
      $stdout.flush
      $stdout.write "\033[24;79HEND\033[1;1H"
      $stdout.flush
    end
  end
end

You might notice that line numbers 3..2x are drawn twice in two different ways. Part of the test. Last one wins, because that's how old school terminals work.
screenshot from 2016-04-26 22 01 07

ccoupe added a commit that referenced this issue Apr 28, 2016

for #236, Linux - more escape/sequences for 'game' mode
* no one cares - lots of work- no benefits. But if someone really needs
  it, tesi.c and gtk-terminal.c (gtk_text_view) can be made to work

* screams for a better test harness. Most of the work is writing test
  Wonder why abstraction is named 'curses'?
* erase line
* erase display
* 80% of an xterm. 20% finished. And that's for Linux.

ccoupe added a commit that referenced this issue May 2, 2016

on #236, osx doesn't use a pty (#224 still a probem)
* compile and call tesi differently on osx.
  don't use a pty on OSX.
* #224 - still a problem.

ccoupe added a commit that referenced this issue May 10, 2016

getting better for osx #224, #236
* now able to output from C to stdout fd - write(fileno(stdout), ...);
* fprintf(stdout) ... not working (aka printf() )
* can write to pipe with cocoa.
* much much more to do

ccoupe added a commit that referenced this issue May 11, 2016

Good enough for working on #236
* hack to use stderr for Ruby since I can't find a better way to
  tell ruby that stdout has changed.

ccoupe added a commit that referenced this issue May 14, 2016

should be the last of the commits for #224
* works well enough stdout wise. Now #236 is calling.

ccoupe added a commit that referenced this issue May 14, 2016

on #236, OSX. allow and use shoes_native_terminal arguments
* like bug236.rb uses.
* foreground (default text color) DOES NOT work.
* Progress but a slow progress. So much too learn.

ccoupe added a commit that referenced this issue May 15, 2016

for #236 -osx can handle fg, bg color args when created
* ground work for setting them from escape sequence. Easy?
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe May 15, 2016

Contributor

Getting closer.
cap236

Now that I almost know what I'm doing on OSX, setting colors from escape sequences should be relatively easy except I should never have used two classes in Cocoa when one would do and I need to change code names from console to terminal.

Contributor

ccoupe commented May 15, 2016

Getting closer.
cap236

Now that I almost know what I'm doing on OSX, setting colors from escape sequences should be relatively easy except I should never have used two classes in Cocoa when one would do and I need to change code names from console to terminal.

ccoupe added a commit that referenced this issue May 16, 2016

for #236 - color escapes and underline working for OSX
* bold is font thing and not so easy to figure out.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe May 16, 2016

Contributor

There is still much to do.
attr

Clear eyes will notice there is no bold. On OSX thats a font change thing and not so easily done, yet. Keydown doesn't work and there are mamy strange problems with Tests/test_video_vlc.rb only works in certain situations and without color and have #224 issues.

Contributor

ccoupe commented May 16, 2016

There is still much to do.
attr

Clear eyes will notice there is no bold. On OSX thats a font change thing and not so easily done, yet. Keydown doesn't work and there are mamy strange problems with Tests/test_video_vlc.rb only works in certain situations and without color and have #224 issues.

ccoupe added a commit that referenced this issue May 18, 2016

lot's of overdue cruft cleaning #236
* don't subclass NSTextView. Change some names to be 'terminal'
  instead of 'console'. Remove commented code that didn't work but
  there is still more to eliminate.
* Less embarassing to see my name as the author of this code.
  It's just better.

ccoupe added a commit that referenced this issue May 19, 2016

ccoupe added a commit that referenced this issue May 19, 2016

Fun with #236. Working on stdin handling.
* downgrade samples/expert-console to good-console. Remove all but
  readline options.
* Who knew? If readline thinks its an xterm then it gets clever with
  \esc[K; sequences. Which we can ignore in log_mode (he believes)

ccoupe added a commit that referenced this issue May 25, 2016

for #236, minor tweaks - fix some compiler warnings
* for readline. tab is 'action' key. good-console.rb on Linux
  just ignores it because so osx mostly does that too.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe May 25, 2016

Contributor

Warning! I'm likely to merge the terminal branch into master soon. OSX may still have some issues
but the branch is much better than what's in master.

If any one is bored and would like to write some ruby, we need an sample/expert-console.rb that uses some of the things found here

Contributor

ccoupe commented May 25, 2016

Warning! I'm likely to merge the terminal branch into master soon. OSX may still have some issues
but the branch is much better than what's in master.

If any one is bored and would like to write some ruby, we need an sample/expert-console.rb that uses some of the things found here

ccoupe added a commit that referenced this issue Jun 26, 2016

some additions for #224, #236, #256
* so many moving peices.

ccoupe added a commit that referenced this issue Jul 4, 2016

issue #260 - appears to be done. #256 is mostly done.
* just a couple of cosmetic things left for #236

ccoupe added a commit that referenced this issue Jul 5, 2016

for #236 - gtk terminal gets copy raw button (for debug esc)
* and copys the optional title if give. Now equivalent to osx.

ccoupe added a commit that referenced this issue Jul 7, 2016

for #236, #256 - commit a place to fall back to.
* need to get the osx terminal to have a scroll bar.

ccoupe added a commit that referenced this issue Jul 8, 2016

ccoupe added a commit that referenced this issue Jul 8, 2016

I herd that heared might be a spelling mistake #236
* also start the removal of game mode.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jul 10, 2016

Contributor

Just when I was almost done with this issue, I got to thinking about Windows (and it's too ugly for Shoes) console. Windows has pipes, and handles and even explicit calls to set/redirect stdin/stdout/stderr. We already have the xterm behavior in tesi/gtk-terminal. We just need to periodically move bytes from stdout pipe to gtk-terminal in a similar way to the osx implementation (probably much easier to do in Windows, BTW just add a gtk idle routine?. Like linux and choes on osx, choes.exe has a launch terminal which works fine (except for xterm escapes). A call to Shoes.terminal would bring up a Shoes window instead of (another) DOS console. Cross platform.

Contributor

ccoupe commented Jul 10, 2016

Just when I was almost done with this issue, I got to thinking about Windows (and it's too ugly for Shoes) console. Windows has pipes, and handles and even explicit calls to set/redirect stdin/stdout/stderr. We already have the xterm behavior in tesi/gtk-terminal. We just need to periodically move bytes from stdout pipe to gtk-terminal in a similar way to the osx implementation (probably much easier to do in Windows, BTW just add a gtk idle routine?. Like linux and choes on osx, choes.exe has a launch terminal which works fine (except for xterm escapes). A call to Shoes.terminal would bring up a Shoes window instead of (another) DOS console. Cross platform.

ccoupe added a commit that referenced this issue Jul 13, 2016

ccoupe added a commit that referenced this issue Jul 15, 2016

branch win-terminal, #236
* compiles with warnings. Would it work on Windows? Don't know.
* yes it could be modified for Linux to not use a pty. Perhaps.

ccoupe added a commit that referenced this issue Jul 16, 2016

bug #236 - branch win-terminal - much to love
* uses g_io_channels connected to pipes for stdout/stderr
* stdin untested. need buffering tweaks.
* doesn't use the gtk timer to move chars. Possible linux replacement
  to remove pty.

ccoupe added a commit that referenced this issue Jul 17, 2016

still on win-terminal branch, #236 - try no pty on linux
* without the timeout polling. Works for stdout/stderr
* much faster for things like profiler terminal output
* dogs mess of ifdef in tesi and gtk-terminal
* problems close/quit shoes - need to close/remove the g_io_channels
  watch is my guess. also affect Windows.
* Input (key -> stdin (readline) incomplete
* has promise - nobody like the pty.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jul 17, 2016

Contributor

The win-terminal branch may have hit a dead end. Without a tty/pty, readline just won't work. if you look at the code for the pure ruby readline (rb-readline gem) you'll see that it too depends on external things (infocmp command, and tty calls) that would be damn near impossible to implement in Shoes. Particularly on Windows.

The good new is that it should be possible to use the g_io_channel structure to monitor the pty fds for stdout/stderr which is noticeably faster on Linux compared to the timer method.

Contributor

ccoupe commented Jul 17, 2016

The win-terminal branch may have hit a dead end. Without a tty/pty, readline just won't work. if you look at the code for the pure ruby readline (rb-readline gem) you'll see that it too depends on external things (infocmp command, and tty calls) that would be damn near impossible to implement in Shoes. Particularly on Windows.

The good new is that it should be possible to use the g_io_channel structure to monitor the pty fds for stdout/stderr which is noticeably faster on Linux compared to the timer method.

ccoupe added a commit that referenced this issue Jul 18, 2016

for #236 - giving up on win-terminal. Too much of ruby depends
on stdin being a tty/pty and there is no pty in Windows.

ccoupe added a commit that referenced this issue Jul 18, 2016

branch master, #236 shorten timeout
* I can see it scroll but it's fast. Smaller numbers
  are *visually* nothing + wait to everything
* very ad-hoc.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jan 21, 2017

Contributor

Closed in Shoes 3.3.2

Contributor

ccoupe commented Jan 21, 2017

Closed in Shoes 3.3.2

@ccoupe ccoupe closed this Jan 21, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment