Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better console #236

Closed
ccoupe opened this issue Apr 6, 2016 · 42 comments
Closed

Better console #236

ccoupe opened this issue Apr 6, 2016 · 42 comments
Assignees
Labels
Milestone

Comments

@ccoupe
Copy link
Contributor

@ccoupe ccoupe commented Apr 6, 2016

The new console in Shoes 3.2 is kind of miserable - particularly on OSX but the tesi/gtk is not a shining example of perfection either. Windows is OK. The nix's use a pty that talks to a secondary event poll in Shoes. What if we could create a subprocess that runs a terminal and communicates over a PTY. Turns out there is a Ruby sdtlib for this.

$ irb
2.2.4 :001 > require 'pty'
 => true 
2.2.4 :002 > PTY.spawn('gnome-terminal')
 => [#<File:/dev/pts/4>, #<File:/dev/pts/4>, 16575] 
2.2.4 :003 > ^C
2.2.4 :003 > 

That doesn't wire up stdout<->stdin between processes but the API and we don't really want bash in the subprocess but lets wave our hands and pretend that is something we can control (probably)

Shoes goes odd if you require 'pty'

Shoes.app do
  stack do
    para "Demonstrate Pty.spawn with new Terminal"
    button "Try It" do
      #require 'pty'
      if RUBY_PLATFORM =~ /darwin/
        puts 'darwin'
        `/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal`
      else
        puts "linux"
        system('gnome-terminal')
      end
    end
  end
end

Yet if you use the Shoes irb (alt-=) you can type the same IRB commands above without error.

OSX is more like Windows. If you start a terminal from an app and you have a terminal on screen it uses that one. If you have no terminals open then it does create a new one (with a error message on it) I don't understand why you can't `PTY,spawn(...) in a Shoes script. Some sort of scope/method missing magic?

@ccoupe ccoupe added the Normal label Apr 6, 2016
@ccoupe ccoupe added this to the 3.3.2 milestone Apr 6, 2016
@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 6, 2016

Nice idea !
PTY,spawn works for me (in your example) in linux, though... well, i don't have a good understanding of all this term/tty/pty thing
Don't we need a IO.pipe like in the official doc of PTY to interact with gnome_terminal ?

otherwise, i can send a 'ls' command, for example, that way and get back the result in a para

PTY.spawn('ls') do |output, input, pid|
        input.write "hi\n"
        output.each { |line| @ret.text = @ret.text + line }
end

There is also this, don't know if it's useful, (for the present case , i mean)
http://ruby-doc.org/stdlib-2.0.0/libdoc/io/console/rdoc/IO.html

Shell library too

@IanTrudel
Copy link
Collaborator

@IanTrudel IanTrudel commented Apr 6, 2016

I had many attempts using such techniques on Windows to integrate the real IRB in Shoes but without success.

@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 6, 2016

I've done something maybe similar, connecting Shoes, IRB and some Gimp input/output :
https://github.com/passenger94/Gimp-Ruby/blob/master/lib/plug-ins/irbconsole.rb
https://github.com/passenger94/Gimp-Ruby/blob/master/lib/plug-ins/shoes_console.rb.
Works nice on my system

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 7, 2016

A bit long winded.

Actually, the Windows code for the new console is pretty nice as is - one could re write the Shoes.irb to use it (but read on before doing that)

Remember that the console works by replacing the low level file_descriptors for stdin/stdout/stderr so they point one side of pty (think one side some pipes) and the other side of the pty(pipes) is a terminal emulator. For nix/osx, that is a poorly written emulator in the same process. Clever but still a hack.

Updated sample:

Shoes.app do
  stack do
    para "Demonstrate Pty.spawn with new Terminal"
    button "Try It" do
      if RUBY_PLATFORM =~ /darwin/
        puts 'darwin'
        `/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal`
      else
        require 'pty.so'
        puts "linux stdin = #{$stdin.fileno} stdout = #{$stdout.fileno}"
        rd, wr, pid = PTY.spawn("gnome-terminal")
        wr.puts "new #{pid}  pty?"
      end
    end
  end
end

Sadly, gnome-terminal has some problems used this way. It runs a shell (and .bashrc.) and changes it's stdin/out/err to that shell process so it's no longer talking to our pty. OSX would have the same problem (probably) What would be better for the nix's is when shoes.show_console is called that the code does a fork() and the new (subprocess) runs a terminal emulator. (which is pretty close to the code above, if gnome-terminal worked here) There's also an issue that gnome-terminal is not in all Linux installs.

conceptually in C,

long pty[3] = create_pty(); 
char *[] command_line(args)
 if (pid = fork()) {
     // in child subprocess
     start_terminal(pty[0], pty[1], pty[2], command_line)  // something we wrote or the user already has
    // never returns
}

For now that's the tesi based 'thing' but now it's running in a different process and doesn't have to
call back to Shoes/ruby to display things. It couldn't because its a different process We'd have to manage the subprocess pid at quit time.

There are several use case to consider: (1) test-unit/rspec, (2) byebug/remote, (3) other gems that think they have a terminal on stdin/out/err, (4) Shoes/irb

1 and 3 are satisfied by this proposal. (2) Byebug/remote is too complicated. Let's create a 'shoesbug' method (which starts the byebug remote in the Shoes process and then creates the console with the byebug command to connect to the (parent process). Details left undiscovered and they are important. We do have Shoes & ruby in the console subprocess so we can execute the ruby code to make the server connection. Irb(4) use case: Unless it has some sort of server mode, we can't poke inside the parent process BUT the existing implementation could just read/write stdin/stdout (if irb uses readline) after creating a console. No need to manage keystrokes in Shoes or call Shoes to display things. Again, details matter. Also, we don't have to do anything for Irb.

There are many terminal emulators in the OSS world we so can get better code than tesi. (search terminal in syntapic) For linux , we also have gtk's vte. On OSX, there are not as many OSS emulators (lots of binary only shareware) and they seem designed to replace Apples terminal which is a lot more code than we need. There are other OSS state machines for handling the vt102/vt220 escapes instead of tesi and we do have OSX code to create a window, set the font, draw characters arriving from the pty and send key strokes to it. Don't have the color stuff or bold or ...

That said, the difference between what we have now and this proposal is a lot of C/Obj-C code. The difference between that start_terminal and a full app with menus .... is

main(argc,argv)  {
  start_terminal(stdin, stdout, stderr, "/bin/sh")
}

Not much different except we don't need to create or handle lots of menus or font selection, and we control what the command line args are.

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 7, 2016

I'm leaning towards using https://github.com/kika/libtsm because it could/should work with OSX as well as Linux and it has a gtk example we can start from. It's not documented in a way most of us would like -- aka misleading and confusing - pty/tty is already a deep dive for the young folk ;^). It's much better than the tesi crap.

Clone it and cd in

./autoconf.sh
./configure --enable-gtktsm
make
@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 7, 2016

The Kids Are Alright !! 😛

indeed more complete library !

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 7, 2016

Actually the way to build libtsm is

./autoconf.sh
./configure
./make 
./make install
./make gtktsm

gtktsm is a properly written program. Much can be learned. I suspect 90% can be copied for Shoes purposes. libtsm (in /usr/local here) or /usr if installed from apt-get) is a new dependency and a new lib to copy into Tight Shoes. Getting gtktsm/libtsm working in OSX will be the larger challenge. I've created a remote branch 'tsm_con' since it's going to change things a lot and we don't know if it's worth it.

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 8, 2016

Now for the problems. There are many. Deal Breakers? The gtktsm sample was not written to handle pty the way we need. I knew that from the beginning but it runs deep in the sample. Secondly, on OSX, the rules for fork() are pretty strict - its needs to be followed by an exec..(path_to_executable) before the child does anything to confuse cocoa. This is true of Gtk apps as well. We need a real executable - not some functions in the duped code. This is a big issue for OSX (new app and build and all the rake files). It also turns out that the gtktsm sample doesn't handle scrolling (and the backing buffer may not be handled). That's a lot of work just to discover it might not work well in a separate process for all the use cases.

Perhaps the current design isn't all that bad, just incomplete (can't handle multi char escapes for setting colors...) There is a lot to like about using NSTextView and GtkTextView if we could handle insert/delete char/line/range which is suspiciously like issue #144 requirements only called from the terminal state machine(tesi or libtsm) -> shoes_native_textedit_ set_font_color(). That would be a lot of methods to add to text_edit but that's the point, text_edit is not edit_box and new methods are desired. And, it wouldn't change the windows implementation of show_console since Windows will never call the pty state machine - no ptys on Windows. (and that's OK in this scheme).

The tesi/llibtsm/vt102/xterm escape sequences can help define what the Shoes native methods need to be for the text_edit widget. Tonight, there's a lot to like about this approach.

@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 9, 2016

too much problems ?

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 10, 2016

What the shoes console code does is backwards to what gtktsm and tesi was writtten for - their assumption is they a collecting key strokes and pass that to /bin/sh is a subprocess. and output from the subprocess is routed to gtk display methods. What Shoes wants is different.

Tesi is not impossible to use or to understand. It does needs as serious clean up and better comments plus function names that fit the purpose (libtsm has that problem too when used in the Shoes way).

Parsing all and responding to all escape sequences in the xterm-256 definition would be very difficult and useless since most will never be used. I short circuited tesi handling at tesi.c:54 and duped some processing into haveChar in gtk-terminal/cocoa-terminal. That hack needs to be undone. Use the 'tesi', Luke!

@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 10, 2016

i've made some basic escape sequence parsing for a ri based Ruby doc Shoes app
https://github.com/passenger94/Dance_floor/blob/master/ruby_doc.rb#L87

A Jedi craves not these things !

ccoupe added a commit that referenced this issue Apr 12, 2016
* uses tesi escape parsing instead of haveCharacter shortcut
* pass testObject* instead of void p* - be explicit.
* advance escape sequences are not written or tested -- just trying
  to get back to what haveCharacter did. Close but
* backspace delete from buffer but not on screen.
ccoupe added a commit that referenced this issue Apr 16, 2016
* fixed tesi handling of escapes for attributes. Big change
* gtk: handle attributes (poorly) - need to use Text Tags. groan.
* added many callbacks from tesi parser - haveCharacter not required
* osx is not even close to working.
* will change names from console to terminal in future.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 16, 2016

This is even harder than I thought. Tesi had bugs and never could parse xterm-color escapes in a useful manner. On OSX, with it's missing stdout, pty related things are going to be much different. Tesi may not be used for OSX - too soon to say.

gtk-terminal.c (only used on Linux) may switch to using Gtk vte instead of gtk_text_view (because VTE does all the missing stuff and seems to be part of every Linux except Raspbian which is a low value target when downloads are counted and they are used to do sudo apt-get install libvte-2.90

One last thing. I'm going to call it Shoes Terminal, visually, verbally and in the code to avoid confusion with Shoes Console in log.rb

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 16, 2016

Gtk terminal has an icon, just like osx! Windows can't play that game. Bonus points to use the packaged app name instead of "Shoes". Here's samples/expert-console.rb
screenshot from 2016-04-16 00 45 54

ccoupe added a commit that referenced this issue Apr 16, 2016
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 17, 2016

There is a lot to like with VTE - except it's a moving target. The api changes and Shoes has no idea if the destination has the API Shoe was compiled against and the internet doc is pathetic. The /usr/share/gtk-doc is better but version specific. I'll commit the gtk-terminal.c code for VTE (version 0.34 api)- who knows what your libvte-2.90 implements. Maybe someone else wants to deal with it.

screenshot from 2016-04-16 22 17 57

ccoupe added a commit that referenced this issue Apr 17, 2016
* some errors (scrolling, bg/fg colors) exist.
* vte is a moving API - not good for Shoes.
@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 17, 2016

term

vte 2.90.34

but no luck with tests in colorized output (like tests for vlc, here testing Shoes::Color)

term_test

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 17, 2016

There are some important things missing in the VTE version. (like scrolling and keyboard input). VTE is the core of gnome-terminal so it parses all the escape sequences of an xterm and responds as an xterm. That also means every Linux out there has a slightly different version of 2.90 with slightly different incompatible .so

@passenger94
Copy link
Contributor

@passenger94 passenger94 commented Apr 17, 2016

is it a lost cause then ? looks better ...

@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 17, 2016

We would have to include libvte.so with Shoes (might work) and doesn't screw up gtk3 on the running system. Odds are high the x86 linux build machine has an older vte than either of us and much older than the guy running 15.10 and the color picker.

Font's and colors can be set in the older terminal and I believe we can support attributes like color, bold, an so on. Just a matter of predefining text tags for them and then creative use of iterators to mark the start and end. Then apply the tag to the text buffer. And then something similar but different on osx.

ccoupe added a commit that referenced this issue Apr 18, 2016
* can't handle nested attributes - like setting background and foreground
  and then the text and then changing the background but not the foreground.
* GtkTextMarks and iters may not behave if your doc/coding is different from
  the gtk you compile against.
* doesn't suck too much and I learned a lot which will be useful for #144
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 18, 2016

Here's the Gtk_TextView version of the console:
screenshot from 2016-04-17 22 25 33

There's much to be done but if we're clever I think we can work in all the attributes and quite likely the very odd scroll regions and xterm cursor movement (aka ncurses)

ccoupe added a commit that referenced this issue Apr 19, 2016
* do not combine with ; in the escape sequence.  tesi bug.
* colors are for character cells - not the window.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Apr 19, 2016

The gtk_text_view terminal continues to improve.
screenshot from 2016-04-18 21 47 00

As noted in the commit, there is a tesi bug dealing with ; in combined sequences (set bold and red for example) and they don't span properly - like underline, green,text,red, text,end_underline. That will take a more clever data structure - I'm not sure its worth the effort.

ccoupe added a commit that referenced this issue Apr 19, 2016
* much to be done with tesi. It's just mostly wrong. This is better. IMO
ccoupe added a commit that referenced this issue May 14, 2016
* like bug236.rb uses.
* foreground (default text color) DOES NOT work.
* Progress but a slow progress. So much too learn.
ccoupe added a commit that referenced this issue May 15, 2016
* ground work for setting them from escape sequence. Easy?
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented May 15, 2016

Getting closer.
cap236

Now that I almost know what I'm doing on OSX, setting colors from escape sequences should be relatively easy except I should never have used two classes in Cocoa when one would do and I need to change code names from console to terminal.

ccoupe added a commit that referenced this issue May 16, 2016
* bold is font thing and not so easy to figure out.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented May 16, 2016

There is still much to do.
attr

Clear eyes will notice there is no bold. On OSX thats a font change thing and not so easily done, yet. Keydown doesn't work and there are mamy strange problems with Tests/test_video_vlc.rb only works in certain situations and without color and have #224 issues.

ccoupe added a commit that referenced this issue May 18, 2016
* don't subclass NSTextView. Change some names to be 'terminal'
  instead of 'console'. Remove commented code that didn't work but
  there is still more to eliminate.
* Less embarassing to see my name as the author of this code.
  It's just better.
ccoupe added a commit that referenced this issue May 19, 2016
ccoupe added a commit that referenced this issue May 19, 2016
* downgrade samples/expert-console to good-console. Remove all but
  readline options.
* Who knew? If readline thinks its an xterm then it gets clever with
  \esc[K; sequences. Which we can ignore in log_mode (he believes)
ccoupe added a commit that referenced this issue May 25, 2016
* for readline. tab is 'action' key. good-console.rb on Linux
  just ignores it because so osx mostly does that too.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented May 25, 2016

Warning! I'm likely to merge the terminal branch into master soon. OSX may still have some issues
but the branch is much better than what's in master.

If any one is bored and would like to write some ruby, we need an sample/expert-console.rb that uses some of the things found here

ccoupe added a commit that referenced this issue Jun 26, 2016
* so many moving peices.
ccoupe added a commit that referenced this issue Jul 4, 2016
* just a couple of cosmetic things left for #236
ccoupe added a commit that referenced this issue Jul 5, 2016
* and copys the optional title if give. Now equivalent to osx.
ccoupe added a commit that referenced this issue Jul 7, 2016
* need to get the osx terminal to have a scroll bar.
ccoupe added a commit that referenced this issue Jul 8, 2016
ccoupe added a commit that referenced this issue Jul 8, 2016
* also start the removal of game mode.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Jul 10, 2016

Just when I was almost done with this issue, I got to thinking about Windows (and it's too ugly for Shoes) console. Windows has pipes, and handles and even explicit calls to set/redirect stdin/stdout/stderr. We already have the xterm behavior in tesi/gtk-terminal. We just need to periodically move bytes from stdout pipe to gtk-terminal in a similar way to the osx implementation (probably much easier to do in Windows, BTW just add a gtk idle routine?. Like linux and choes on osx, choes.exe has a launch terminal which works fine (except for xterm escapes). A call to Shoes.terminal would bring up a Shoes window instead of (another) DOS console. Cross platform.

ccoupe added a commit that referenced this issue Jul 13, 2016
ccoupe added a commit that referenced this issue Jul 15, 2016
* compiles with warnings. Would it work on Windows? Don't know.
* yes it could be modified for Linux to not use a pty. Perhaps.
ccoupe added a commit that referenced this issue Jul 16, 2016
* uses g_io_channels connected to pipes for stdout/stderr
* stdin untested. need buffering tweaks.
* doesn't use the gtk timer to move chars. Possible linux replacement
  to remove pty.
ccoupe added a commit that referenced this issue Jul 17, 2016
* without the timeout polling. Works for stdout/stderr
* much faster for things like profiler terminal output
* dogs mess of ifdef in tesi and gtk-terminal
* problems close/quit shoes - need to close/remove the g_io_channels
  watch is my guess. also affect Windows.
* Input (key -> stdin (readline) incomplete
* has promise - nobody like the pty.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Jul 17, 2016

The win-terminal branch may have hit a dead end. Without a tty/pty, readline just won't work. if you look at the code for the pure ruby readline (rb-readline gem) you'll see that it too depends on external things (infocmp command, and tty calls) that would be damn near impossible to implement in Shoes. Particularly on Windows.

The good new is that it should be possible to use the g_io_channel structure to monitor the pty fds for stdout/stderr which is noticeably faster on Linux compared to the timer method.

ccoupe added a commit that referenced this issue Jul 18, 2016
on stdin being a tty/pty and there is no pty in Windows.
ccoupe added a commit that referenced this issue Jul 18, 2016
* I can see it scroll but it's fast. Smaller numbers
  are *visually* nothing + wait to everything
* very ad-hoc.
@ccoupe
Copy link
Contributor Author

@ccoupe ccoupe commented Jan 21, 2017

Closed in Shoes 3.3.2

@ccoupe ccoupe closed this Jan 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants