OSX: can't run test framework - sometimes #256

Closed
ccoupe opened this Issue Jun 13, 2016 · 19 comments

Comments

Projects
None yet
3 participants
@ccoupe
Contributor

ccoupe commented Jun 13, 2016

Running the Tests/ tests_video_vlc.rb and tests_svg.rb as written fails on OSX unless its done with a $ ./cshoes -w Tests/...rb command line invocation. When run **without** the -w switch (or from a Splash screen pick) then the null osx $stdout has been _captured_ in a test/unit internal variable (io) in therequire`s or setup of minitest before the call to Shoes.terminal (or deprecated Shoes.show_console).

Although this seems related to issue #224 and #236 (which are mostly fixed), it's different. The nascent test framework GUI from @passenger94 doesn't account for the idea that stdout is defined later. I suspect test/unit might have an option to change that io variable. Or it should have.

@ccoupe ccoupe added OSX Normal labels Jun 13, 2016

@ccoupe ccoupe added this to the 3.3.2 milestone Jun 13, 2016

@ccoupe ccoupe self-assigned this Jun 13, 2016

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Jun 13, 2016

Contributor

test/unit do provide an interface to hook a gui, i choose to go the console route so that i could use shoes terminal, was more interesting to me :-)

Contributor

passenger94 commented Jun 13, 2016

test/unit do provide an interface to hook a gui, i choose to go the console route so that i could use shoes terminal, was more interesting to me :-)

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 13, 2016

Contributor

I may try to move the stdout/stderr notification center loops (if you will) into regular Shoes start up on osx and unless their is a terminal window, they wouldn't update anything just flush. That way $stdout is setup earlier as it is when run with -w on the commandline.

Contributor

ccoupe commented Jun 13, 2016

I may try to move the stdout/stderr notification center loops (if you will) into regular Shoes start up on osx and unless their is a terminal window, they wouldn't update anything just flush. That way $stdout is setup earlier as it is when run with -w on the commandline.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 14, 2016

Contributor

There's a curiosity or four with test-unit and Shoes/OSX. On OSX the xterm-256color is not being found in ENV['TERM'] or it's not being passed to the right places. That looks like a test-unit bug.

There are options to TestRunner.new that might help with setting output to $stderr -but I've yet to find any doc about where run() is located and what options it supports and what I have seen say's no options.

Contributor

ccoupe commented Jun 14, 2016

There's a curiosity or four with test-unit and Shoes/OSX. On OSX the xterm-256color is not being found in ENV['TERM'] or it's not being passed to the right places. That looks like a test-unit bug.

There are options to TestRunner.new that might help with setting output to $stderr -but I've yet to find any doc about where run() is located and what options it supports and what I have seen say's no options.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jun 14, 2016

Collaborator

test/unit do provide an interface to hook a gui, i choose to go the console route so that i could use shoes terminal, was more interesting to me :-)

Shoes users would benefit from a GUI version. we should always consider the long term benefits of extending Shoes ecosystem, encouraging users to use its features and report issues.

Collaborator

BackOrder commented Jun 14, 2016

test/unit do provide an interface to hook a gui, i choose to go the console route so that i could use shoes terminal, was more interesting to me :-)

Shoes users would benefit from a GUI version. we should always consider the long term benefits of extending Shoes ecosystem, encouraging users to use its features and report issues.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 14, 2016

Contributor

@BackOrder , so far the tests are only of value to people willing to fix shoes internals. Those folks can handle the terminal output and there is GUI to select which tests and button to not display details. I'm perfectly happy running them -w on the osx command line. The problem is they should also run loaded from the splash screen on osx. It behaves badly because Shoes/OSX doesn't have working ruby STDOUT at the time test/unit is loaded.

Contributor

ccoupe commented Jun 14, 2016

@BackOrder , so far the tests are only of value to people willing to fix shoes internals. Those folks can handle the terminal output and there is GUI to select which tests and button to not display details. I'm perfectly happy running them -w on the osx command line. The problem is they should also run loaded from the splash screen on osx. It behaves badly because Shoes/OSX doesn't have working ruby STDOUT at the time test/unit is loaded.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 15, 2016

Contributor

Some progress. I updated the test/unit gem to 3.2.0 (osx was 3.0.8, linux: 3.1.7) - no change on the colorization. I think I've found the bug that causes it to fail when run w/o -w. It's likely the monkey patch on IO.puts. Completely unsure why it works with -w.

One more clarification for @BackOrder, the GUI tests in test/unit are Gtk2, Tk, or Fox. You have to 'gem install test-unit-full to get them. Not part of normal Ruby. Shoes could be added to the list but would have to be co-ordinated/distributed by test/unit developers. Not much gain for Shoes since @passenger94's code is pretty and works (mostly). Shoes Scripters who want tests are either working on a non GUI code base or they want visual testing - button press -> something on screen. The latter is very difficult cross platform/cross-gtk-theme. The former was always available to anyone who wanted to spend sometime learning tests frameworks and Shoes which @passenger has done.

Perhaps someone will be inspired to do a gui for mini-test since Ruby delivers both and evidently opinions about which is better runs deep.

Contributor

ccoupe commented Jun 15, 2016

Some progress. I updated the test/unit gem to 3.2.0 (osx was 3.0.8, linux: 3.1.7) - no change on the colorization. I think I've found the bug that causes it to fail when run w/o -w. It's likely the monkey patch on IO.puts. Completely unsure why it works with -w.

One more clarification for @BackOrder, the GUI tests in test/unit are Gtk2, Tk, or Fox. You have to 'gem install test-unit-full to get them. Not part of normal Ruby. Shoes could be added to the list but would have to be co-ordinated/distributed by test/unit developers. Not much gain for Shoes since @passenger94's code is pretty and works (mostly). Shoes Scripters who want tests are either working on a non GUI code base or they want visual testing - button press -> something on screen. The latter is very difficult cross platform/cross-gtk-theme. The former was always available to anyone who wanted to spend sometime learning tests frameworks and Shoes which @passenger has done.

Perhaps someone will be inspired to do a gui for mini-test since Ruby delivers both and evidently opinions about which is better runs deep.

ccoupe added a commit that referenced this issue Jun 15, 2016

for osx console testing #256
* no monkey patch works for OSX and test/unit
* bugs/bug236.rb is broken - that explains the missing color
* it's like something didn't get merged from terminal branch

ccoupe added a commit that referenced this issue Jun 16, 2016

on #256, better, but not correct and still very odd behaviour.
* mostly a commit in case I need to revert.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 16, 2016

Contributor

It gets deeper and deeper. So many moving parts if you can't depend on STDOUT or $stdout.
This is ./cshoes -w and looks almost OK until you look at the what the test_svg.rb has been modified to produce.. Compare to this . Look carefully - Both are wrong. Yet It's the same Shoes.terminal code run from the same script. The difference is when Shoes.terminal is called.

My current thinking is that Shoes OSX needs to create it's stdout fd from an NSPipe before initializing Ruby, (world.c, shoes_init() and gems on the paths) and store the NSFileHandles and fds globally, Then when the event loop is started add an NSNotifcationCenter observer that basically just empties the pipe so it doesn't hang Shoes with a pipe full condition. IF a script calls Shoes.terminal (or Shoes.show_console) then cocoa-term.m would replace that Observer sink with a new one that puts the chars into the NSTextView like is done now.

Would it be easier to hook into minitest as an plugin sort of thing that interprets test results and appends to a Shoes edit_box? Maybe, Maybe not.

None of that explains why bugs/bug236.rb can work (-w or not) when displaying colors.

Contributor

ccoupe commented Jun 16, 2016

It gets deeper and deeper. So many moving parts if you can't depend on STDOUT or $stdout.
This is ./cshoes -w and looks almost OK until you look at the what the test_svg.rb has been modified to produce.. Compare to this . Look carefully - Both are wrong. Yet It's the same Shoes.terminal code run from the same script. The difference is when Shoes.terminal is called.

My current thinking is that Shoes OSX needs to create it's stdout fd from an NSPipe before initializing Ruby, (world.c, shoes_init() and gems on the paths) and store the NSFileHandles and fds globally, Then when the event loop is started add an NSNotifcationCenter observer that basically just empties the pipe so it doesn't hang Shoes with a pipe full condition. IF a script calls Shoes.terminal (or Shoes.show_console) then cocoa-term.m would replace that Observer sink with a new one that puts the chars into the NSTextView like is done now.

Would it be easier to hook into minitest as an plugin sort of thing that interprets test results and appends to a Shoes edit_box? Maybe, Maybe not.

None of that explains why bugs/bug236.rb can work (-w or not) when displaying colors.

@passenger94

This comment has been minimized.

Show comment
Hide comment
@passenger94

passenger94 Jun 18, 2016

Contributor

following from a bit too far away
what a journey ... !

Contributor

passenger94 commented Jun 18, 2016

following from a bit too far away
what a journey ... !

ccoupe added a commit that referenced this issue Jun 19, 2016

on branch osx-term, #256
* Shoes.terminal works again but still oddly compared to -w launch
  No gain for branch. Boo for online Apple doc.
* -h or -g list -l don't display results so thats not good. Suggests the
  Gui (a window) needs to be on screen for Notication Center to work.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 23, 2016

Contributor

Before I could really get into what osx is doing I decided I need a better launch script for cshoes on OSX. This doesn't fix the terminal issues on OSX but if you comment out the Shoes.terminal in the tests, they do run and display colors on the launch terminal and brings back the double script bug so shoes.rb is not osx correct yet and the optsparse handling needs a little rewrite which affects all platforms - a whole lot of testing, more delays.

Contributor

ccoupe commented Jun 23, 2016

Before I could really get into what osx is doing I decided I need a better launch script for cshoes on OSX. This doesn't fix the terminal issues on OSX but if you comment out the Shoes.terminal in the tests, they do run and display colors on the launch terminal and brings back the double script bug so shoes.rb is not osx correct yet and the optsparse handling needs a little rewrite which affects all platforms - a whole lot of testing, more delays.

ccoupe added a commit that referenced this issue Jun 26, 2016

some additions for #224, #236, #256
* so many moving peices.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 28, 2016

Contributor

Now that cshoes can run the test using the launch terminal or the Shoes.terminal, I picked Tests/tests_color.rb to use for further testing of all the strange visual results. It's just a matter of commenting out the Shoes.terminal line. First I'll show linux and osx when using the native terminal.
ct-term-linux
CT-term-linux.png

ct-term-osx
CT-term-osx.png

Which is better? Which is more Correct? I'm guessing xterm-256color is not the same as whatever Linux announces and test/unit cares. There could be version differences for test/unit as well.
Let's uncomment the Shoes.terminal call and use it instead.
ct-shoes-linux
'CT-Shoes-linux.png`

Whoa! Shoes got the wrong color and dropped some of the colored characters. Not good. Also word wrapped. That a hint for dropped chars.

ct-shoes-osx
CT-Shoes-osx.png

No colors. Dropped a line of dashes and maybe more.

Shoes linux and Shoes OSX are both wrong. So many moving parts and bugs! This may never get fixed properly (I haven't tried on Windows yet which has both DOS and Msys consoles (4 more tests). OSX users can also change their emulation to xterm and xterm-16color (and many more) - another hint and series of tests to show that Shoes.terminal is not a replacement for the launch terminal.

Clear eyes might notice that OSX doesn't have a vertical scroll bar. You can scroll with the wheel though.

Contributor

ccoupe commented Jun 28, 2016

Now that cshoes can run the test using the launch terminal or the Shoes.terminal, I picked Tests/tests_color.rb to use for further testing of all the strange visual results. It's just a matter of commenting out the Shoes.terminal line. First I'll show linux and osx when using the native terminal.
ct-term-linux
CT-term-linux.png

ct-term-osx
CT-term-osx.png

Which is better? Which is more Correct? I'm guessing xterm-256color is not the same as whatever Linux announces and test/unit cares. There could be version differences for test/unit as well.
Let's uncomment the Shoes.terminal call and use it instead.
ct-shoes-linux
'CT-Shoes-linux.png`

Whoa! Shoes got the wrong color and dropped some of the colored characters. Not good. Also word wrapped. That a hint for dropped chars.

ct-shoes-osx
CT-Shoes-osx.png

No colors. Dropped a line of dashes and maybe more.

Shoes linux and Shoes OSX are both wrong. So many moving parts and bugs! This may never get fixed properly (I haven't tried on Windows yet which has both DOS and Msys consoles (4 more tests). OSX users can also change their emulation to xterm and xterm-16color (and many more) - another hint and series of tests to show that Shoes.terminal is not a replacement for the launch terminal.

Clear eyes might notice that OSX doesn't have a vertical scroll bar. You can scroll with the wheel though.

@ccoupe ccoupe added Linux Low and removed Normal labels Jun 28, 2016

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jun 28, 2016

Collaborator

I like the last screenshot the best. You still can use the colours to highlight successful and failed tests, or have an option to turn it on/off.

Collaborator

BackOrder commented Jun 28, 2016

I like the last screenshot the best. You still can use the colours to highlight successful and failed tests, or have an option to turn it on/off.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 28, 2016

Contributor

@BackOrder - you missed the point. All four of them should be similar (colored). Three are are, one incorrectly with cyan instead of green and the one you like, should have color - it's the default setting for test/unit. Shoes is not handling default color settings for two of them properly.

Contributor

ccoupe commented Jun 28, 2016

@BackOrder - you missed the point. All four of them should be similar (colored). Three are are, one incorrectly with cyan instead of green and the one you like, should have color - it's the default setting for test/unit. Shoes is not handling default color settings for two of them properly.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 28, 2016

Contributor

If you look closer, OSX native terminal sets a background of green with black text and Linux native terminal draws green dots. That's nothing to do with Shoes or the script settings (and both test/units are 3.20.0 now). Shoes.terminal ought to be close. It's not.

It could be that test/unit is sending a identify yourself escape sequence to Shoes.terminal and Shoes is dumping it on the floor and then ruby/test/unit picks something from ENV['TERM'] as a fallback and then sends half baked color setttings. That is something I can test for and supply if needed.

Contributor

ccoupe commented Jun 28, 2016

If you look closer, OSX native terminal sets a background of green with black text and Linux native terminal draws green dots. That's nothing to do with Shoes or the script settings (and both test/units are 3.20.0 now). Shoes.terminal ought to be close. It's not.

It could be that test/unit is sending a identify yourself escape sequence to Shoes.terminal and Shoes is dumping it on the floor and then ruby/test/unit picks something from ENV['TERM'] as a fallback and then sends half baked color setttings. That is something I can test for and supply if needed.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 30, 2016

Contributor

I modified the terminal code to capture the data stream before it's interpreted for esc sequences. On Linux it's sending an escape sequence I didn't handle for setting colors and it would be a mess to implement it properly - Any one want to type in an 256 element table of RGB values in C?

On OSX + Shoes.terminal there are no escape sequences being sent. zip, nada, none but the capture does have other controls chars so it is getting the raw stream from test/unit. debugging test/unit in Shoes is probably easier on OSX but I'm not looking forward to it.

Contributor

ccoupe commented Jun 30, 2016

I modified the terminal code to capture the data stream before it's interpreted for esc sequences. On Linux it's sending an escape sequence I didn't handle for setting colors and it would be a mess to implement it properly - Any one want to type in an 256 element table of RGB values in C?

On OSX + Shoes.terminal there are no escape sequences being sent. zip, nada, none but the capture does have other controls chars so it is getting the raw stream from test/unit. debugging test/unit in Shoes is probably easier on OSX but I'm not looking forward to it.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jun 30, 2016

Contributor

OSX color diagnosed. test/unit checks if the output is a tty? before checking for ENV["TERM"] and that tty test will fail on Shoes/OSX in Shoes.terminal. No way to fix that in Shoes. Tests need to explicitly ask for color.

@test_result = Test::Unit::UI::Console::TestRunner.run(tests, {use_color: true})

you can set to it false if you don't like colors. The shade of green is a little different between native xterm-256color and Shoes.terminal but I consider that very small problem unlikely to be fixed or even noticed.

Contributor

ccoupe commented Jun 30, 2016

OSX color diagnosed. test/unit checks if the output is a tty? before checking for ENV["TERM"] and that tty test will fail on Shoes/OSX in Shoes.terminal. No way to fix that in Shoes. Tests need to explicitly ask for color.

@test_result = Test::Unit::UI::Console::TestRunner.run(tests, {use_color: true})

you can set to it false if you don't like colors. The shade of green is a little different between native xterm-256color and Shoes.terminal but I consider that very small problem unlikely to be fixed or even noticed.

ccoupe added a commit that referenced this issue Jun 30, 2016

For #256 - explicity ask for colors.
* because OSX wants it and it can't be fixed in Shoes.

ccoupe added a commit that referenced this issue Jun 30, 2016

many changes - #256 and more
* raw capture data can be enabled for the 'copy' button.
  Very useful for debuging escape sequences:
* which still aren't correct
* removed VTE code
* will remove game-mode in the future. Clever but DOA for osx.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jul 1, 2016

Contributor

Tests/tests_video.rb has been a real head spinner of escape parsing because test/unit sets fg and bg in one ESC sequence sometimes. On my Linux system the audio test fails (known problem, nothing to do with Shoes or vlc) First the Linux without Shoes.terminal
cv-linux-term
Then what Shoes.terminal does (after the commit).
cv-linux-shoes

If you look closely, you'll find some oddities that only matter if you care how the tests look.

Contributor

ccoupe commented Jul 1, 2016

Tests/tests_video.rb has been a real head spinner of escape parsing because test/unit sets fg and bg in one ESC sequence sometimes. On my Linux system the audio test fails (known problem, nothing to do with Shoes or vlc) First the Linux without Shoes.terminal
cv-linux-term
Then what Shoes.terminal does (after the commit).
cv-linux-shoes

If you look closely, you'll find some oddities that only matter if you care how the tests look.

ccoupe added a commit that referenced this issue Jul 1, 2016

move CSI 38;... and 48 parsing into tesi.c #256
* convert some 256 color indices to 8 (dos?) colors.
  very incomplete.

ccoupe added a commit that referenced this issue Jul 1, 2016

close enough for #256 on osx
* not sure anyone would notice or care about the bugs.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jul 2, 2016

Contributor

For consistency I'm going to add the osx screen captures for Tests/tests_video.rb showing the error they find - a different error from linux but the point is to show the text formatting and colors.
cv-osx-term

cv-osx-shoes

There are color differences which will be important to someone I suppose. The solution is too
implement the full 256 entry table of rgb instead of my hack. There's probably a copy in libvte if I can find it and it isn't too macro-ed up. Shorter term, there is a {:color_scheme => something} option to use 8 colors if anybody can figure what the name is (a class?, a constant? ). @passenger94 ?

I think I'll create a new github issue about the color256 problem and close this one since one can run tests with or without color. The colors may be wrong in some peoples opinion but since this a very unlikely to be used part of Shoes.

I'm also going to remove any cursor positioning code. Bad idea and who cares?

Contributor

ccoupe commented Jul 2, 2016

For consistency I'm going to add the osx screen captures for Tests/tests_video.rb showing the error they find - a different error from linux but the point is to show the text formatting and colors.
cv-osx-term

cv-osx-shoes

There are color differences which will be important to someone I suppose. The solution is too
implement the full 256 entry table of rgb instead of my hack. There's probably a copy in libvte if I can find it and it isn't too macro-ed up. Shorter term, there is a {:color_scheme => something} option to use 8 colors if anybody can figure what the name is (a class?, a constant? ). @passenger94 ?

I think I'll create a new github issue about the color256 problem and close this one since one can run tests with or without color. The colors may be wrong in some peoples opinion but since this a very unlikely to be used part of Shoes.

I'm also going to remove any cursor positioning code. Bad idea and who cares?

ccoupe added a commit that referenced this issue Jul 4, 2016

issue #260 - appears to be done. #256 is mostly done.
* just a couple of cosmetic things left for #236

ccoupe added a commit that referenced this issue Jul 7, 2016

for #236, #256 - commit a place to fall back to.
* need to get the osx terminal to have a scroll bar.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jul 8, 2016

Contributor

One mystery solved.The Gtk theme determines how/which test/unit escapes are sent. on the pi2, it looks like it does on OSX - green background, black character whereas on my Ubuntu theme it's green characters on a white background. I don't know whether to laugh, cry or sigh.

Contributor

ccoupe commented Jul 8, 2016

One mystery solved.The Gtk theme determines how/which test/unit escapes are sent. on the pi2, it looks like it does on OSX - green background, black character whereas on my Ubuntu theme it's green characters on a white background. I don't know whether to laugh, cry or sigh.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jan 21, 2017

Contributor

Closed in Shoes 3.3.2

Contributor

ccoupe commented Jan 21, 2017

Closed in Shoes 3.3.2

@ccoupe ccoupe closed this Jan 21, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment