Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work with preloaders #6

Open
trans opened this issue Oct 31, 2013 · 33 comments
Open

Work with preloaders #6

trans opened this issue Oct 31, 2013 · 33 comments

Comments

@trans
Copy link
Member

trans commented Oct 31, 2013

From issue #5:

"I have tried https://github.com/rubyworks/tapout/wiki/Quick-Start, but it can't work well with rails 4, i think to add config like Minitest::Reporters.use! Minitest::Reporters::SpecReporter.new would be great. then it could work with spring or zeus."

@doabit
Copy link
Contributor

doabit commented Nov 15, 2013

With spring, can use spring rake test | tapout pretty, with guard-minitest,

guard :minitest, spring: true, cli: '| tapout pretty'  do
  # Rails 4
  watch(%r{^app/(.+)\.rb})                               { |m| "test/#{m[1]}_test.rb" }
  watch(%r{^app/controllers/application_controller\.rb}) { 'test/controllers' }
  watch(%r{^app/controllers/(.+)_controller\.rb})        { |m| "test/integration/#{m[1]}_test.rb" }
  watch(%r{^app/views/(.+)_mailer/.+})                   { |m| "test/mailers/#{m[1]}_mailer_test.rb" }
  watch(%r{^lib/(.+)\.rb})                               { |m| "test/lib/#{m[1]}_test.rb" }
  watch(%r{^test/.+_test\.rb})
  watch(%r{^test/test_helper\.rb}) { 'test' }
end

@doabit
Copy link
Contributor

doabit commented Nov 17, 2013

@trans Sorry for my questions, minitap can't work with pry, if use binding.pry in test file, run ruby -Itest test/models/post_test.rb | tapout, tests will stop, input exit, got:

ruby -Itest test/models/post_test.rb | tapout
exit
/Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:369:in `parse': (<unknown>): control characters are not allowed at line 1 column 1 (Psych::SyntaxError)
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:369:in `parse_stream'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:317:in `parse'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:244:in `load'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/parsers/yaml.rb:47:in `handle'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/parsers/yaml.rb:34:in `<<'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/parsers/yaml.rb:24:in `consume'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/cli.rb:80:in `cli'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/bin/tapout:3:in `<top (required)>'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/bin/tapout:23:in `load'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/bin/tapout:23:in `<main>'

post_test.rb

require 'test_helper'

class PostTest < ActiveSupport::TestCase
  test "the truth" do
    @post = Fabricate(:post)
    assert @post.valid?
    binding.pry
  end
end

@trans
Copy link
Member Author

trans commented Nov 17, 2013

Take off the | tapout and look at the raw output. What does it look like?

@trans
Copy link
Member Author

trans commented Nov 17, 2013

Btw, what's happening is you're piping pry's interface into tapout, and tapout is trying to interpret that as TAP-Y/J test output. I probably can teach tapout to stop when it hits a document end marker, i.e. .... But I am not sure how to release control back to Pry. Maybe tapout would have to fall into a mode where it is doing nothing but routing stdin and stdout. I don't know, that seems odd. Will have to think about it. Any suggestions welcome.

@doabit
Copy link
Contributor

doabit commented Nov 17, 2013

Take off the | tapout, it works,

$ rake test
---
type: suite
start: '2013-11-18 00:23:45'
count: 18
seed: 62546
rev: 4

From: /Users/doabit/Sites/test/rails/tmp/minit/blog/test/models/post_test.rb @ line 7 PostTest#test_the_truth:

    4: test "the truth" do
    5:   @post = Fabricate(:post)
    6:   assert @post.valid?
 => 7:   binding.pry
    8: end
$ rake test | tapout
exit
/Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:369:in `parse': (<unknown>): control characters are not allowed at line 1 column 1 (Psych::SyntaxError)
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:369:in `parse_stream'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:317:in `parse'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/psych-2.0.1/lib/psych.rb:244:in `load'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/parsers/yaml.rb:47:in `handle'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/parsers/yaml.rb:34:in `<<'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/parsers/yaml.rb:24:in `consume'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/lib/tapout/cli.rb:80:in `cli'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/tapout-0.4.2/bin/tapout:3:in `<top (required)>'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/bin/tapout:23:in `load'
    from /Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/bin/tapout:23:in `<main>'

With spring, it couldn't work, either with | tapout or not. https://gist.github.com/doabit/7515078

@trans
Copy link
Member Author

trans commented Nov 17, 2013

Hmm... is spring+pry working when you don't use tap output?

@doabit
Copy link
Contributor

doabit commented Nov 17, 2013

Yes, i also tested minitest-reporters, it can work with spring + pry. i thought, tapout wait minitap to return yml or json format result, but binding.pry broken it

@trans
Copy link
Member Author

trans commented Nov 17, 2013

It pipes the output as it happens. If it waited then nothing would output until the very end, after all tests were run.

Not sure why it shouldn't work with spring if the output is not being piped to tapout. It's just a standard Minitest reporter, so it should work with anything. But then I am not sure how spring works. Did you say that minitap + spring works okay w/o pry in the mix?

@doabit
Copy link
Contributor

doabit commented Nov 17, 2013

Yes, if not use minitap, everything works well, i also can work with spring + pry + minitest-reporters. I thought, minitap reuturn json or yml fomat result, but when tests stoped, would get JSON::ParserError or (Psych::SyntaxError)

$ rake test | tapout
exit
/Users/doabit/.rbenv/versions/2.0.0-p247/gemsets/4.1.0.beta/gems/json-1.8.1/lib/json/common.rb:155:in
 `parse': 795: unexpected token at '{"type"=>"suite", "start"=>"2013-11-18 01:01:50", "count"=>18, "seed"=>56585, "rev"=>4} (JSON::ParserError)

@doabit
Copy link
Contributor

doabit commented Nov 17, 2013

Pry with minitest-reporterts

$ rake test
Started

PostTest
[WARN] cannot load such file -- awesome_print
$ gem install awesome_print

From: /Users/doabit/Sites/test/rails/tmp/minit/blog/test/models/post_test.rb @ line 7 PostTest#test_the_truth:

    4: test "the truth" do
    5:   @post = Fabricate(:post)
    6:   assert @post.valid?
 => 7:   binding.pry
    8: end

[1] pry(#<PostTest>)> exit
  test_the_truth                                                  PASS (2.18s)

PostsControllerTest
  test_should_create_post                                         PASS (0.02s)
  test_should_destroy_post                                        PASS (0.01s)
  test_should_get_edit                                            PASS (0.04s)
  test_should_get_index                                           PASS (0.01s)
  test_should_get_new                                             PASS (0.05s)
  test_should_show_post                                           PASS (0.01s)
  test_should_update_post                                         PASS (0.01s)

PostsTest
  test_the_truth                                                  PASS (0.05s)

UserTest
  test_the_truth                                                  PASS (0.00s)

UsersControllerTest
  test_should_create_user                                         PASS (0.02s)
  test_should_destroy_user                                        PASS (0.01s)
  test_should_get_edit                                            PASS (0.01s)
  test_should_get_index                                           PASS (0.01s)
  test_should_get_new                                             PASS (0.01s)
  test_should_show_user                                           PASS (0.01s)
  test_should_update_user                                         PASS (0.01s)

UsersTest
  test_login_player                                               PASS (0.03s)

Finished in 2.48854s
18 tests, 30 assertions, 0 failures, 0 errors, 0 skips

Spring + pry + minitest-reporters

$ spring rake test
Started

PostTest
[WARN] cannot load such file -- awesome_print
$ gem install awesome_print

From: /Users/doabit/Sites/test/rails/tmp/minit/blog/test/models/post_test.rb @ line 7 PostTest#test_the_truth:

    4: test "the truth" do
    5:   @post = Fabricate(:post)
    6:   assert @post.valid?
 => 7:   binding.pry
    8: end

[1] pry(#<PostTest>)> exit
  test_the_truth                                                  PASS (2.42s)

PostsControllerTest
  test_should_create_post                                         PASS (0.02s)
  test_should_destroy_post                                        PASS (0.01s)
  test_should_get_edit                                            PASS (0.09s)
  test_should_get_index                                           PASS (0.01s)
  test_should_get_new                                             PASS (0.01s)
  test_should_show_post                                           PASS (0.01s)
  test_should_update_post                                         PASS (0.01s)

PostsTest
  test_the_truth                                                  PASS (0.05s)

UserTest
  test_the_truth                                                  PASS (0.00s)

UsersControllerTest
  test_should_create_user                                         PASS (0.02s)
  test_should_destroy_user                                        PASS (0.01s)
  test_should_get_edit                                            PASS (0.01s)
  test_should_get_index                                           PASS (0.01s)
  test_should_get_new                                             PASS (0.05s)
  test_should_show_user                                           PASS (0.01s)
  test_should_update_user                                         PASS (0.01s)

UsersTest
  test_login_player                                               PASS (0.02s)

Finished in 2.77927s
18 tests, 30 assertions, 0 failures, 0 errors, 0 skips

@trans
Copy link
Member Author

trans commented Nov 17, 2013

Yea, it makes perfect sense why Pry doesn't work when piping output to tapout. Unlike minitest-reporters which just integrates itself into minitest so there is only one program running, with minitap we have two different programs running, minitest and tapout, and they are communicating via a pipe --the output from minitest is being passed into tapout, and tapout is interpreting that input as tap-y/j formatted test output. If suddenly some other data comes across the pipe, e.g. a Pry command prompt, tapout cannot interpret that and a YAML or JSON syntax error is going to happen.

I have an idea that should work around this, see rubyworks/tapout#6. But I still have to figure out the details which may take a bit of time.

However none of this explains why spring + minitap + pry doesn't work, just as long as the output is NOT being piped to the tapout command (no | tapout). It is weird that spring + minitap would work, and pry + minitap would work, but not all three. I will have to dig in and see what I can find.

@doabit
Copy link
Contributor

doabit commented Nov 17, 2013

Got it, thanks for your job.

@trans
Copy link
Member Author

trans commented Nov 18, 2013

Just released minitap-0.5.2 (along with tapout-0.4.3). For the most part this should resolve this issue and many related ones. For optimal results one should emit a DLE ASCII code (16.chr) to tell tapout to pause test result processing, and then send a ETB ASCII code (23.chr) to tell it to resume. e.g.

test "the truth" do
    @post = Fabricate(:post)
    assert @post.valid?

    STDOUT.puts 16.chr
    binding.pry
    STDOUT.puts 23.chr
end

I use STDOUT here and not $stdout b/c test runners generally cache $stdout as part of test results. In the future this might need to be addressed more precisely in Minitest's api, but STDOUT does the trick for now.

Note, this doesn't necessarily address spring + pry + minitap (but one can hope!), as there appears to be something else afoot there. Nonetheless I am going to go ahead can close this issue. We can start new more specific issues for anything that remains.

@doabit
Copy link
Contributor

doabit commented Nov 19, 2013

Thanks, i have tested, but it looks still not work well

$ rake test | tapout
exit
Started w/ Seed: 64378

From: /Users/doabit/Sites/test/rails/tmp/minit/mit/test/models/post_test.rb @ line 6 PostTest#test_the_truth:

    4: test "the truth" do
    5:   STDOUT.puts "..."  0;34# tells tapout to pause processing
 => 6:    binding.pry
    7:    STDOUT.puts "---"
    8:   assert true
    9: end

---
---
type: case
subtype: ''
label: PostTest
level: 0
stdout: "$ gem install awesome_print\n\e[0G"
stderr: |
  [WARN] cannot load such file -- awesome_print
---
type: test
subtype: ''
status: pass
label: test_the_truth
time: 1.295685
---
type: final
time: 1.296649
counts:
  total: 1
  pass: 1
  fail: 0
  error: 0
  omit: 0
  todo: 0
...

I thought the main resean is tapout. Now the process is minitap -> tapout, if use binding.pry, tapout won't accept correct result. Maybe minitest-reporters's process is the only way

@trans
Copy link
Member Author

trans commented Nov 19, 2013

It's just a bug in the yaml parser. It should work if you use 16.chr and 23.chr. I don't get the warning about awesome_print though. What does your Rakefile test take look like?

@doabit
Copy link
Contributor

doabit commented Nov 19, 2013

I use rails deault Rakefile, if not use rails

$ ruby -Itest test/main_test.rb | tapout
[1] pry(#<Destroying AR models>)> exit
Started w/ Seed: 3239

From: /Users/doabit/Sites/test/rails/tmp/minit/dd/test/main_test.rb @ line 8 Destroying AR models#test_0001_true test:

     5: it "true test" do
     6:   assert true
     7:   STDOUT.puts "..."  0;34# tells tapout to pause processing
 =>  8:    binding.pry
     9:    STDOUT.puts "---"
    10: end

---
---
type: case
subtype: ''
label: Destroying AR models
level: 0
stdout: "$ gem install awesome_print\n\e[0G"
stderr: |
  [WARN] cannot load such file -- awesome_print
---
type: test
subtype: ''
status: pass
label: true test
time: 2.988121
---
type: test
subtype: ''
status: fail
label: fail test
exception:
  message: Failed assertion, no message given.
  class: Minitest::Assertion
  file: test/main_test.rb
  line: 13
  source: assert false
  snippet:
  - 11: ''
  - 12: '  it "fail test" do'
  - 13: '    assert false'
  - 14: '  end'
  - 15: end
  backtrace:
  - test/main_test.rb:13
time: 2.989053
---
type: final
time: 2.990012
counts:
  total: 2
  pass: 1
  fail: 1
  error: 0
  omit: 0
  todo: 0
...

test_helper.rb

ENV["RACK_ENV"] = "test"
require "minitest"
require "minitest/autorun"
require "minitap"

Minitest.reporter = Minitap::TapY

@trans
Copy link
Member Author

trans commented Nov 19, 2013

I have the YAML parser fixed and will release shorty. It turns out the new document marker isn't even needed to restart. But really the more robust solution is to use the codes 16.chr and 23.chr. For convenience, in test_helper.rb one could do something like:

def pry!(binding_of_caller)
  STDOUT.puts 16.chr
  binding_of_caller.pry
  STDOUT.puts 23.chr
end

Then use it:

test "the truth" do
    @post = Fabricate(:post)
    assert @post.valid?

    pry!(binding)
end

@trans
Copy link
Member Author

trans commented Nov 19, 2013

Pushed tapout-0.4.4.

@doabit
Copy link
Contributor

doabit commented Nov 19, 2013

I have tested, maybe my problem, it still not work, run

$ ruby -Itest test/main_test.rb | tapout

stoped here:

[1] pry(#<Destroying AR models>)>

Then, input exit

1] pry(#<Destroying AR models>)> exit

From: /Users/doabit/Sites/test/rails/tmp/minit/dd/test/main_test.rb @ line 6 Destroying AR models#test_0001_true test:

    5: it "true test" do
 => 6:   pry!(binding)
    7:   assert true
    8: end

Started w/ Seed: 15732
.


Finished in 2.693s (0.371 test/s, 2.692543s avg.)

1 tests: 1 pass, 0 fail, 0 errs, 0 todo, 0 omit

I also tried to modify minitap like this

    def tapout_before_suite()
      result = super().to_json
      Tapout::JsonParser.new({format: 'dot', input: StringIO.new(result)})
    end

But it not work, i'll try it again, thanks.

@trans
Copy link
Member Author

trans commented Nov 19, 2013

That's confusing. Your output almost looks correct except somehow Pry stepped in before the From ... stuff?

@doabit I really appreciate all your help on this. You've already helped improve minitap/tapout in ways I did not think possible. Thank you.

It's been a bit difficult to compare results b/c we are not working off the same code base. Do you think you can create a public test repo of a mock Rails project that we could both use for this?

@doabit
Copy link
Contributor

doabit commented Nov 19, 2013

@trans I have created a demo app

@trans
Copy link
Member Author

trans commented Nov 19, 2013

Awesome! That is a Huge help.

I've already tracked down one issue that solves spring rake test. -- It was the arity code, which I expected might be a problem. I wish there was a cleaner way to do it. But I was able to improve it and get it working.

The other issue with | tapout is most likely a stdout flushing issue. I need to figure out where and how to ensure stdout is not cached.

@doabit
Copy link
Contributor

doabit commented Nov 19, 2013

I thought to use | tapout maybe a not good idea. The problem is minitap must return a complete json or yml result, then tapout converted them and print. But, when we use pry, it would break the process, minitap cound't return an compelte result. I have no idea how to solve it, look forward to your solution, thanks

@trans
Copy link
Member Author

trans commented Nov 19, 2013

Not so, it doesn't have to return a complete json or yml. It's a data stream, so if everything is working as it should, it processes the stream continuously in discrete chunks, not all at once.

I made some headway in understanding the problem. Have a look at pry/pry#1011

@doabit
Copy link
Contributor

doabit commented Nov 20, 2013

Oh, i see. I always thought tapout would format result until minitap completed. eg.

$ rake test | tapout
Started w/ Seed: 64378

From: /Users/doabit/Sites/test/rails/tmp/minit/mit/test/models/post_test.rb @ line 6 PostTest#test_the_truth:

    4: test "the truth" do
    5:   STDOUT.puts "..."  0;34# tells tapout to pause processing
 => 6:    binding.pry
    7:    STDOUT.puts "---"
    8:   assert true
    9: end

---
type: test
subtype: ''
status: pass
label: test_the_truth
time: 1.295685
---
type: final
time: 1.296649
counts:
  total: 1
  pass: 1
  fail: 0
  error: 0
  omit: 0
  todo: 0
...

It didn't format results. but YML.

@trans
Copy link
Member Author

trans commented Nov 20, 2013

I am suddenly getting:

Could not find gem 'arel (~> 5.0.0) ruby', which is required by gem 'rails (>= 0) ruby', in any of the sources.

when I run bundle install on the minitap_demo project.

@doabit
Copy link
Contributor

doabit commented Nov 20, 2013

arel (~> 5.0.0) ? I should arel (4.0.1) https://github.com/doabit/minitap_demo/blob/master/Gemfile.lock#L60

@trans
Copy link
Member Author

trans commented Nov 20, 2013

I had to rebundle. It's okay, I fixed by adding gem 'arel', github: 'rails/arel'.

@doabit
Copy link
Contributor

doabit commented Nov 20, 2013

Oh, may be rails updated. It works in my computer

@trans
Copy link
Member Author

trans commented Nov 20, 2013

Okay. Things are looking up, finally. Turns out it had nothing to do with Pry, Rails or Rake. All that was needed was to add io.sync = true. I had tried that before --which is why I was thinking it had something to do with one of the afore-mentioned projects. But now I think it was actually Minitest affecting it, b/c all I had to do was make sure I called io.sync = true after initializing MiniTest's reporter base class, instead of before, and it worked.

@doabit
Copy link
Contributor

doabit commented Nov 20, 2013

Great, i have updated minitap to 0.5.3, it works with pry now. But there is still a problem, we must use:

def pry!(binding_of_caller)
  STDOUT.puts 16.chr
  binding_of_caller.pry
  STDOUT.puts 23.chr
end

I thought many people may be more like to use binding.pry directly.

@trans
Copy link
Member Author

trans commented Nov 20, 2013

I've been playing with some ways to improve on that, but it's just a matter of how things work. I don't think there is any way to avoid it all together. One idea:

require 'tapout/utitlity'

tapout.pause { binding.pry }

Another option is to override the #pry method, but that probably has it's own set of problems.

Maybe just offering an alternative:

require 'tapout/utitlity'

binding.tapry  # ?

@doabit
Copy link
Contributor

doabit commented Nov 20, 2013

Yes, that is indeed, Add it into README maybe the best way now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants