Fix macOS/Bash 3.2 breakage; eliminate subshells from exec-test, preprocess #210
Conversation
When running under Bash 3.2.57(1)-release on macOS, the following tests
would fail because `BATS_ERROR_STACK_TRACE` would be empty, and hence no
information about the actual error would get printed:
- one failing test
- failing test with significant status
- failing test file outside of BATS_CWD
This is because each of these cases use `FIXTURE_ROOT/failing.bats`, and
the `ERR` trap would not fire for its `eval "( exit ${STATUS:-1} )"`
line. Changing it to `exit ${STATUS:-1}` produced the same effect, and
changing it to `return ${STATUS:-1}` would cause the output to point to
the previous line, which executes `true`.
However, the correct status would be reported to the `EXIT` trap, so now
we call `bats_error_trap` at the very beginning of `bats_teardown_trap`.
All the existing tests now pass under Bash 3.2.57(1)-release, Bash
4.2.25(1)-release (the version from the default Ubuntu 12.04.5/Precise
image on Travis CI), and Bash 4.4.12(1)-release.
Preserves existing behavior. Next step will be to take the target variable name as the second argument.
This is part of the effort to improve performance by reducing the number of command substitutions/subshells spawned by `bats_debug_trap`. Under Bash 3.2.57(1)-release on a MacBook Pro with a 2.9GHz Intel Core i5 CPU and 8GB 1867MHz DDR3 RAM, this makes `bin/bats test/` go from: 44 tests, 0 failures real 0m7.565s user 0m3.664s sys 0m3.368s to: real 0m6.449s user 0m3.290s sys 0m2.665s
Also eliminates a subshell.
Also replaces `sed` invocation with a `while` loop, saving a subprocess.
Also, `bats-preprocess` now converts DOS/Windows CRLF line endings.
Somehow this is ever-so-slightly faster.
This is part of the effort to improve performance by reducing the number of command substitutions/subshells spawned by `bats_debug_trap`. Under Bash 3.2.57(1)-release on a MacBook Pro with a 2.9GHz Intel Core i5 CPU and 8GB 1867MHz DDR3 RAM, this makes `bin/bats test/` go from the following for the previous commit: 44 tests, 0 failures real 0m5.293s user 0m2.853s sys 0m2.087s to: real 0m4.319s user 0m2.559s sys 0m1.454s
This is in anticipation of refactoring away the `$(eval echo "$quoted_name")` command substitution.
This is part of the effort to improve performance by reducing the number of command substitutions/subshells. Under Bash 3.2.57(1)-release on a MacBook Pro with a 2.9GHz Intel Core i5 CPU and 8GB 1867MHz DDR3 RAM, this shaves off O(0.15s) from the test suite at the previous commit, but I anticipate this effect being magnified on Windows platforms.
|
AMAZING! I also noticed all those time spent in the debug handler and was trying to do something about that, but... These commits are great! Unfortunately it seems you won't have your commits merged... :( |
|
@Sylvain303 Is there any news on #150 that hasn't been posted to that thread? There hasn't been any response to it since I threw my hat in the ring as a potential maintainer. @ret2libc I've since done a few more optimizations, posted to my fork as the mbland/bats optimized-20170317 tag. It's effectively "done" in terms of optimizing by eliminating subshells and subprocesses. No improvements as dramatic as in this PR, but still substantial. I've debated posting them to this PR, or opening a new PR, but decided to wait until this one and #150 finally get traction. |
|
@Sylvain303 thanks I'm already aware of that ;) @mbland that's really cool, thanks. I hope this thing will get some attention soon. |
|
Oh, and for anyone who isn't familiar with managing repos with multiple remotes, the Git commands to pull my changes into your repo should be something like: $ git remote add mbland git@github.com:mbland/bats.git
$ git fetch mbland --tags
$ git checkout -b optimized optimized-20170317To see the tag comments with the summaries (whereby $ git show optimized-20170205
$ git show optimized-20170317 |
|
@mbland, no, no special news for #150, I'm just optimist. :) Coming back to your PR, here: https://github.com/sstephenson/bats/pull/210/files#diff-669b297cddac27dc2ecc950eb92aa5fcL316 Are changes equivalent? I'm trying to merge your PR in my internal unittest. I'm using a "python trick like" sourcing detection: My current bats-preprocess version (with your modifications). As you changed in a sub-shell is spawned for: It would be easier for me to keep source detection feature, than having to rely on some other trick. Could you also enable issues, on your bats fork, so I can comment there? |
|
@mbland, I found where you've handled '\r' dos end-of-line: https://github.com/sstephenson/bats/pull/210/files#diff-e5a78c356c58edaf23c12ce3dae62023R39 great! |
| body="${BASH_REMATCH[2]}" | ||
| name="$(eval echo "$quoted_name")" |
There was a problem hiding this comment.
doesn't removing eval, line 41 on $quoted_name, remove expected behavior on variable expansion in the string or such?
These changes produce some substantial performance benefits across all platforms, but most especially on Windows. I'm using them in the mainline of my own project as of mbland/go-script-bash#165 (Travis CI results, Coveralls results).
If this PR contains too many changes at once, I'm happy to break it up. Also, I just noticed that this PR addresses #98, without even having looked at #122 before.
cc: @ztombol @stroupaloop
macOS fix
The first commit in this set fixes the three existing tests that failed for Bash 3.2.57(1)-release, the default version for macOS 10.12.3 and other earlier versions of macOS and OS X. This was due to the
ERRtrap not always firing as it should; adding some guard logic tobats_error_trapbefore calling it frombats_teardown_trapresolved the issue.Performance impact on Bats's own test suite
The remaining commits methodically eliminate subshells to improve overall test performance, especially those spawned by
bats_capture_stack_traceviabats_debug_trap.Under Bash 3.2.57(1)-release on a MacBook Pro with a 2.9GHz Intel Core i5 CPU and 8GB 1867MHz DDR3 RAM, the timing for the original set of tests (which don't include the four test name-related cases added by one of the later commits in this series) run via
bin/bats test/went from:to:
Performance impact on a large Bats test suite
In my https://github.com/mbland/go-script-bash project, I'd already optimized the tests to disable Bats stack trace collection almost as much as possible, bringing the time for the full
./go testsuite on the same machine, OS, and Bash version from O(7-8min) down to the following at mbland/go-script-bash@69ad67f:Running again at the same commit, but using Bats with these changes included, the suite now runs in:
Performance impact on Windows
The impact is even more dramatic on Windows. On the same MacBook Pro, running Windows 10 under VMware Fusion 8.5.3 with 4GB RAM, and the timing for
./go testbefore my changes to go-script-bash was in the O(50-60min) range for every version of Bash I have installed. Afterwards, it was down in the O(20+min range).Using Bats with these changes included, running the MSYS2-based Bash 4.3.46(2)-release that ships with (what is probably a slightly outdated version of) Git for Windows, the suite now runs in:
Running Bash 4.4.11(2)-release running under Cygwin:
And running Bash 4.3.11(1)-release as part of the Windows Subsystem for Linux (i.e. Ubuntu on Windows 10):
Other versions tested
In addition to the Bash versions mentioned above, I've also tested these changes using the following Bash versions:
All of the go-script-bash tests passed on these other platforms in less than half the time from before. All of the Bats tests passed on every platform with the exception of a few on Alpine Linux; I can perhaps address these failures in the future.
Future steps
There are other subshells that can likely be eliminated in other parts of Bats, particularly those that happen at startup when
bin/batsis collecting information on the test suite, which introduces a bit of a startup pause proportional to the number of tests to run (especially noticeable on Windows). However, this seems like a good place to stop for now.