Incorporate bats extension projects

[nkakouros]

I recently came across bats-support and bats-assert. Maybe there are others too. It seems to me that there are useful features in these projects and there is the danger that they become obsolete due to lack of maintenance.

I would like to suggest merging such extensions directly into bats due to the following advantages:

• ensured maintenance side-by-side core bats
• same quality of code as core bats
• easy discovery by users
• better overview of project and its dynamics, instead of having to keep track of several git repos.

Has there been past discussion on this?

[psteinb]

see also the discussion in #40 ... this merge is needed from my point of view

[mbland]

I'm open to the possibility—I've done a bunch of my own work in https://github.com/mbland/go-script-bash (see lib/bats) that could be merged with some of the existing libraries into a standard assertion library.

That said, the trend across languages lately appears to be to provide assertion libraries separately from frameworks. Would like to hear more perspectives, particularly from the other @bats-core/bats-core members.

[jasonkarns]

Not incidentally, the two people involved in those projects are @ztombol and myself :D
And @ztombol has been extended an invitation to join bats-core as a maintainer (#4) so I would fully expect these projects to be folded in at some point.

[BraisGabin]

I think that the framework and the assertion libraries are two different things. So there's no need to merge them.

We just need to point of those libraries from this one in the documentation. And maybe later create a meta-package that includes all of them.

[sublimino]

As we already have assertions in core I'd be tempted to support this proposal and merge the better assertion libraries into core.

Assertion libraries tend to provide different styles of assertion (e.g. http://chaijs.com/guide/styles/ for assert, expect, and should).

How many styles do we expect bats to have? AFAICS it's a single style, assert: e.g. <function> <...args>. should's fluent interface is not possible in BASH, and expect is possible to imitate, but it would be messy without OO.

(It might be worth comparing some other languages here)

The assertion libraries listed here significantly improve the usability of bats. I'd prefer to carefully merge and fold the existing libraries into core, so core's good enough that it doesn't need extending any more.

[szaydel]

Are there any plans to allow for a more defined extension process, like being able to include extensions that could then be used from all tests by just invoking functions and/or using variables and such from included extensions? I ask because for example I have use cases where I want to include some comparator logic, or some result evaluation logic that I want to be available to several tests, perhaps tests in multiple git repos which are otherwise mostly unrelated projects. I don't believe there's anything like that right now.

[mbland]

I'd certainly like to look into this soon-ish, as I just mentioned in #40 and #96.

Right now you could come up with your own scheme to source relative files, or via symlinks, but I agree it'd be nice if we crafted a library scheme accessible via a standard command (beyond the current capabilities of load).

[szaydel]

I feel like there's probably room for two methods to do this, one method is to have "include-style" references in individual test files and another method being globally accessible plugins that are enabled at the time command is first executed, and all tests then have access to whatever command(s) a plugin or extension offers. As a means of dealing with name collisions, perhaps it would be possible to have namespaces so that one could have a_useful_thing, b_useful_thing and c_useful_thing.

[nylen]

I think it is a good idea to keep the bats-core lean. Otherwise it is not really bats-core, but bats-plus-the-kitchen-sink and it will contain a lot of code that most people won't use.

In the case of assertion libraries, bash already has [ and [[ which are standard in all shell scripts and very powerful. bats-core should focus on better support for these assertions (ala #24, #96). If people want to go beyond that (and learn a new syntax for assertions) then I think it's fine to expect them to go to a separate repo.

I agree it'd be nice if we crafted a library scheme accessible via a standard command (beyond the current capabilities of load).

+1

Like the rest of bats-core, interactions between the core and libraries should have good automated tests. We should link to any well-coded and well-tested libraries as recommendations from the main readme, with clear instructions to install them in individual projects.

[sublimino]

Bash is not a uniform beast, and I would expect Bats to provide a comprehensive range of assertions to prevent users from making standard-bash-mistakes (double brackets are a perfect example - to fix them as per #24 we need to preempt the Bash parser's behaviour).

If people want to go beyond that (and learn a new syntax for assertions) then I think it's fine to expect them to go to a separate repo.

The library https://github.com/ztombol/bats-assert is a standard assert_equal x y syntax. The expected alternative [[ 1 == 2 ]] doesn't work out of the box, and there are plenty other edge cases in Bash. Personally, I use Bats exclusively with the two libraries listed here as the core assertions are poor IMHO.

interactions between the core and libraries should have good automated tests
This is unenforceable unless those libraries are in core.

Agree that library loading should be easy nonetheless.

[nylen]

Interactions between the core and libraries should have good automated tests ... This is unenforceable unless those libraries are in core.

The bats-core tests would include simple dummy libraries that mimic the way libraries are commonly used. The libraries themselves should also have tests, presumably they wouldn't be an officially recommended library until then.

The expected alternative [[ 1 == 2 ]] doesn't work out of the box

This is only true for bash 3.x, correct?

and there are plenty other edge cases in Bash.

What are some other examples that may bite people who write tests using bats?

I think such issues should be worked around, and/or appropriate warnings printed (as suggested in the discussion in #24).

[sublimino]

This is only true for bash 3.x, correct?

Yep, everything from 3.2.57(1) onwards is supported https://github.com/bats-core/bats-core#supported-bash-versions.

What are some other examples that may bite people who write tests using bats?

Shellcheck has a good list of gotchas in Bash conditionals https://github.com/koalaman/shellcheck/blob/master/README.md#conditionals

The reason #24 has stalled is due to requiring a Bash parser in Bash - which is not a solved problem. There's no Bash-based Bash AST that I know of, some exist in node and golang (like the awesome, well-maintained https://github.com/mvdan/sh - but have a look at the issues list for an idea of the number of edge cases it handles).

I would suggest that as we probably can't get an AST in pure Bash, in-Bash static analysis of test code to generate appropriate warnings will be difficult and fraught with edge-cases - which a test runner should avoid at all costs as it should be relied upon for stability. Using a common, well-tested assertion library is my preference as the bounds are well-known and easier to achieve IMHO.

[nylen]

Here's another way to tackle the issues with #24.

If running a buggy version of bash, and the test is simple (no multi-line statements or tricks like ]] inside quotes), go ahead and add the || return 1. Trying to parse bash statements is not a good idea, but we can reliably detect whether a statement is simple enough to patch.

If the [[ assertion is not of a form that we can reliably parse and correct, show a warning. (This could also be done anyway, for all [[ assertions.)

To me the other gotchas fall more under "shells are weird". In order to be successful with shell programming it's necessary to learn those quirks, there's no way around that.

[grayhemp]

Just FYI - another extension-candidate to join the family #147 (bats-mock).

[schrepfler]

While decoupling is good design, considering the project almost died, with the speed of change and execution environment, I think ease of use and user adoption should come first. In other words, you either deliver the modular system quickly and be done with it or you park it and prioritise according to "batteries included" approach.

[szaydel]

I agree with @schrepfler... I also want to add that at least from where I look, bats is not a particularly easily-extensible system. It allows you to include bits, but I don't know that what is implemented now is all that good, and should be seen as a fine mechanism for extensibility, long-term.

[espoelstra]

I'm open to the possibility—I've done a bunch of my own work in https://github.com/mbland/go-script-bats (see lib/bats) that could be merged with some of the existing libraries into a standard assertion library.

That said, the trend across languages lately appears to be to provide assertion libraries separately from frameworks. Would like to hear more perspectives, particularly from the other @bats-core/bats-core members.

@mbland Will GitHub let you update your comment to reflect the new name of your repo? I keep ending up here and trying to click the link and then have to navigate back to your user and to all your repositories to find it's new name.

[mbland]

Whoops, that was a typo. Updated to https://github.com/mbland/go-script-bash.

[dotmpe]

I just referenced the issue (#135 ). There is a lot here on helper/support script, I think that is best not solved in Bats. Only add something like this maybe?

bats --load=my-load.sh

And let bats include any script and override load or other core if it wants to. Hacky perhaps. Maybe add a safe-mode (by default) to check for just that function. Whatever. But something like that will let the project maintainer or devops choose an official or third party setups or maintain a customized one.

I'm afraid I cant even think of a good, common denominator improvement for 'load'--so just provide the current basic one. But allow to overload things...

The current setup makes thinking of Bats extensions (say modules and functions) hard, because it all leans on export PATH="$BATS_ROOT/libexec/bats-core:$PATH", exec, and passing some exports and flags to libexec plumping commands.

I think maybe we can think about how or where to customize the current plumbing, and maybe introduce some types of porcelain extension points and probably extension packages. Move bits of current Bats to modules too if we can. A --load and --load-safe could be a start. Im doing an experiment with --override-preproc here tpl

@sublimino I read you look for a Bash parser, ymmv its not in Bash but I recently found a Python one with complete Bash AST in Oil's ASDL (oilshell/oil#200).

[JohnVonNeumann]

Can I just echo some support for this idea? I'm currently working with bats-core and I would greatly prefer it if I didn't have to manage multiple dependencies for what is effectively a singular purpose (testing), other testing frameworks do not ask for addons/extensions to be installed for the use of assertions as I would assume them to be fairly axiomatic in the pursuit of test coverage. Cheers for the hard work all, love the tool, thanks for keeping it alive.

[3tilley]

I fully agree with merging these helpers in. I appreciate the want for lightweight components and extensibility, but that's generally promoted in an environment with a reliable package manager. This is bash! My "package manager" is conda which is used in the data science world. It feed contains bats-core, but not the extensions. It's possible for me to try and package them myself, but that's extra stuff to maintain and go wrong. It'd be nice if the useful functionality was bundled in here and came along for the ride downstream.

[Potherca]

There have been efforts to create "a reliable package manager" for bash (Basher and BPKG come to mind) and I recall some BATS package already support NPM... But I agree it would make sense to bundle (at the very least) an assert library with BATS-core.

I know @jasonkarns is busy getting bats-assert to a state where it could be bundled. For asserts there isn't really another logical alternative.

As far as Mocks go, that's a different story... There currently isn't really one clear winner.

[3tilley]

The thing about bats is that it could be used in projects with so many different runtimes, and therefore so many different package managers. I use conda because I use Python, I'm sure there are a couple of different JS managers, there may be a C++ usecase, and even a Haskell / .NET Core users. In my opinion it would be more consistent for all of these downstream groups if there was a fatter GitHub blob to build their packages from.

Selfishly I would love assert to be included, I could be persuaded either way with mock.

[sublimino]

Just re-iterating my previous support for assertion libraries in core, specifically @jasonkarns' bats-assert.

1.2 release roadmap is here. Once the timing PR and refactoring lands in #246 then we can look where to add this "assertion library" feature.

I'd suggest this feature is v2.0 -- we maintain backwards compatibility as top priority, but also bump versions as addition of new functions in the namespace may clash with existing test suites. We'll also need to handle suites that load bats-assert already. Or we find a way to handle that condition for ultimate backwards compatibility for existing suites (my strong preference if it's possible).

There is also no bats regression/timing suite yet, that would be a useful thing to ship before we start integrating further assertions.

[Potherca]

+1 for making this a v2.0 feature.

[grayhemp]

+1 for the regression/performance tests. I stopped using some bats addons because of performance issues.

[schrepfler]

+1

[stephengroat]

+1