Implement test coverage via kcov in our CI

[mbland]

I've used https://github.com/SimonKagstrom/kcov to great effect to collect coverage on mbland/go-script-bash and publish it to Coveralls.

The only wrinkle is that I clone and build the kcov repo fresh every time, since the Ubuntu package is a bit old. For this project and that one, I should look into finding a way to install an updated binary.

[particleflux]

You could just cache the compiled kcov folder. This is our Circle-CI implementation (should be similar on travis):

dependencies:
  cache_directories:
    - kcov-34
  override:
    - |
      if [[ ! -d "kcov-34" ]]; then
        wget 'https://github.com/SimonKagstrom/kcov/archive/v34.zip' -O /tmp/kcov.zip
        unzip /tmp/kcov.zip
        cd kcov-34
        cmake .
        make -j$(nproc)
      else
        cd kcov-34
      fi
      sudo make install

[alexRambo]

Not sure if this is the right place to ask, but does anyone have an example (command and index.html output) of kcov working with bats???
Problem: When the following command is run (kcov being fed bats bash file with arguments):
kcov [outputdir]/coverage/ bats [bats_test_path]/test_AFF_Simple.bats
the OUTPUT is 0% code coverage for the "bats" only with 20 lines instrumented and 0 hit.
This is the file output:

Coverage Report

 
Command:batsLine numberDate:2018-06-13 19:44:55Instrumented lines:20HitsCode covered:0.0%Executed lines:0Order | Command: | bats | Line number | Date: | 2018-06-13 19:44:55 | Instrumented lines: | 20 | Hits | Code covered: | 0.0% | Executed lines: | 0 | Order
Command: | bats | Line number
Date: | 2018-06-13 19:44:55 | Instrumented lines: | 20 | Hits
Code covered: | 0.0% | Executed lines: | 0 | Order
 
#!/usr/bin/env bash
2
30 set -e
4
50 export BATS_READLINK='true'
60 if command -v 'greadlink' >/dev/null; then
70 BATS_READLINK='greadlink'
80 elif command -v 'readlink' >/dev/null; then
90 BATS_READLINK='readlink'
10 fi
11
12 bats_resolve_link() {
130 if ! "$BATS_READLINK" "$1"; then
140 return 0
15 fi
16 }
17
18 bats_absolute_path() {
190 local cwd="$PWD"
200 local path="$1"
210 local result="$2"
22
230 while [[ -n "$path" ]]; do
240 cd "${path%/}"
250 path="$(bats_resolve_link "${path##/}")"
26 done
27
280 printf -v "$result" -- '%s' "$PWD"
290 cd "$cwd"
30 }
31
320 export BATS_ROOT
330 bats_absolute_path "$0" 'BATS_ROOT'
340 BATS_ROOT="${BATS_ROOT%/*}"
350 exec "$BATS_ROOT/libexec/bats" "$@"

Any and ALL help is appreciated.
Note: I managed to get code coverage for PYTHON files; but NOT for BAT files....

Thanks,
-Alex

[mbland]

@alexRambo I'm having a hard time parsing what you're doing, but here's an example:

https://travis-ci.org/mbland/go-script-bash/jobs/390296132#L641

which corresponds to:

https://coveralls.io/builds/17410861

Also note that it's using kcov v35:

https://travis-ci.org/mbland/go-script-bash/jobs/390296132#L521

@particleflux I definitely need to steal that for both bats-core and go-script-bash! 🙂

[alexRambo]

Hi @mbland, thanks for the quick reply and useful links. Trying to understand some of the flags your CI is using... they seem to coming from "kcov --uncommon-options"... not sure if there are some 'special flags' that will allow me to get kcov code coverage for .bats files...

Basically, I have a .bats file that is calling my .sh file:
kcov [outputdir]/coverage/ bats [bats_test_path]/test_AFF_Simple.bats
BUT, looks like something is wrong when calling bats because I don't see my .sh files in the code coverage (even though output says tests are passing with flying colors).

Looking through your comments.
Note: I am inside a docker container running latest kcov github repo from a few days ago.

[mbland]

Ah. The latest kcov has issues. I put a bunch of notes in mbland/go-script-bash#244 that I'll follow up on eventually. I'd recommend trying to use v35 explicitly and see if that helps.

[alexRambo]

Apologies, running kcov v33 as per this github docker image:
https://github.com/Ragnaroek/kcov_docker.
Again, NOT running kcov v35.

Update: (bad news)
Even trying to run the command "kcov --include-path=. coverage bats tests/" with the 2 files as outlined in this repo "https://github.com/particleflux/kcov-bats-circleci-codeclimate/blob/master/circle.yml" Kcov is still giving me 0% and only the "bats" file is shown (but not the actual hello.sh file).

[alexRambo]

Update:
So after many failed permutation attempts trying to get kcov to work with BATs, I'm skeptical it even works. I see that kcov works with .sh files; I've been successful myself with this approach at the lab.
Unfortunately, we are using the BATs framework for testing our bash files and it isn't playing well with kcov. The result is that only "bats" executable shows up with 0% code coverage; the ACTUAL bash files under test do not even show up.
Any ideas and suggestions would be appreciated.
I'll be deep-diving into the kcov code to understand what may be going wrong; I'll also look into some other bash testing frameworks besides BATs.

Thanks,
-Alex

[mbland]

The problem almost certainly is with kcov or how you're calling it, not with Bats itself. Try something like the following, where "$include_pattern" matches file and directory names (relative to $PWD) you want included in the coverage report, and "$exclude_pattern" matches file and directory names you want excluded:

$ kcov --bash-dont-parse-binary-dir \
   --include-pattern="$include_pattern " \
   --exclude-pattern="$exclude_pattern" \
   path/to/coverage/results/dir \
   bats path/to/tests

For example, the command from https://travis-ci.org/mbland/go-script-bash/jobs/390296132, which posted coverage results to: https://coveralls.io/builds/17410861

$ tests/kcov/src/kcov \
  --bash-dont-parse-binary-dir \
  --include-pattern=go,go-core.bash,go-template,lib/,libexec/,scripts/ \
  --exclude-pattern=/tmp/,tests/bats/ \
  --coveralls-id=390296132 \
  tests/coverage \
  /home/travis/build/mbland/go-script-bash/go test

[alexRambo]

@mbland, thanks for the quick feedback.
Unfortunately, tried the following with same results... empty "bat" executable showing 0% covered....
kcov --bash-dont-parse-binary-dir --include-pattern=/user/src/xyz/Bats_Tests/,/user/src/xyz/Bats_Tests/tests,/usr/src/xyz/,/user/src/xyz/,/user/src/,/user/src/ /usr/src/xyz/coverage/ bats /usr/src/xyz/Bats_Tests/tests/

Folder structure:
/usr/src/xyz/Bats_Tests/ => contains the "actual bash files" such as "hello.sh" being called by test files
/usr/src/xyz/Bats_Tests/tests/ => contains bats test files (including hello1.bats"

Contents of test file "/usr/src/xyz/Bats_Tests/tests/hello1.bats":
#!/usr/bin/env bats

@test "hello without arguments" {
run ~/../hello.sh
#result=$(bash $BATS_TEST_DIRNAME/../hello.sh)
#[[ "$result" == "Hello unknown" ]]
[ $status -eq "Hello unknown" ]
}

@test "hello with world argument" {
result=$(bash $BATS_TEST_DIRNAME/../hello.sh world)
[[ "$result" == "Hello World" ]]
}

Contents of test file "/usr/src/xyz/Bats_Tests/hello.sh":
#!/bin/bash

if [[ "$1" == "world" ]]; then
echo "Hello World"
else
echo "Hello unknown"
fi

Again, running the hello1.bats file succeeds in running the unit test (UT). It shows 1 success and 1 failure for the "/usr/src/xyz/Bats_Tests/tests/hello1.bats"; this is expected as we'd like to see 50% code coverage for 'sanity check' later on.

One naive questions:

1. what is the $PWD when executing kcov in the above context? How to check this? Does it look correct?

Note:
If I issue this command instead, the code coverage tool shows the hello1.bats file with 0% coverage. Including this for completeness; please note there is no call to "bats" executable; it is implicitly being
done inside hello1.bats.

kcov --bash-dont-parse-binary-dir --include-pattern=/user/src/xyz/Bats_Tests/,/user/src/xyz/Bats_Tests/tests,/usr/src/xyz/,/user/src/xyz/,/user/src/,/user/src/ /usr/src/xyz/coverage/ /usr/src/xyz/Bats_Tests/tests/hello1.bats

[mbland]

I think I might see what's happening here...it's an expectations thing.

kcov can't give you coverage on the .bats files, because Bats preprocesses them and executes the preprocessed output. (There is a hacky way to try to get coverage, I think...but I'd feel dirty for mentioning it.)

However, I've never been concerned with collecting coverage for .bats files, only for the files Bats is testing.

This would explain why you're only seeing coverage for the files under test, and why I thought you might be invoking kcov directly. It's working just as I'd expected, but not as you expected.

So it's true that if you want coverage for the test files themselves, Bats may not be the best choice. However, I'd suggest considering whether collecting coverage for .bats files provides any significant benefit. If your test cases are sufficiently complex, it might be worth extracting helper functions into separate files that you can source or load, which should allow you to collect coverage on those functions.

[alexRambo]

@mbland, I'm confused by your previous post...

1. "kcov can't give you coverage on the .bats files"
   I am not looking for .bats file coverage :/
2. "However, I've never been concerned with collecting coverage for .bats files, only for the files Bats is testing."
   I am expecting to get coverage only for the .sh files under test; namely "hello.sh" which is being called by hello1.bats file. Again, I am NOT expecting code coverage for hello1.bats; only "hello.sh" which is being called by hello1.bats file.
3. "This would explain why you're only seeing coverage for the files under test, and why I thought you might be invoking kcov directly"
   Problem: The "files under test" do NOT show up in code coverage. Only "bats" executable shows up 0% covered OR (in some other previous test runs; when invoked without passing bats to kcov) the "file under test" shows up with 0% code coverage.

For clarity, I want to get coverage on the CODE UNDER TEST file "hello.sh"; which is being called from UNIT TEST file hello1.bats.

Hope this clears things up... and you understand my frustration with getting such a simple file (hello.sh) to show coverage :)
-Alex

[mbland]

@alexRambo Sorry about that...but now I've finally reproduced what's happening locally and have a concrete recommendation.

First, though, I never answered your question about $PWD: I was using that to mean the current working directory, i.e. the directory from which you're running bats.

Second, I updated your test files to the following, where ~/src/tmp is the directory in which I created the test "project" (i.e. [[ "$PWD" == "$HOME/src/tmp" ]]):

---

~/src/tmp/hello.sh:

#!/bin/bash

if [[ "$1" == "world" ]]; then
  echo "Hello World"
else
  echo "Hello unknown"
fi

---

~/src/tmp/tests/hello1.bats:

#!/usr/bin/env bats

# Works when "$PWD" is equivalent to "$BATS_TEST_DIRNAME/.."
HELLO_CMD='./hello.sh'

# Works from any "$PWD"
#HELLO_CMD="$BATS_TEST_DIRNAME/../hello.sh"

# An alternative to the above that chomps off the last path component
#HELLO_CMD="${BATS_TEST_DIRNAME%/*}/hello.sh"

@test "hello without arguments" {
  run "$HELLO_CMD"
  [ "$output" == "Hello unknown" ]
}

@test "hello with world argument" {
  run "$HELLO_CMD" world
  [ "$output" == "Hello World" ]
}

---

I cloned bats-core v1.1.0 under ~/src/bats-core/bats-core and cloned and built SimonKagstrom/kcov v34 under ~/src/SimonKagstrom/kcov (v33 failed to build on Ubuntu 18.04). Then I invoked kcov and checked the coverage report thus:

# When `HELLO_CMD='./hello.sh'`, we must execute them in `~/src/tmp`.
$ cd ~/src/tmp

# Remove the coverage directory between runs, or else the data accumulates.
$ rm -rf ~/src/tmp/coverage

# Removed --bash-dont-parse-binary-dir as it's a v35 option.
# Note --include-path here, instead of --include-pattern.
$ ~/src/SimonKagstrom/kcov/src/kcov --include-path=~/src/tmp ~/src/tmp/coverage \
    ~/src/bats-core/bats-core/bin/bats ~/src/tmp/tests/

1..2
ok 1 hello without arguments
ok 2 hello with world argument

$ google-chrome "file://$HOME/src/tmp/coverage/index.html"

Happily, hello.sh is the only file in the report, and the coverage is 100%. All three versions of HELLO_CMD work.

Best I can tell, --include-path is the better option than --include-pattern here. You can use them in combination to limit what from --include-path is included, though.

Let me know whether or not this helps.

[alexRambo]

Update 07/12/2018: No coverage still :/
Grabbed a CentOS docker base image and set the DockerImage to build bats-core 1.1.0 and "kcov v34" from source as suggested. Created the same directory and 2 files in the same directory and ran the procedure.

Here's the '0%' coverage output from command "vi ~/src/tmp/coverage/index.json":
var data = {files:[
{"link":"bats/index.html","title":"bats","summary_name":"bats","covered_class":"lineNoCov","covered":"0.0","covered_lines":"0","uncovered_lines":"0","total_lines" : "0"},
], merged_files:[]};
var percent_low = 25;var percent_high = 75;
var header = { "command" : "bats", "date" : "2018-07-12 17:22:32", "instrumented" : 0, "covered" : 0,};

SUGGESTION:
I am releasing the Dockerfile which contains non-proprietary info with the only change being the substitution of "theUSER" for my real username; contains @mbland's steps to setup environment.
If time allows, can u follow the instructions in the Dockerfile to build the image and run the test we've been discussing? It shows 0% coverage; wondering if I messed something up or if you too get 0% coverage inside the container.
Note: Copy/Paste the commands under "HOW TO RUN THIS..."

Watch out:

1. The only thing u need is to create the 2 files being discussed above.
2. Had to rename the Dockerfile to Dockerfile.txt in order to be able to paste it in this post; change it back to no extension when u download it.

Dockerfile:
Dockerfile.txt

Hope someone can help still as kcov and Bats seems to "work" for everyone else :/

Thanks,
-@alexRambo

---

OLD:
This looks very promising. Trying it out...
Update #1: installed bats-core v1.1.0 as outlined above and moved the 2 files to match your setup... coverage does not show hello.sh file (it only shows bats exe) and it is still showing 0%.
The only difference is I am running kcov (and bats) inside a docker image (provided by kcov docker image) and haven't been able to successfully install kcov v34. It shouldn't matter.
I spent about 3 hours trying to get kcov to build with no success inside the docker file (but that's a separate issue and I will put it aside).

On the positive side, I learned more about the need of "$BATS_TEST_DIRNAME".
If I ever get this to work I will post my solution.

Thanks @mland.

[particleflux]

Since it was mentioned here - my example for kcov and bats on Circle CI has been updated to support kcov 36 and Circle CI 2: https://github.com/particleflux/kcov-bats-circleci-codeclimate/

There is some trickery in the script to correctly report the coverage with the Codeclimate reporter - may be this is relevant for Coveralls too:
https://github.com/particleflux/kcov-bats-circleci-codeclimate/blob/d93343674e4a10263a13aeff10248c74808324d2/.circleci/config.yml#L53-L59

One thing to note though: I am for whatever reason not able to get it to run inside docker - it needs to use the machineexecutor on Circle.

[fuero]

@particleflux @mbland
I've tried to reproduce this on 4 different environment, to no avail. I get the same results as @alexRambo.

Tested environments:
For all environments: bats 1.1.0, kcov 36

• Arch Linux: GNU bash, version 5.0.7(1)-release (x86_64-pc-linux-gnu)
• CentOS 7 VM: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
• CentOS 7 Container: --"--
• Ubuntu Container: GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)

Here's the example Ubuntu container:

FROM ubuntu:bionic

ENV KCOV_VERSION=36

RUN \
    export DEBIAN_FRONTEND=non-interactive && \
    apt update && \
    apt-get install -y software-properties-common && \
    apt-add-repository -y ppa:duggan/bats && \
    apt update && \
    apt-get install -y bats jq binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev wget unzip cmake make build-essential python git && \
    git clone https://github.com/SimonKagstrom/kcov /tmp/kcov && \
    cd /tmp/kcov && \
    git checkout v${KCOV_VERSION} && \
    cmake . && \
    make && \
    make install && \
    cd /tmp && \
    git clone https://github.com/particleflux/kcov-bats-circleci-codeclimate && \
    cd kcov-bats-circleci-codeclimate/ && \
    kcov --include-path . coverage bats tests/ && \
    cat coverage/*/coverage.json

Calling kcov like this kcov --clean --bash-dont-parse-binary-dir coverage /usr/bin/bats tests/ (on Arch) yields this:

{
  "files": [
    {"file": "/tmp/bats.6999.src", "percent_covered": "27.59", "covered_lines": "8", "total_lines": "29"},
    {"file": "/usr/bin/bats", "percent_covered": "86.21", "covered_lines": "25", "total_lines": "29"},
    {"file": "/usr/lib/bats-core/bats-preprocess", "percent_covered": "70.59", "covered_lines": "24", "total_lines": "34"},
    {"file": "/usr/lib/bats-core/bats-exec-test", "percent_covered": "65.47", "covered_lines": "146", "total_lines": "223"},
    {"file": "/usr/lib/bats-core/bats", "percent_covered": "43.42", "covered_lines": "33", "total_lines": "76"}
  ],
  "percent_covered": "60.36",
  "covered_lines": 236,
  "total_lines": 391,
  "percent_low": 25,
  "percent_high": 75,
  "command": "bats",
  "date": "2019-06-25 09:09:52"
}

So kcov never seems to find the "script under test", and it doesn't seem to matter which script I call, neither my own or @particleflux's, the output stays similar.

[ko1nksm]

It could be a problem with kcov. I wrote a comment here.

[Potherca]

Not sure if this is in any way relevant but is kcov a hard demand? Or are other options (like lcov.sh, bcov, bashcov, lcov, etc) also viable?

Part of my motivation for asking is my ongoing saga in search of libraries to use with bats-core.

After looking at asserts and mocks I am now looking in code coverage...

[ivopieniak]

Hi everyone. I am currently trying to run kcov v38 with BATS 1.3.0 with a following command

kcov --bash-dont-parse-binary-dir --include-pattern=.sh/ --exclude-pattern=lib/bats coverage \
lib/bats-core/bin/bats tests/monitor.pipeline.bats

What I get is:

kcov: error: Can't write helper
kcov: error: Can't start/attach to /Users/i.pieniak//bats-prototyping/lib/bats-core/bin/bats
kcov: error: Can't open directory /usr/local/bin/coverage/

If I decide to specify full paths then the kcov launches, however it measures the coverage of the bats scripts ( as described in this thread by @fuero. Maybe I am being naive, but has anybody came up with a solution for this problem?

I have been trying only on MacOS Big Sur 11.2.3 on zsh 5.8 (x86_64-apple-darwin20.0) and GNU bash, version 5.0.18(1)-release.

If you would guys require more details, I am happy to provide answers to any questions, but unfortunately I am bit novice to this topic.

[NetraPK]

@mbland, thanks for the quick feedback. Unfortunately, tried the following with same results... empty "bat" executable showing 0% covered.... kcov --bash-dont-parse-binary-dir --include-pattern=/user/src/xyz/Bats_Tests/,/user/src/xyz/Bats_Tests/tests,/usr/src/xyz/,/user/src/xyz/,/user/src/,/user/src/ /usr/src/xyz/coverage/ bats /usr/src/xyz/Bats_Tests/tests/

Folder structure: /usr/src/xyz/Bats_Tests/ => contains the "actual bash files" such as "hello.sh" being called by test files /usr/src/xyz/Bats_Tests/tests/ => contains bats test files (including hello1.bats"

Contents of test file "/usr/src/xyz/Bats_Tests/tests/hello1.bats": #!/usr/bin/env bats

@test "hello without arguments" { run ~/../hello.sh #result=$(bash $BATS_TEST_DIRNAME/../hello.sh) #[[ "$result" == "Hello unknown" ]] [ $status -eq "Hello unknown" ] }

@test "hello with world argument" { result=$(bash $BATS_TEST_DIRNAME/../hello.sh world) [[ "$result" == "Hello World" ]] }

Contents of test file "/usr/src/xyz/Bats_Tests/hello.sh": #!/bin/bash

if [[ "$1" == "world" ]]; then echo "Hello World" else echo "Hello unknown" fi

Again, running the hello1.bats file succeeds in running the unit test (UT). It shows 1 success and 1 failure for the "/usr/src/xyz/Bats_Tests/tests/hello1.bats"; this is expected as we'd like to see 50% code coverage for 'sanity check' later on.

One naive questions:

1. what is the $PWD when executing kcov in the above context? How to check this? Does it look correct?

Note: If I issue this command instead, the code coverage tool shows the hello1.bats file with 0% coverage. Including this for completeness; please note there is no call to "bats" executable; it is implicitly being done inside hello1.bats.

kcov --bash-dont-parse-binary-dir --include-pattern=/user/src/xyz/Bats_Tests/,/user/src/xyz/Bats_Tests/tests,/usr/src/xyz/,/user/src/xyz/,/user/src/,/user/src/ /usr/src/xyz/coverage/ /usr/src/xyz/Bats_Tests/tests/hello1.bats

@alexRambo Did you resolved this issue? As I am getting the same error!!
Can you help me?

[davehouser1]

I have this working (sort of)
I am running kcov v40, and bats 1.5.0.

Here is the command I run. Note I am sending output to an apache2 web directory to view output report

sudo kcov --clean --include-path=/home/jenkins /var/www/html/ bats ./myscript.bats

kcov does its job, and creates a report. I have not run into the issues others have been having.
The problem I am having is that I need bats to run as sudo to perform my tests. If I run the command as above, my tests will fail, however code coverage will be tested. I need to run the command like this:

sudo kcov --clean --include-path=/home/jenkins /var/www/html/ sudo bats ./myscript.bats

But, doing so, kcov only sees sudo and does not test coverage of any parameters past sudo. Is this a kcov issue or something I can do in bats / my syntax to fix?

EDIT

Disregard the problem I stated, I misread what kcov was doing for output. bats works with kcov properly. I just need to run my script without sudo. so the first command is what worked.

[brunolnetto]

The following approach worked for me:

1. Create shell script that makes the work done on desired folder e.g. src;
2. Create folder src/tests;
3. Create test .bats;
4. Run command below from folder src

kcov --report-only \
     --clean \
     --bash-dont-parse-binary-dir \
     --include-pattern=.sh \
     --exclude-pattern=tests \
     --include-path=. ./coverage/ bats ./tests

I am able to check the content in folder ./coverage. It has HTML and js files, although I expected a JSON file also.

[andregri]

@brunolnetto thank you for sharing your solution!

However, I had troubles running your command with --report-only option because kcov returned the following error:

kcov: error: report-only selected, but the target directory ./coverage//bats.ce650042 does not exist

Without --report-only option, kcov worked successfully.

I share with you the Dockerfile I used to run kcov.
I get the coverage results into /src/coverage/.
dgoss is the script I am testing, so I removed the option --include-pattern=.sh from kcov command:

FROM kcov/kcov

RUN apt update && \
    apt install -y git && \
    git clone https://github.com/bats-core/bats-core.git && \
    cd bats-core && \
    ./install.sh /usr/local

COPY dgoss /src/dgoss
COPY test/dgoss.bats /src/tests/dgoss.bats

WORKDIR /src

CMD kcov \
     --clean \
     --report-only \
     --bash-dont-parse-binary-dir \
     --exclude-pattern=tests \
     --include-path=. ./coverage/ bats ./tests

[brunolnetto]

Insightful! :-)