Heroku CI rails app test coverage

[a-b]

Please include Heroku CI specific manual, how to run test coverage for rails app.
Inspired by codeclimate/ruby-test-reporter#183

[toddmazierski]

Thanks, @a-b. We're still working on getting the required environment variables from Heroku CI as mentioned in this comment. Please stand by!

[brynary]

Update: Heroku now supports a HEROKU_TEST_RUN_COMMIT_VERSION env variable. I think we can get this working with that.

(codeclimate/ruby-test-reporter#183 (comment))

[efueger]

Cross-posting this comment:

@dblandin mentioned: Hey @joshwlewis,

Thank you for the update! Out of those three, HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP would definitely be the most important as it's a required value, either from the git repo itself (which is unavailable during Heroku CI runs) or the environment.

If we had that value available, we could address and close this issue easily. Thanks!

[dblandin]

Thanks @efueger!

@brynary I think we're still blocked until we have the git committed at value available.

[dblandin]

Hey @a-b,

We've discussed this internally and while we're waiting on Heroku to support this environment variable via Heroku CI, you can use the following (not entirely recommended) workaround:

$ cc-test-reporter before-build
$ GIT_COMMITTED_AT="$(date +%s)" cc-test-reporter after-build

This fakes the git commit "committed at" data point with the current time.

Note that this might produce undesirable behavior when rendering test coverage information on codeclimate.com or annotations on github.com (via the Code Climate browser extension) where we depend upon accurate committed at data, but that should be fairly uncommon.

[dja]

Is there a recommended way to install/setup cc-test-reporter for Heroku, and give it all the env variables we can until they expose a better GIT_COMMITTED_AT variable?

[dblandin]

Hey @dja, you can use the workaround posted in #226 (comment). Just be aware that there might be some weirdness rendering data if the generated data and commited_at date are significantly different.

[dja]

I have this almost fully working, except getting this issue when passing the workaround GIT_COMMITTED_AT="$(date +%s)": Error: strconv.Atoi: parsing "$(date +%s)": invalid syntax

The date as formatted is 1511140351

I'm setting variables as so:

GIT_BRANCH=HEROKU_TEST_RUN_BRANCH
GIT_COMMIT_SHA=HEROKU_TEST_RUN_COMMIT_VERSION
CI_NAME='HEROKU'
CI_BUILD_ID=HEROKU_TEST_RUN_ID
GIT_COMMITTED_AT='$(date +%s)'

[dblandin]

Hey @dja,

It looks like your subshell value isn't excuting correctly. I think that's because you're wrapping the subshell command in single quotes which doesn't trigger interpolation.

I'm also wondering if you're GIT_BRANCH and GIT_COMMIT_SHA values are correct as they're missing the leading $.

Could you try the following?

export CI_BUILD_ID="$HEROKU_TEST_RUN_ID"
export CI_NAME="heroku"
export GIT_BRANCH="$HEROKU_TEST_RUN_BRANCH"
export GIT_COMMIT_SHA="$HEROKU_TEST_RUN_COMMIT_VERSION"
export GIT_COMMITTED_AT="$(date +%s)"

[dja]

That helped, but getting this error:

== Run Code Climate after-build ==
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Error: exit status 128

[TaylorBriggs]

As far as I can tell, this setup is working on Heroku CI. We have an rspec test suite with simplecov and a mocha/chai/sinon/enzyme test suite run by karma and reporting via istanbul.

app.json:

{
  "environments": {
    "test": {
      "addons": [
        "heroku-postgresql"
      ],
      "buildpacks": [
        { "url": "heroku/nodejs" },
        { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" },
        { "url": "heroku/ruby" }
      ],
      "env": {
        "NODE_ENV": "test",
        "RAILS_ENV": "test"
      },
      "scripts": {
        "test-setup": "./script/ci_setup",
        "test": "./script/ci"
      }
    }
  }
}

script/ci_setup:

#!/bin/bash

# download the codeclimate test reporter
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
chmod +x ./cc-test-reporter

./cc-test-reporter before-build

script/ci:

#!/bin/bash

export CI_BUILD_ID="$HEROKU_TEST_RUN_ID"
export CI_NAME="heroku"
export GIT_BRANCH="$HEROKU_TEST_RUN_BRANCH"
export GIT_COMMIT_SHA="$HEROKU_TEST_RUN_COMMIT_VERSION"
export GIT_COMMITTED_AT="$(date +%s)"

ruby_filename="coverage/codeclimate.rb.json"
js_filename="coverage/codeclimate.js.json"

# run the ruby test suite
bundle exec rake spec
# format ruby coverage
./cc-test-reporter format-coverage --output $ruby_filename

# run the JS test suite
yarn test
# format js coverage
./cc-test-reporter format-coverage --input-type lcov --output $js_filename coverage/karma/lcov.info

# sum code coverage and send to codeclimate
./cc-test-reporter sum-coverage $ruby_filename $js_filename
./cc-test-reporter upload-coverage

P.S. 👋 @dblandin

[dblandin]

Hey @TaylorBriggs!! 👋

I just merged and released #305 which brings in support for the environment variables currently available during Heroku CI builds.

Not all the way there yet, but at least you can remove a few of those export lines now 😄

@joshwlewis Any update on support for the CI_NAME and/or GIT_COMMITTED_AT environment variables?

[TaylorBriggs]

Thanks @dblandin!

[niclas-ahden]

@joshwlewis @appleton We just switched to HerokuCI and noticed the incomplete support for Code Climate. Any update on the CI_NAME and GIT_COMMITTED_AT environment variables?

Details in this issue and:
codeclimate/ruby-test-reporter#183 (comment)
#305

Thanks!

[sergiopantoja]

@TaylorBriggs's solution worked perfectly for code coverage, but we found that Heroku CI wouldn't report failing tests with that. Not sure if it's our RSpec config, but just want to give a heads up so others can make sure their test failures are being reported properly.

Will send another update if we figure it out.

[TaylorBriggs]

@sergiopantoja I ran into the issue with Heroku CI not reporting the test failures. I updated my script as follows to catch the error code and exit the script:

#!/bin/bash

exit_if_error() {
  if [ $1 -ne 0 ]; then
    exit 1
  fi
}

export CI_NAME="heroku"
export GIT_COMMITTED_AT="$(date +%s)"

ruby_filename="coverage/codeclimate.rb.json"
js_filename="coverage/codeclimate.js.json"

# lint the ruby code
bin/rubocop
exit_if_error $?

# ruby test suite
bin/rake spec
exit_if_error $?

./cc-test-reporter format-coverage --output $ruby_filename

# JS linting and test suite
yarn test
exit_if_error $?

./cc-test-reporter format-coverage --input-type lcov --output $js_filename coverage/karma/lcov.info

# sum code coverage and send to codeclimate
./cc-test-reporter sum-coverage $ruby_filename $js_filename
./cc-test-reporter upload-coverage

[TaylorBriggs]

I haven't been working on that project since March, but it was working as of then. Maybe Heroku CI has changed since so you may need to adjust accordingly.

[wfleming]

I'd suggest considering adding set -e to the top of the bash script as an alternative to exit_if_error. set -e will exit the script if any command exits non-zero. AFAICT the only important commands you're not explicitly checking are the test-reporter invocations. Personally I think I'd want my CI to error if those were erroring to catch potential misconfigurations, but if you're alright with those commands erroring you could use set +x to put the shell back in a mode where erroring commands don't abort the script, which would still be a bit less code to write.

e.g.

#!/bin/bash
set -e

# do stuff that must exit 0

set +x

# do stuff that is allowed to exit non-zero without failing CI

[sergiopantoja]

@TaylorBriggs @wfleming Man you guys are great! Thanks so much for the quick response and help. 🙏

[rosscooperman]

@sergiopantoja you can also pass an exit status to the reporter like cc-test-reporter --exit-code $EXIT_CODE. So, something like:

./run-tests.sh
RETURN_VALUE=$?
 ./cc-test-reporter after-build --exit-code $RETURN_VALUE

or what we do (since ^^ seems unecessary)

#!/bin/bash

export CI_NAME="heroku"
export GIT_COMMITTED_AT="$(date +%s)"

bin/rspec
RETURN_VALUE=$?
./cc-test-reporter after-build
exit $RETURN_VALUE

The downside of set -e is that you won't get any coverage measured if there are test failures which may not be what you want (though some test runners don't run coverage when there are failures anyway).

[fedegl]

Any updates on how to setup up CodeClimate reporting on Heroku CI? I am having some of the same issues.