Add Testpack API implementation to support Heroku CI #92

joshwlewis · 2016-10-21T21:32:08Z

Heroku CI is supporting a new Testpack API, which will allow buildpack users to have their tests compiled and ran with minimal configuration (hopefully none).

This PR implements that API so that

Test runs only install dependencies needed for test
Test runs compile the app for testing
finally mix test is run

You can see a passing test run on this example app.

Heroku CI is in private beta right now, but I can certainly flag buildpack maintainers in.

ericmj · 2016-10-22T00:01:07Z

lib/misc_funcs.sh

@@ -61,7 +61,7 @@ function export_mix_env() {
    if [ -d $env_path ] && [ -f $env_path/MIX_ENV ]; then
      export MIX_ENV=$(cat $env_path/MIX_ENV)
    else
-      export MIX_ENV=prod
+      export MIX_ENV=${1:-prod}


Sorry, I am not very familiar with bash syntax. How does this ensure we build in the test environment?

This just means use the 1st argument provided to the function, or if there wasn't a first argument, use 'prod'.

That leaves the original usage (export_mix_env) unchanged, but you can now provide a fallback other than prod with export_mix_env 'test'.

What's nice here is that if a user did specify something like MIX_ENV=dev for a test run (albeit that would be weird), this wouldn't overwrite it.

ericmj · 2016-10-22T00:04:41Z

This looks very interesting, I am excited to try this feature out. I would appreciate it if you can flip the beta flag for me.

Should I create an heroku app from your example app? Are there any instructions on how to enable and run the CI?

Do you have any example apps that are using the TAP output? I would be interested in seeing how it compares to mix test's default output.

joshwlewis · 2016-10-22T02:29:06Z

Cool, I just enabled ci for your publicly displayed email. You should get an email soon with directions. I'll collaborate you onto the example app too so you can check it out.

As far as TAP output goes, I haven't seen an Elixir project using it yet. Here's some TAP output from one of my Ruby projects:

-----> Running test command `bundle exec rake`...
1..4
ok 1 - test_we_can_find_a_sausage
ok 2 - test_we_have_sausages
not ok 3 - test_a_is_a
  # FAIL test/models/sausage_test.rb:20
  # Expected: "wubalub"
  #   Actual: "a"
ok 4 - test_there_are_a_lot_of_sausages
rake aborted!
Command failed with status (255): [ruby -w -I"lib:test" -W0 -I"/app/vendor/bundle/ruby/2.2.0/gems/rake-11.1.2/lib" "/app/vendor/bundle/ruby/2.2.0/gems/rake-11.1.2/lib/rake/rake_test_loader.rb" "test/**/*_test.rb" | grep -e '^[^/]' | tapout tap]
Tasks: TOP => default => test
(See full trace by running task with --trace)

It looks like ex_unit has support for custom formatters, which may make supporting TAP feasible.

HashNuke · 2016-10-22T08:39:51Z

@joshwlewis This is very cool. Can I try it out too?

ericmj

Looks good to me.

HashNuke · 2016-10-23T12:41:08Z

@ericmj Going y the TestPack API doc, TAP output isn't mandatory. So we can merge this right away unless you have something in mind.

We can suggest a list of TAP formatters on the readme. @Stratus3D has a TAP implementation for ExUnit https://github.com/Stratus3D/spout.

HashNuke

LGTM ~! 🖖

ericmj · 2016-10-23T12:42:39Z

Going y the TestPack API doc, TAP output isn't mandatory. So we can merge this right away unless you have something in mind.

Yeah, I just wanted to see an example of how Heroku diplays the TAP output so I know if it's something worth working on for ExUnit :)

ericmj · 2016-10-23T12:43:00Z

Thank you @joshwlewis!

joshwlewis · 2016-10-23T14:11:58Z

Thanks, y'all.

@HashNuke, I've flipped the bit for you too, expect an email soon. I also collaborated you on the example app, so you can check it out.

joshwlewis · 2016-10-23T14:54:31Z

@ericmj, @HashNuke -- Regarding TAP formatters, I wasn't able to get Stratus3D/spout working. Additionally, it seems to be dumping the output to a file at the end, rather than using STDOUT streaming (streaming output is one of the strongest features of TAP over say JUnit).

I've got my own formatter in progress at joshwlewis/tapex. Though, maybe it's something that would fit in ExUnit itself?

ericmj · 2016-10-23T16:48:55Z

@joshwlewis That's great. I will talk with the rest of the core team about what they think of additional formatters in ExUnit. It should be straight-forward to use as a dependency also.

Stratus3D · 2016-10-27T01:53:54Z

@ericmj @joshwlewis I got hung up on the best way to test the formatter and never completed spout. I've got a whole bunch of uncommitted changes locally that I'll sort though tomorrow.

I started the project because we were going to need TAP formatters for Erlang and Elixir for something at work, but things changed and we didn't need them so I haven't had any motivation to finish it.

cbarratt · 2016-12-01T11:00:07Z

@joshwlewis - I've recently been added to the beta and I'm attempting to get my tests running using this awesome new feature!

-----> The Run buildpack does not run tests. Skipping.
-----> Running Elixir buildpack tests...
==> maggie
** (Mix) Cannot run tests because test helper file "test/test_helper.exs" does not exist
-----> Elixir buildpack tests failed with exit status 1

I believe I am receiving this error due to my application being an Umbrella app, I can't seem to figure out or diagnose why this is.

If remove deps and _build locally and follow the same steps the CI takes:

MIX_ENV=test mix deps.get
MIX_ENV=test mix test

it works fine :(

ericmj · 2016-12-01T11:03:13Z

@cbarratt Can you share your project or create a project that reproduces?

cbarratt · 2016-12-01T11:07:48Z

@ericmj - Apologies for stirring this conversation up. I've managed to fix it!

Turns out that I have the test folder in my .slugignore :( Which made sense pre Heroku CI!

ericmj · 2016-12-01T11:10:24Z

No worries, thanks for updating us.

joshwlewis · 2016-12-01T14:40:32Z

I'm glad you got it fixed @cbarratt. If you have any additional problems or feedback, you can ask the Heroku CI team here: heroku-ci-feedback@heroku.com.

jah2488 · 2017-06-01T17:07:31Z

Has anyone been using this buildpack with Heroku CI recently? I was able to get everything setup in a fresh elixir/phoenix project today, but I can't seem to get the tests to run under Heroku CI.

Tests run fine locally
App runs fine in production on Heroku
Tests run on Circle CI

-----> Running Elixir buildpack tests...
16:53:10.967 [error] GenServer #PID<0.192.0> terminating
** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused - :econnrefused
    (db_connection) lib/db_connection/connection.ex:148: DBConnection.Connection.connect/2
    (connection) lib/connection.ex:622: Connection.enter_connect/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: nil

The error above occurs when the tests go to run, after setup, but when looking into the test setup, there are various errors along the way such as:

cp: cannot copy a directory, '/tmp/cache430691226/erlang', into itself, '/app/.platform_tools/erlang/erlang'
-----> Installing rebar 16:52:57.921 [error] GenServer #PID<0.118.0> terminating ** (stop) normal (... continues for 100 lines)
cp: cannot copy a directory, '/app/.hex/', into itself, '/app/.hex/.hex'

When attempting to debug with heroku ci:debug the same error occurs during setup and when I try to run mix test manually I get /app/.platform_tools/elixir/bin/elixir: 126: exec: erl: not found, so perhaps the cp error with erlang above is causing the whole issue?

Any insight? I'm in the middle of writing an email to the heroku-ci team, since @joshwlewis was gracious enough to add that here, but thought I'd post first.

cblavier · 2017-06-26T12:24:59Z

hey @jah2488
did you solve your problem? I'm facing the same problem right now

jah2488 · 2017-06-26T14:23:08Z

@cblavier Yep, I needed to change my test.exs to more or less mirror my prod.exs when running under CI. I wrote about it here. https://blog.lunarcollective.co/running-our-tests-on-heroku-ci-with-elixir-and-phoenix-fe01f63a9ebb

cblavier · 2017-06-26T14:31:02Z

Ok, thanks, didn't know it was your post! I had already read it, but unfortunately it's not helping :/
See my question here :
https://stackoverflow.com/questions/44760975/cant-make-my-phoenix-app-build-on-heroku-ci

Did you manage to get heroku ci:debug working?

jah2488 · 2017-06-26T16:18:23Z

@cblavier From the error you posted on SO, it looks like your test DB is trying to connect directly to your prod DB. I got that error once, because I had copied over my database_url from production to the testing pipeline, which was the wrong approach. The only env vars I have setup for CI are pool_size and ci, which are set to 2 and true respectively.

It may be worth checking out the example app that heroku has up. https://github.com/joshwlewis/giphy-mcgiface/blob/master/config/test.exs

* Add scripts required for Heroku CI * Don't overwrite existing MIX_ENV * bin/test and bin/test-compile should be executable * export_mix_env should fallback without an argument * No need to specify MIX_ENV, it's already set

joshwlewis added 5 commits October 21, 2016 15:38

Add scripts required for Heroku CI

de4a181

Don't overwrite existing MIX_ENV

4eb77b5

bin/test and bin/test-compile should be executable

8e97655

export_mix_env should fallback without an argument

32bf492

No need to specify MIX_ENV, it's already set

5496dd9

ericmj reviewed Oct 22, 2016

View reviewed changes

ericmj approved these changes Oct 23, 2016

View reviewed changes

HashNuke approved these changes Oct 23, 2016

View reviewed changes

ericmj merged commit 307897d into HashNuke:master Oct 23, 2016

dentarg mentioned this pull request Dec 14, 2019

Buildpack fails on heroku-16/18 stack Starkast/heroku-buildpack-cmake#17

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Testpack API implementation to support Heroku CI #92

Add Testpack API implementation to support Heroku CI #92

joshwlewis commented Oct 21, 2016

ericmj Oct 22, 2016

joshwlewis Oct 22, 2016

ericmj commented Oct 22, 2016

joshwlewis commented Oct 22, 2016

HashNuke commented Oct 22, 2016

ericmj left a comment

HashNuke commented Oct 23, 2016

HashNuke left a comment

ericmj commented Oct 23, 2016

ericmj commented Oct 23, 2016

joshwlewis commented Oct 23, 2016

joshwlewis commented Oct 23, 2016 •

edited

Loading

ericmj commented Oct 23, 2016

Stratus3D commented Oct 27, 2016

cbarratt commented Dec 1, 2016

ericmj commented Dec 1, 2016

cbarratt commented Dec 1, 2016

ericmj commented Dec 1, 2016

joshwlewis commented Dec 1, 2016

jah2488 commented Jun 1, 2017 •

edited

Loading

cblavier commented Jun 26, 2017

jah2488 commented Jun 26, 2017

cblavier commented Jun 26, 2017

jah2488 commented Jun 26, 2017

Add Testpack API implementation to support Heroku CI #92

Add Testpack API implementation to support Heroku CI #92

Conversation

joshwlewis commented Oct 21, 2016

ericmj Oct 22, 2016

Choose a reason for hiding this comment

joshwlewis Oct 22, 2016

Choose a reason for hiding this comment

ericmj commented Oct 22, 2016

joshwlewis commented Oct 22, 2016

HashNuke commented Oct 22, 2016

ericmj left a comment

Choose a reason for hiding this comment

HashNuke commented Oct 23, 2016

HashNuke left a comment

Choose a reason for hiding this comment

ericmj commented Oct 23, 2016

ericmj commented Oct 23, 2016

joshwlewis commented Oct 23, 2016

joshwlewis commented Oct 23, 2016 • edited Loading

ericmj commented Oct 23, 2016

Stratus3D commented Oct 27, 2016

cbarratt commented Dec 1, 2016

ericmj commented Dec 1, 2016

cbarratt commented Dec 1, 2016

ericmj commented Dec 1, 2016

joshwlewis commented Dec 1, 2016

jah2488 commented Jun 1, 2017 • edited Loading

cblavier commented Jun 26, 2017

jah2488 commented Jun 26, 2017

cblavier commented Jun 26, 2017

jah2488 commented Jun 26, 2017

joshwlewis commented Oct 23, 2016 •

edited

Loading

jah2488 commented Jun 1, 2017 •

edited

Loading