New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Testpack API implementation to support Heroku CI #92

Merged
merged 5 commits into from Oct 23, 2016

Conversation

Projects
None yet
7 participants
@joshwlewis
Contributor

joshwlewis commented Oct 21, 2016

Heroku CI is supporting a new Testpack API, which will allow buildpack users to have their tests compiled and ran with minimal configuration (hopefully none).

This PR implements that API so that

  • Test runs only install dependencies needed for test
  • Test runs compile the app for testing
  • finally mix test is run

You can see a passing test run on this example app.

Heroku CI is in private beta right now, but I can certainly flag buildpack maintainers in.

joshwlewis added some commits Oct 21, 2016

@@ -61,7 +61,7 @@ function export_mix_env() {
if [ -d $env_path ] && [ -f $env_path/MIX_ENV ]; then
export MIX_ENV=$(cat $env_path/MIX_ENV)
else
export MIX_ENV=prod
export MIX_ENV=${1:-prod}

This comment has been minimized.

@ericmj

ericmj Oct 22, 2016

Collaborator

Sorry, I am not very familiar with bash syntax. How does this ensure we build in the test environment?

@ericmj

ericmj Oct 22, 2016

Collaborator

Sorry, I am not very familiar with bash syntax. How does this ensure we build in the test environment?

This comment has been minimized.

@joshwlewis

joshwlewis Oct 22, 2016

Contributor

This just means use the 1st argument provided to the function, or if there wasn't a first argument, use 'prod'.

That leaves the original usage (export_mix_env) unchanged, but you can now provide a fallback other than prod with export_mix_env 'test'.

What's nice here is that if a user did specify something like MIX_ENV=dev for a test run (albeit that would be weird), this wouldn't overwrite it.

@joshwlewis

joshwlewis Oct 22, 2016

Contributor

This just means use the 1st argument provided to the function, or if there wasn't a first argument, use 'prod'.

That leaves the original usage (export_mix_env) unchanged, but you can now provide a fallback other than prod with export_mix_env 'test'.

What's nice here is that if a user did specify something like MIX_ENV=dev for a test run (albeit that would be weird), this wouldn't overwrite it.

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Oct 22, 2016

Collaborator

This looks very interesting, I am excited to try this feature out. I would appreciate it if you can flip the beta flag for me.

Should I create an heroku app from your example app? Are there any instructions on how to enable and run the CI?

Do you have any example apps that are using the TAP output? I would be interested in seeing how it compares to mix test's default output.

Collaborator

ericmj commented Oct 22, 2016

This looks very interesting, I am excited to try this feature out. I would appreciate it if you can flip the beta flag for me.

Should I create an heroku app from your example app? Are there any instructions on how to enable and run the CI?

Do you have any example apps that are using the TAP output? I would be interested in seeing how it compares to mix test's default output.

@joshwlewis

This comment has been minimized.

Show comment
Hide comment
@joshwlewis

joshwlewis Oct 22, 2016

Contributor

Cool, I just enabled ci for your publicly displayed email. You should get an email soon with directions. I'll collaborate you onto the example app too so you can check it out.

As far as TAP output goes, I haven't seen an Elixir project using it yet. Here's some TAP output from one of my Ruby projects:

-----> Running test command `bundle exec rake`...
1..4
ok 1 - test_we_can_find_a_sausage
ok 2 - test_we_have_sausages
not ok 3 - test_a_is_a
  # FAIL test/models/sausage_test.rb:20
  # Expected: "wubalub"
  #   Actual: "a"
ok 4 - test_there_are_a_lot_of_sausages
rake aborted!
Command failed with status (255): [ruby -w -I"lib:test" -W0 -I"/app/vendor/bundle/ruby/2.2.0/gems/rake-11.1.2/lib" "/app/vendor/bundle/ruby/2.2.0/gems/rake-11.1.2/lib/rake/rake_test_loader.rb" "test/**/*_test.rb" | grep -e '^[^/]' | tapout tap]
Tasks: TOP => default => test
(See full trace by running task with --trace)

It looks like ex_unit has support for custom formatters, which may make supporting TAP feasible.

Contributor

joshwlewis commented Oct 22, 2016

Cool, I just enabled ci for your publicly displayed email. You should get an email soon with directions. I'll collaborate you onto the example app too so you can check it out.

As far as TAP output goes, I haven't seen an Elixir project using it yet. Here's some TAP output from one of my Ruby projects:

-----> Running test command `bundle exec rake`...
1..4
ok 1 - test_we_can_find_a_sausage
ok 2 - test_we_have_sausages
not ok 3 - test_a_is_a
  # FAIL test/models/sausage_test.rb:20
  # Expected: "wubalub"
  #   Actual: "a"
ok 4 - test_there_are_a_lot_of_sausages
rake aborted!
Command failed with status (255): [ruby -w -I"lib:test" -W0 -I"/app/vendor/bundle/ruby/2.2.0/gems/rake-11.1.2/lib" "/app/vendor/bundle/ruby/2.2.0/gems/rake-11.1.2/lib/rake/rake_test_loader.rb" "test/**/*_test.rb" | grep -e '^[^/]' | tapout tap]
Tasks: TOP => default => test
(See full trace by running task with --trace)

It looks like ex_unit has support for custom formatters, which may make supporting TAP feasible.

@HashNuke

This comment has been minimized.

Show comment
Hide comment
@HashNuke

HashNuke Oct 22, 2016

Owner

@joshwlewis This is very cool. Can I try it out too?

Owner

HashNuke commented Oct 22, 2016

@joshwlewis This is very cool. Can I try it out too?

@ericmj

ericmj approved these changes Oct 23, 2016

Looks good to me.

@HashNuke

This comment has been minimized.

Show comment
Hide comment
@HashNuke

HashNuke Oct 23, 2016

Owner

@ericmj Going y the TestPack API doc, TAP output isn't mandatory. So we can merge this right away unless you have something in mind.

We can suggest a list of TAP formatters on the readme. @Stratus3D has a TAP implementation for ExUnit https://github.com/Stratus3D/spout.

Owner

HashNuke commented Oct 23, 2016

@ericmj Going y the TestPack API doc, TAP output isn't mandatory. So we can merge this right away unless you have something in mind.

We can suggest a list of TAP formatters on the readme. @Stratus3D has a TAP implementation for ExUnit https://github.com/Stratus3D/spout.

@HashNuke

LGTM ~! 🖖

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Oct 23, 2016

Collaborator

Going y the TestPack API doc, TAP output isn't mandatory. So we can merge this right away unless you have something in mind.

Yeah, I just wanted to see an example of how Heroku diplays the TAP output so I know if it's something worth working on for ExUnit :)

Collaborator

ericmj commented Oct 23, 2016

Going y the TestPack API doc, TAP output isn't mandatory. So we can merge this right away unless you have something in mind.

Yeah, I just wanted to see an example of how Heroku diplays the TAP output so I know if it's something worth working on for ExUnit :)

@ericmj ericmj merged commit 307897d into HashNuke:master Oct 23, 2016

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Oct 23, 2016

Collaborator

Thank you @joshwlewis!

Collaborator

ericmj commented Oct 23, 2016

Thank you @joshwlewis!

@joshwlewis

This comment has been minimized.

Show comment
Hide comment
@joshwlewis

joshwlewis Oct 23, 2016

Contributor

Thanks, y'all.

@HashNuke, I've flipped the bit for you too, expect an email soon. I also collaborated you on the example app, so you can check it out.

Contributor

joshwlewis commented Oct 23, 2016

Thanks, y'all.

@HashNuke, I've flipped the bit for you too, expect an email soon. I also collaborated you on the example app, so you can check it out.

@joshwlewis

This comment has been minimized.

Show comment
Hide comment
@joshwlewis

joshwlewis Oct 23, 2016

Contributor

@ericmj, @HashNuke -- Regarding TAP formatters, I wasn't able to get Stratus3D/spout working. Additionally, it seems to be dumping the output to a file at the end, rather than using STDOUT streaming (streaming output is one of the strongest features of TAP over say JUnit).

I've got my own formatter in progress at joshwlewis/tapex. Though, maybe it's something that would fit in ExUnit itself?

Contributor

joshwlewis commented Oct 23, 2016

@ericmj, @HashNuke -- Regarding TAP formatters, I wasn't able to get Stratus3D/spout working. Additionally, it seems to be dumping the output to a file at the end, rather than using STDOUT streaming (streaming output is one of the strongest features of TAP over say JUnit).

I've got my own formatter in progress at joshwlewis/tapex. Though, maybe it's something that would fit in ExUnit itself?

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Oct 23, 2016

Collaborator

@joshwlewis That's great. I will talk with the rest of the core team about what they think of additional formatters in ExUnit. It should be straight-forward to use as a dependency also.

Collaborator

ericmj commented Oct 23, 2016

@joshwlewis That's great. I will talk with the rest of the core team about what they think of additional formatters in ExUnit. It should be straight-forward to use as a dependency also.

@Stratus3D

This comment has been minimized.

Show comment
Hide comment
@Stratus3D

Stratus3D Oct 27, 2016

@ericmj @joshwlewis I got hung up on the best way to test the formatter and never completed spout. I've got a whole bunch of uncommitted changes locally that I'll sort though tomorrow.

I started the project because we were going to need TAP formatters for Erlang and Elixir for something at work, but things changed and we didn't need them so I haven't had any motivation to finish it.

Stratus3D commented Oct 27, 2016

@ericmj @joshwlewis I got hung up on the best way to test the formatter and never completed spout. I've got a whole bunch of uncommitted changes locally that I'll sort though tomorrow.

I started the project because we were going to need TAP formatters for Erlang and Elixir for something at work, but things changed and we didn't need them so I haven't had any motivation to finish it.

@cbarratt

This comment has been minimized.

Show comment
Hide comment
@cbarratt

cbarratt Dec 1, 2016

@joshwlewis - I've recently been added to the beta and I'm attempting to get my tests running using this awesome new feature!

-----> The Run buildpack does not run tests. Skipping.
-----> Running Elixir buildpack tests...
==> maggie
** (Mix) Cannot run tests because test helper file "test/test_helper.exs" does not exist
-----> Elixir buildpack tests failed with exit status 1

I believe I am receiving this error due to my application being an Umbrella app, I can't seem to figure out or diagnose why this is.

If remove deps and _build locally and follow the same steps the CI takes:

MIX_ENV=test mix deps.get
MIX_ENV=test mix test

it works fine :(

cbarratt commented Dec 1, 2016

@joshwlewis - I've recently been added to the beta and I'm attempting to get my tests running using this awesome new feature!

-----> The Run buildpack does not run tests. Skipping.
-----> Running Elixir buildpack tests...
==> maggie
** (Mix) Cannot run tests because test helper file "test/test_helper.exs" does not exist
-----> Elixir buildpack tests failed with exit status 1

I believe I am receiving this error due to my application being an Umbrella app, I can't seem to figure out or diagnose why this is.

If remove deps and _build locally and follow the same steps the CI takes:

MIX_ENV=test mix deps.get
MIX_ENV=test mix test

it works fine :(

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Dec 1, 2016

Collaborator

@cbarratt Can you share your project or create a project that reproduces?

Collaborator

ericmj commented Dec 1, 2016

@cbarratt Can you share your project or create a project that reproduces?

@cbarratt

This comment has been minimized.

Show comment
Hide comment
@cbarratt

cbarratt Dec 1, 2016

@ericmj - Apologies for stirring this conversation up. I've managed to fix it!

Turns out that I have the test folder in my .slugignore :( Which made sense pre Heroku CI!

cbarratt commented Dec 1, 2016

@ericmj - Apologies for stirring this conversation up. I've managed to fix it!

Turns out that I have the test folder in my .slugignore :( Which made sense pre Heroku CI!

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Dec 1, 2016

Collaborator

No worries, thanks for updating us.

Collaborator

ericmj commented Dec 1, 2016

No worries, thanks for updating us.

@joshwlewis

This comment has been minimized.

Show comment
Hide comment
@joshwlewis

joshwlewis Dec 1, 2016

Contributor

I'm glad you got it fixed @cbarratt. If you have any additional problems or feedback, you can ask the Heroku CI team here: heroku-ci-feedback@heroku.com.

Contributor

joshwlewis commented Dec 1, 2016

I'm glad you got it fixed @cbarratt. If you have any additional problems or feedback, you can ask the Heroku CI team here: heroku-ci-feedback@heroku.com.

@jah2488

This comment has been minimized.

Show comment
Hide comment
@jah2488

jah2488 Jun 1, 2017

Has anyone been using this buildpack with Heroku CI recently? I was able to get everything setup in a fresh elixir/phoenix project today, but I can't seem to get the tests to run under Heroku CI.

  • Tests run fine locally
  • App runs fine in production on Heroku
  • Tests run on Circle CI
-----> Running Elixir buildpack tests...
16:53:10.967 [error] GenServer #PID<0.192.0> terminating
** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused - :econnrefused
    (db_connection) lib/db_connection/connection.ex:148: DBConnection.Connection.connect/2
    (connection) lib/connection.ex:622: Connection.enter_connect/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: nil

The error above occurs when the tests go to run, after setup, but when looking into the test setup, there are various errors along the way such as:

  • cp: cannot copy a directory, '/tmp/cache430691226/erlang', into itself, '/app/.platform_tools/erlang/erlang'
  • -----> Installing rebar 16:52:57.921 [error] GenServer #PID<0.118.0> terminating ** (stop) normal (... continues for 100 lines)
  • cp: cannot copy a directory, '/app/.hex/', into itself, '/app/.hex/.hex'

When attempting to debug with heroku ci:debug the same error occurs during setup and when I try to run mix test manually I get /app/.platform_tools/elixir/bin/elixir: 126: exec: erl: not found, so perhaps the cp error with erlang above is causing the whole issue?

Any insight? I'm in the middle of writing an email to the heroku-ci team, since @joshwlewis was gracious enough to add that here, but thought I'd post first.

jah2488 commented Jun 1, 2017

Has anyone been using this buildpack with Heroku CI recently? I was able to get everything setup in a fresh elixir/phoenix project today, but I can't seem to get the tests to run under Heroku CI.

  • Tests run fine locally
  • App runs fine in production on Heroku
  • Tests run on Circle CI
-----> Running Elixir buildpack tests...
16:53:10.967 [error] GenServer #PID<0.192.0> terminating
** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused - :econnrefused
    (db_connection) lib/db_connection/connection.ex:148: DBConnection.Connection.connect/2
    (connection) lib/connection.ex:622: Connection.enter_connect/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: nil

The error above occurs when the tests go to run, after setup, but when looking into the test setup, there are various errors along the way such as:

  • cp: cannot copy a directory, '/tmp/cache430691226/erlang', into itself, '/app/.platform_tools/erlang/erlang'
  • -----> Installing rebar 16:52:57.921 [error] GenServer #PID<0.118.0> terminating ** (stop) normal (... continues for 100 lines)
  • cp: cannot copy a directory, '/app/.hex/', into itself, '/app/.hex/.hex'

When attempting to debug with heroku ci:debug the same error occurs during setup and when I try to run mix test manually I get /app/.platform_tools/elixir/bin/elixir: 126: exec: erl: not found, so perhaps the cp error with erlang above is causing the whole issue?

Any insight? I'm in the middle of writing an email to the heroku-ci team, since @joshwlewis was gracious enough to add that here, but thought I'd post first.

@cblavier

This comment has been minimized.

Show comment
Hide comment
@cblavier

cblavier Jun 26, 2017

hey @jah2488
did you solve your problem? I'm facing the same problem right now

cblavier commented Jun 26, 2017

hey @jah2488
did you solve your problem? I'm facing the same problem right now

@jah2488

This comment has been minimized.

Show comment
Hide comment
@jah2488

jah2488 Jun 26, 2017

@cblavier Yep, I needed to change my test.exs to more or less mirror my prod.exs when running under CI. I wrote about it here. https://blog.lunarcollective.co/running-our-tests-on-heroku-ci-with-elixir-and-phoenix-fe01f63a9ebb

jah2488 commented Jun 26, 2017

@cblavier Yep, I needed to change my test.exs to more or less mirror my prod.exs when running under CI. I wrote about it here. https://blog.lunarcollective.co/running-our-tests-on-heroku-ci-with-elixir-and-phoenix-fe01f63a9ebb

@cblavier

This comment has been minimized.

Show comment
Hide comment
@cblavier

cblavier Jun 26, 2017

Ok, thanks, didn't know it was your post! I had already read it, but unfortunately it's not helping :/
See my question here :
https://stackoverflow.com/questions/44760975/cant-make-my-phoenix-app-build-on-heroku-ci

Did you manage to get heroku ci:debug working?

cblavier commented Jun 26, 2017

Ok, thanks, didn't know it was your post! I had already read it, but unfortunately it's not helping :/
See my question here :
https://stackoverflow.com/questions/44760975/cant-make-my-phoenix-app-build-on-heroku-ci

Did you manage to get heroku ci:debug working?

@jah2488

This comment has been minimized.

Show comment
Hide comment
@jah2488

jah2488 Jun 26, 2017

@cblavier From the error you posted on SO, it looks like your test DB is trying to connect directly to your prod DB. I got that error once, because I had copied over my database_url from production to the testing pipeline, which was the wrong approach. The only env vars I have setup for CI are pool_size and ci, which are set to 2 and true respectively.

It may be worth checking out the example app that heroku has up. https://github.com/joshwlewis/giphy-mcgiface/blob/master/config/test.exs

jah2488 commented Jun 26, 2017

@cblavier From the error you posted on SO, it looks like your test DB is trying to connect directly to your prod DB. I got that error once, because I had copied over my database_url from production to the testing pipeline, which was the wrong approach. The only env vars I have setup for CI are pool_size and ci, which are set to 2 and true respectively.

It may be worth checking out the example app that heroku has up. https://github.com/joshwlewis/giphy-mcgiface/blob/master/config/test.exs

trinode added a commit to sky-uk/heroku-buildpack-elixir-appd that referenced this pull request Aug 10, 2017

Add Testpack API implementation to support Heroku CI (#92)
* Add scripts required for Heroku CI

* Don't overwrite existing MIX_ENV

* bin/test and bin/test-compile should be executable

* export_mix_env should fallback without an argument

* No need to specify MIX_ENV, it's already set
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment