buildbox-artifact hanging

[tarragon]

This initially seemed similar to #30 however I am not seeing any certificate errors, and I've also upgraded to the version listed in this issue (0.2-beta.3) but I am still seeing the issue.

Basically it seems that artifacts are uploaded to S3 (Amazon) correctly but the process never finishes, and builds will hang forever on this step. Running an strace against the process doesn't yield anything particularly useful, other than it looping through some kind of wait state.

[keithpitt]

Does running the buildbox-artifact command show anything useful? As you can see here: https://github.com/buildboxhq/buildbox-agent/blob/master/templates/bootstrap.sh#L77 we supress any output from buildbox-artifact. Are you doing something similar?

[tarragon]

Nothing particularly useful. We don't suppress the output on our builds. I've run it manually and replicated the resultant hang. I copied the buildbox env settings from a running build, and also set the AWS env variables.

ci@ci2:~/.buildbox/ci2/ferocia/johanna$ ~ci/.buildbox/buildbox-artifact upload "coverage/**/*; tmp/capybara/screenshot_*.png; tmp/capybara/screenshot_*.html;log_files.zip;spec/ui/**/candidate.png;spec/ui/**/diff.png" "s3://ferocia-build-artifacts/testing" --url https://agent.buildbox.io/v1
2014/06/11 11:35:21 Uploading 34 files that match "coverage/**/*; tmp/capybara/screenshot_*.png; tmp/capybara/screenshot_*.html;log_files.zip;spec/ui/**/candidate.png;spec/ui/**/diff.png"
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/.last_run.json
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/.resultset.json
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets
2014/06/11 11:35:24 Error uploading artifact /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets (Failed to read file /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets (read /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets: is a directory))
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/covered_percent
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/index.html
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/simplecov.lockfile
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0
2014/06/11 11:35:24 Error uploading artifact /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0 (Failed to read file /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0 (read /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0: is a directory))
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/application.css
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/application.js
2014/06/11 11:35:24 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox
2014/06/11 11:35:24 Error uploading artifact /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox (Failed to read file /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox (read /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox: is a directory))
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/favicon_green.png
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/favicon_red.png
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/favicon_yellow.png
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/loading.gif
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/magnify.png
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness
2014/06/11 11:35:27 Error uploading artifact /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness (Failed to read file /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness (read /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness: is a directory))
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox/border.png
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox/controls.png
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox/loading.gif
2014/06/11 11:35:27 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/colorbox/loading_background.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images
2014/06/11 11:35:34 Error uploading artifact /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images (Failed to read file /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images (read /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images: is a directory))
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_flat_0_aaaaaa_40x100.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_flat_75_ffffff_40x100.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_glass_55_fbf9ee_1x400.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_glass_65_ffffff_1x400.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_glass_75_dadada_1x400.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_glass_75_e6e6e6_1x400.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_glass_95_fef1ec_1x400.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-bg_highlight-soft_75_cccccc_1x100.png
2014/06/11 11:35:34 Uploading /home/ci/.buildbox/ci2/ferocia/johanna/coverage/assets/0.8.0/smoothness/images/ui-icons_222222_256x240.png

It's hasn't returned to the shell after that last file.

Note the "Failed to read file ... is a directory" errors are expected and normal.

[keithpitt]

So it just gets stuck after that?

[tarragon]

Yep. I let it sit for about 10 minutes then had to CTRL-C it.

[keithpitt]

What happens as a test, if you remove the "upload the s3" part, and upload to BB. Does that change anything? (Sorry about the hassle)

[keithpitt]

Also try a "--debug" on the command to get more info.

[tarragon]

As an aside, and it might be a different issue, but we're getting a lot of stuck builds at the moment.

Given the issues with artifacts, I simply disabled them for the time being yesterday so we could get our builds going. However I am now finding that buildbox-agent simply stops responding, on both servers, usually within about 10-15 mins of each other (worth noting our build time is about 15 mins).

I downgraded one of the systems to 0.1 for comparison, but both are still seeing the same symptoms.

I'm now running both in a console with --debug, we'll see what comes out of that.

One particularly annoying part of this is that sometimes the agent is telling the buildbox server that it's picked up a build, but then never reports that it's even started the first part of the build. This results in the buildbox server never giving the ci box another job, even after restarting the agent. The only way I can track these down (short of stopping every queued build and re-queueing them) is by looking at the agent status and seeing what build it thinks it's running, then re-queueing that one.

[keithpitt]

@tarragon when you see that behaviour again (with the stuck agent) can you ping me? I'd like to hop into the DB when it's happening to see what's going on. I'm rewriting the logging for the agent here: #33 hopefully when this gets merged in, I'll be able to know more about what's getting stuck.

[tarragon]

@keithpitt will do. Oddly enough it's been quite stable now that I'm running it manually (not as a service) and with debug on.

[tarragon]

I haven't seen the hanging issue again while running the agent in the foreground with debug enabled.

I've just updated to the latest beta (beta8) and re-enabled artifacts, I'll see how it goes.

[tarragon]

I am using beta8 on three systems at the moment, running in a terminal with --debug.

2 of the 3 agents have frozen, are no longer polling, and stopped putting any information into the debug terminal.

One froze at 12:43:08, the other at 12:19:30.

[keithpitt]

@sj26 has just made some tweaks to how artifacts are handled. When you get some time, can you upgrade to https://github.com/buildboxhq/buildbox-agent/releases/tag/v0.2-beta.10

[tarragon]

Since beta9 and beta10 it's been fairly stable, I haven't seen the buildbox process hang and artifacts haven't hung for a while now.

[keithpitt]

I'm going to close this. Hopefully it doesn't come back :)