curl failing sporadically during pre-receive hook #509

Closed
rlister opened this Issue Mar 26, 2014 · 10 comments

Projects

None yet

8 participants

@rlister
rlister commented Mar 26, 2014

I have several dokku setups running on multiple 12.04 instances in separate providers and datacenters. They all work just fine, except sporadically git push deployments will fail (on about 1 in 5 occasions) with:

       New app detected loading default bundler cache
 !
 !     Command: 'set -o pipefail; curl --fail --retry 3 --retry-delay 1 --connect-timeout 3 --max-time 30 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.0.0-p451-default-cache.tgz -s -o - | tar zxf -' failed unexpectedly:
 !
 !     gzip: stdin: unexpected end of file
 !     tar: Unexpected EOF in archive
 !     tar: Unexpected EOF in archive
 !     tar: Error is not recoverable: exiting now

On these occasions, redoing the push one or two times will succeed. That full curl command always succeeds when run manually.

@nickl-
nickl- commented Apr 1, 2014

Unexpected EOF sounds like network congestion or possibly early timeouts try increase connection timeout to be more patient and accommodating for slow transfers.

Could also be a myriad of other culprits so my advice, tick this of your list first... I have also experienced aws prematurely abandoning connections from time to time.

@rlister
rlister commented Apr 4, 2014

This is not a network issue on that curl. I currently can run the curl command with 100% reliability from outside of the dokku hook, but it fails when called from the hook.

@nickl-
nickl- commented Apr 6, 2014

@rlister you seem to have it all figured out then, well done!

@rlister
rlister commented Apr 8, 2014

@nickl- sorry, I don't understand this comment; the curl continues to fail when called from the buildstep. This is happening from multiple providers and datacenters and is absolutely not a network issue.

@Tarang
Tarang commented Apr 13, 2014

@rlister Maybe check the checksum of the file on your instance to check if it matches that at https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.0.0-p451-default-cache.tgz

@elia
Contributor
elia commented Apr 13, 2014

I see this regularly too. I ended up forking the buildpack and modifying it to cache downloads, but I'm not happy with that nor proud.

@cameron-martin
Contributor

I was having this problem too, and it turned out that it was just a timeout issue. It was sometimes taking over 10 seconds to connect to s3. It turns out you can control --connect-timeout and --max-time with the CURL_CONNECT_TIMEOUT and CURL_TIMEOUT environment variables. See https://github.com/heroku/heroku-buildpack-ruby/blob/master/lib/language_pack/fetcher.rb#L31-L41.

Not sure if there is already a way to set these, but I wrote a plugin to do so. https://github.com/cameron-martin/dokku-build-env. See if that fixes your problem.

@henry74
henry74 commented Aug 6, 2014

@cameron-martin Thanks cameron - someone had suggested using the built-in dokku config command which didn't work for me. Your plugin did the trick.

@kylesuss

Just had this problem and it was as a result of using Ruby 2.1.3. I guess the buildpack for this Ruby version is restricted for some reason right now. More on that here. For now, I just downgrade my Ruby version and it worked.

@josegonzalez
Member

Closing as you can use @cameron-martin's plugin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment