Cache parsed bundle #69

Merged
merged 6 commits into from Jun 19, 2013

6 participants

@dpiddy
Heroku member

Currently we're fetching and setting up bundler many times during the detect and compile phases which slows things down.

This at least gets it down to one fetch + setup for each phase. Might consider vendoring bundler to prevent needing to fetch it at all.

@schneems

Hey, can you pull and run tests against your changes? Let me know if you have any questions.

Do you have any numbers or benchmarks by any chances on the speed changes involved here?

@dpiddy
Heroku member

Trying to run specs now. I had to add heroku to the Gemfile to silence warnings.

@dpiddy
Heroku member

The specs passed. I was going to paste their run time on my branch vs the run time on master but I can't get master to reliably pass. Not sure why.

@schneems

I'm out of the country right now, can you give me the failure output of master in an email? I can look at this indepth next week around wednesday.

@dpiddy
Heroku member
% be rspec -fs spec
Run options: include {:focused=>true}

All examples were filtered out; ignoring {:focused=>true}

Bugs
  MRI 1.8.7
https://api.anvilworks.org/slugs/c08fb54f-c775-4f6e-a5fc-a35a95ce1a29.tgz
    should install nokogiri

No Lockfile
Checking for buildpack files to sync... done, 0 files needed
Checking for app files to sync... done, 0 files needed
Launching build process... done 
Preparing app for compilation... done 
Fetching buildpack... done 
Detecting buildpack... done, Ruby/NoLockfile 
Fetching cache... empty 
Compiling app... 
 !
    ERROR: Gemfile.lock required. Please check it in.
 !
  should not deploy

Rails 2.3.x
https://api.anvilworks.org/slugs/a42fe782-abae-4a75-9ed6-b5178ea4252c.tgz
  should deploy on ruby 1.8.7

Rails 3.x
https://api.anvilworks.org/slugs/5cc47c0f-5d6d-422a-b665-de299c786813.tgz
  should deploy on ruby 1.9.3
  when not using the rails gem
https://api.anvilworks.org/slugs/2535bac7-a04f-4fd8-b19d-a6893663bb2c.tgz
    should deploy on ruby 1.9.3

Ruby Versions
https://api.anvilworks.org/slugs/4693e421-a3ef-4b89-9893-c287015afef1.tgz
  should deploy ruby 1.8.7 properly
https://api.anvilworks.org/slugs/1442d5a3-ba60-4d7c-aa57-80bae481f99c.tgz
  should deploy ruby 1.9.2 properly (FAILED - 1)
Counting objects: 8, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 840 bytes, done.
Total 8 (delta 0), reused 8 (delta 0)

-----> Fetching custom git buildpack... done
-----> Ruby/Rack app detected
-----> Using Ruby version: ruby-1.9.2
-----> Installing dependencies using Bundler version 1.3.2
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
       Fetching gem metadata from https://rubygems.org/..........
       Installing rack (1.5.0)
       Using bundler (1.3.2)
       Your bundle is complete! It was installed into ./vendor/bundle
       Cleaning up the bundler cache.
-----> Discovering process types
       Procfile declares types     -> (none)
       Default types for Ruby/Rack -> console, rake, web

-----> Compiled slug size: 10.3MB
-----> Launching... done, v5
       http://test-app-1366119244-665417.herokuapp.com deployed to Heroku

To git@heroku.com:test-app-1366119244-665417.git
 * [new branch]      master -> master
  should deploy ruby 1.9.2 properly (git)
https://api.anvilworks.org/slugs/71dce456-4faf-4a46-8b26-bc1160a87d3d.tgz
Checking for buildpack files to sync... done, 0 files needed
Checking for app files to sync... done, 0 files needed
Launching build process... done 
Preparing app for compilation... done 
Fetching buildpack... done 
Detecting buildpack... done, Ruby/Rack 
Fetching cache... empty 
Compiling app... 
  Using Ruby version: ruby-1.9.3
  Installing dependencies using Bundler version 1.3.2
    Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
    Fetching gem metadata from https://rubygems.org/..........
    Installing rack (1.5.0)
    Using bundler (1.3.2)
    Your bundle is complete! It was installed into ./vendor/bundle
    Cleaning up the bundler cache.
Putting cache... done 
Creating slug... done 
Uploading slug... done 
Success, slug is https://api.anvilworks.org/slugs/71dce456-4faf-4a46-8b26-bc1160a87d3d.tgz 
Releasing to http://test-app-1366119381-557229.herokuapp.com
  should deploy ruby 1.9.3 properly (FAILED - 2)
https://api.anvilworks.org/slugs/e5083c40-ac4e-4281-b584-81bc2dec04ca.tgz
  should deploy ruby 2.0.0 properly (FAILED - 3)

Failures:

  1) Ruby Versions should deploy ruby 1.9.2 properly
     Failure/Error: expect(Excon.get("http://#{app.name}.herokuapp.com").body.chomp).to eq("ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]")

       expected: "ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]"
            got: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n    <html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n        <title>Heroku | Welcome to your new app!</title>\n        <style type='text/css'> \n      body {\n        background-color: white;\n        color: #333333;\n        font-family: Arial, sans-serif;\n        margin: 0;\n        padding: 36px;\n        line-height: 18px;\n        font-size: 14px; }\n\n      .section {\n        margin-bottom: 36px; }\n        .section.friendly {\n          color: #222222; }\n          .section.friendly h1 {\n            font-size: 26px;\n            background-color: #dad8e4;\n            padding: 18px 22px 15px 22px;\n            margin: 0;\n            overflow: hidden; }\n            .section.friendly h1 strong {\n              display: inline-block;\n              float: left; }\n            .section.friendly h1 small {\n              display: inline-block;\n              float: right;\n              text-align: right;\n              font-size: 18px;\n              padding-top: 4px;\n              color: #333333; }\n          .section.friendly .article {\n            border: 4px solid #dad8e4;\n            padding: 24px 18px 18px 18px; }\n            .section.friendly .article h3 {\n              font-size: 20px;\n              margin: 0 0 18px 0; }\n            .section.friendly .article a {\n              color: #6b6ceb; }\n              .section.friendly .article a:visited {\n                color: #1d1d3b; }\n            .section.friendly .article p {\n              font-size: 14px; }\n            .section.friendly .article ul {\n              list-style-type: square; }\n      .section.original {\n        background-color: #eeeeee;\n        color: #444444; }\n        .section.original h2 {\n          background-color: #dddddd;\n          margin: 0;\n          padding: 18px 22px 18px 22px;\n          font-size: 20px; }\n        .section.original pre {\n          margin: 0;\n          padding: 18px 22px 18px 22px;\n          overflow: auto;\n          font-family: monaco, monospaced; }\n          .section.original pre code {\n            display: block;\n            font-size: 11px;\n            width: 100%; }\n    </style>\n    </head>\n    <body>\n        <div class='container'> \n          <div class='section friendly'> \n            <h1><strong>Heroku | Welcome to your new app!</strong></h1>\n            <div class='article'> \n              <p>Refer to the <a href=\"//devcenter.heroku.com/\">documentation</a> if you need help deploying.</p>\n            </div> \n          </div> \n        </div>\n    </body>\n    </html>"

       (compared using ==)

       Diff:
       @@ -1,2 +1,80 @@
       -ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]
       +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
       +    <html>
       +    <head>
       +        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
       +        <title>Heroku | Welcome to your new app!</title>
       +        <style type='text/css'> 
       +      body {
       +        background-color: white;
       +        color: #333333;
       +        font-family: Arial, sans-serif;
       +        margin: 0;
       +        padding: 36px;
       +        line-height: 18px;
       +        font-size: 14px; }
       +
       +      .section {
       +        margin-bottom: 36px; }
       +        .section.friendly {
       +          color: #222222; }
       +          .section.friendly h1 {
       +            font-size: 26px;
       +            background-color: #dad8e4;
       +            padding: 18px 22px 15px 22px;
       +            margin: 0;
       +            overflow: hidden; }
       +            .section.friendly h1 strong {
       +              display: inline-block;
       +              float: left; }
       +            .section.friendly h1 small {
       +              display: inline-block;
       +              float: right;
       +              text-align: right;
       +              font-size: 18px;
       +              padding-top: 4px;
       +              color: #333333; }
       +          .section.friendly .article {
       +            border: 4px solid #dad8e4;
       +            padding: 24px 18px 18px 18px; }
       +            .section.friendly .article h3 {
       +              font-size: 20px;
       +              margin: 0 0 18px 0; }
       +            .section.friendly .article a {
       +              color: #6b6ceb; }
       +              .section.friendly .article a:visited {
       +                color: #1d1d3b; }
       +            .section.friendly .article p {
       +              font-size: 14px; }
       +            .section.friendly .article ul {
       +              list-style-type: square; }
       +      .section.original {
       +        background-color: #eeeeee;
       +        color: #444444; }
       +        .section.original h2 {
       +          background-color: #dddddd;
       +          margin: 0;
       +          padding: 18px 22px 18px 22px;
       +          font-size: 20px; }
       +        .section.original pre {
       +          margin: 0;
       +          padding: 18px 22px 18px 22px;
       +          overflow: auto;
       +          font-family: monaco, monospaced; }
       +          .section.original pre code {
       +            display: block;
       +            font-size: 11px;
       +            width: 100%; }
       +    </style>
       +    </head>
       +    <body>
       +        <div class='container'> 
       +          <div class='section friendly'> 
       +            <h1><strong>Heroku | Welcome to your new app!</strong></h1>
       +            <div class='article'> 
       +              <p>Refer to the <a href="//devcenter.heroku.com/">documentation</a> if you need help deploying.</p>
       +            </div> 
       +          </div> 
       +        </div>
       +    </body>
       +    </html>
     # ./spec/rubies_spec.rb:14:in `block (3 levels) in <top (required)>'
     # ./spec/rubies_spec.rb:12:in `block (2 levels) in <top (required)>'

  2) Ruby Versions should deploy ruby 1.9.3 properly
     Failure/Error: expect(Excon.get("http://#{app.name}.herokuapp.com").body.chomp).to eq("ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]")

       expected: "ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]"
            got: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n    <html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n        <title>Heroku | Welcome to your new app!</title>\n        <style type='text/css'> \n      body {\n        background-color: white;\n        color: #333333;\n        font-family: Arial, sans-serif;\n        margin: 0;\n        padding: 36px;\n        line-height: 18px;\n        font-size: 14px; }\n\n      .section {\n        margin-bottom: 36px; }\n        .section.friendly {\n          color: #222222; }\n          .section.friendly h1 {\n            font-size: 26px;\n            background-color: #dad8e4;\n            padding: 18px 22px 15px 22px;\n            margin: 0;\n            overflow: hidden; }\n            .section.friendly h1 strong {\n              display: inline-block;\n              float: left; }\n            .section.friendly h1 small {\n              display: inline-block;\n              float: right;\n              text-align: right;\n              font-size: 18px;\n              padding-top: 4px;\n              color: #333333; }\n          .section.friendly .article {\n            border: 4px solid #dad8e4;\n            padding: 24px 18px 18px 18px; }\n            .section.friendly .article h3 {\n              font-size: 20px;\n              margin: 0 0 18px 0; }\n            .section.friendly .article a {\n              color: #6b6ceb; }\n              .section.friendly .article a:visited {\n                color: #1d1d3b; }\n            .section.friendly .article p {\n              font-size: 14px; }\n            .section.friendly .article ul {\n              list-style-type: square; }\n      .section.original {\n        background-color: #eeeeee;\n        color: #444444; }\n        .section.original h2 {\n          background-color: #dddddd;\n          margin: 0;\n          padding: 18px 22px 18px 22px;\n          font-size: 20px; }\n        .section.original pre {\n          margin: 0;\n          padding: 18px 22px 18px 22px;\n          overflow: auto;\n          font-family: monaco, monospaced; }\n          .section.original pre code {\n            display: block;\n            font-size: 11px;\n            width: 100%; }\n    </style>\n    </head>\n    <body>\n        <div class='container'> \n          <div class='section friendly'> \n            <h1><strong>Heroku | Welcome to your new app!</strong></h1>\n            <div class='article'> \n              <p>Refer to the <a href=\"//devcenter.heroku.com/\">documentation</a> if you need help deploying.</p>\n            </div> \n          </div> \n        </div>\n    </body>\n    </html>"

       (compared using ==)

       Diff:
       @@ -1,2 +1,80 @@
       -ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]
       +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
       +    <html>
       +    <head>
       +        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
       +        <title>Heroku | Welcome to your new app!</title>
       +        <style type='text/css'> 
       +      body {
       +        background-color: white;
       +        color: #333333;
       +        font-family: Arial, sans-serif;
       +        margin: 0;
       +        padding: 36px;
       +        line-height: 18px;
       +        font-size: 14px; }
       +
       +      .section {
       +        margin-bottom: 36px; }
       +        .section.friendly {
       +          color: #222222; }
       +          .section.friendly h1 {
       +            font-size: 26px;
       +            background-color: #dad8e4;
       +            padding: 18px 22px 15px 22px;
       +            margin: 0;
       +            overflow: hidden; }
       +            .section.friendly h1 strong {
       +              display: inline-block;
       +              float: left; }
       +            .section.friendly h1 small {
       +              display: inline-block;
       +              float: right;
       +              text-align: right;
       +              font-size: 18px;
       +              padding-top: 4px;
       +              color: #333333; }
       +          .section.friendly .article {
       +            border: 4px solid #dad8e4;
       +            padding: 24px 18px 18px 18px; }
       +            .section.friendly .article h3 {
       +              font-size: 20px;
       +              margin: 0 0 18px 0; }
       +            .section.friendly .article a {
       +              color: #6b6ceb; }
       +              .section.friendly .article a:visited {
       +                color: #1d1d3b; }
       +            .section.friendly .article p {
       +              font-size: 14px; }
       +            .section.friendly .article ul {
       +              list-style-type: square; }
       +      .section.original {
       +        background-color: #eeeeee;
       +        color: #444444; }
       +        .section.original h2 {
       +          background-color: #dddddd;
       +          margin: 0;
       +          padding: 18px 22px 18px 22px;
       +          font-size: 20px; }
       +        .section.original pre {
       +          margin: 0;
       +          padding: 18px 22px 18px 22px;
       +          overflow: auto;
       +          font-family: monaco, monospaced; }
       +          .section.original pre code {
       +            display: block;
       +            font-size: 11px;
       +            width: 100%; }
       +    </style>
       +    </head>
       +    <body>
       +        <div class='container'> 
       +          <div class='section friendly'> 
       +            <h1><strong>Heroku | Welcome to your new app!</strong></h1>
       +            <div class='article'> 
       +              <p>Refer to the <a href="//devcenter.heroku.com/">documentation</a> if you need help deploying.</p>
       +            </div> 
       +          </div> 
       +        </div>
       +    </body>
       +    </html>
     # ./spec/rubies_spec.rb:29:in `block (3 levels) in <top (required)>'
     # ./spec/rubies_spec.rb:26:in `block (2 levels) in <top (required)>'

  3) Ruby Versions should deploy ruby 2.0.0 properly
     Failure/Error: expect(Excon.get("http://#{app.name}.herokuapp.com").body.chomp).to eq("ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]")

       expected: "ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]"
            got: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n    <html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n        <title>Heroku | Welcome to your new app!</title>\n        <style type='text/css'> \n      body {\n        background-color: white;\n        color: #333333;\n        font-family: Arial, sans-serif;\n        margin: 0;\n        padding: 36px;\n        line-height: 18px;\n        font-size: 14px; }\n\n      .section {\n        margin-bottom: 36px; }\n        .section.friendly {\n          color: #222222; }\n          .section.friendly h1 {\n            font-size: 26px;\n            background-color: #dad8e4;\n            padding: 18px 22px 15px 22px;\n            margin: 0;\n            overflow: hidden; }\n            .section.friendly h1 strong {\n              display: inline-block;\n              float: left; }\n            .section.friendly h1 small {\n              display: inline-block;\n              float: right;\n              text-align: right;\n              font-size: 18px;\n              padding-top: 4px;\n              color: #333333; }\n          .section.friendly .article {\n            border: 4px solid #dad8e4;\n            padding: 24px 18px 18px 18px; }\n            .section.friendly .article h3 {\n              font-size: 20px;\n              margin: 0 0 18px 0; }\n            .section.friendly .article a {\n              color: #6b6ceb; }\n              .section.friendly .article a:visited {\n                color: #1d1d3b; }\n            .section.friendly .article p {\n              font-size: 14px; }\n            .section.friendly .article ul {\n              list-style-type: square; }\n      .section.original {\n        background-color: #eeeeee;\n        color: #444444; }\n        .section.original h2 {\n          background-color: #dddddd;\n          margin: 0;\n          padding: 18px 22px 18px 22px;\n          font-size: 20px; }\n        .section.original pre {\n          margin: 0;\n          padding: 18px 22px 18px 22px;\n          overflow: auto;\n          font-family: monaco, monospaced; }\n          .section.original pre code {\n            display: block;\n            font-size: 11px;\n            width: 100%; }\n    </style>\n    </head>\n    <body>\n        <div class='container'> \n          <div class='section friendly'> \n            <h1><strong>Heroku | Welcome to your new app!</strong></h1>\n            <div class='article'> \n              <p>Refer to the <a href=\"//devcenter.heroku.com/\">documentation</a> if you need help deploying.</p>\n            </div> \n          </div> \n        </div>\n    </body>\n    </html>"

       (compared using ==)

       Diff:
       @@ -1,2 +1,80 @@
       -ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
       +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
       +    <html>
       +    <head>
       +        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
       +        <title>Heroku | Welcome to your new app!</title>
       +        <style type='text/css'> 
       +      body {
       +        background-color: white;
       +        color: #333333;
       +        font-family: Arial, sans-serif;
       +        margin: 0;
       +        padding: 36px;
       +        line-height: 18px;
       +        font-size: 14px; }
       +
       +      .section {
       +        margin-bottom: 36px; }
       +        .section.friendly {
       +          color: #222222; }
       +          .section.friendly h1 {
       +            font-size: 26px;
       +            background-color: #dad8e4;
       +            padding: 18px 22px 15px 22px;
       +            margin: 0;
       +            overflow: hidden; }
       +            .section.friendly h1 strong {
       +              display: inline-block;
       +              float: left; }
       +            .section.friendly h1 small {
       +              display: inline-block;
       +              float: right;
       +              text-align: right;
       +              font-size: 18px;
       +              padding-top: 4px;
       +              color: #333333; }
       +          .section.friendly .article {
       +            border: 4px solid #dad8e4;
       +            padding: 24px 18px 18px 18px; }
       +            .section.friendly .article h3 {
       +              font-size: 20px;
       +              margin: 0 0 18px 0; }
       +            .section.friendly .article a {
       +              color: #6b6ceb; }
       +              .section.friendly .article a:visited {
       +                color: #1d1d3b; }
       +            .section.friendly .article p {
       +              font-size: 14px; }
       +            .section.friendly .article ul {
       +              list-style-type: square; }
       +      .section.original {
       +        background-color: #eeeeee;
       +        color: #444444; }
       +        .section.original h2 {
       +          background-color: #dddddd;
       +          margin: 0;
       +          padding: 18px 22px 18px 22px;
       +          font-size: 20px; }
       +        .section.original pre {
       +          margin: 0;
       +          padding: 18px 22px 18px 22px;
       +          overflow: auto;
       +          font-family: monaco, monospaced; }
       +          .section.original pre code {
       +            display: block;
       +            font-size: 11px;
       +            width: 100%; }
       +    </style>
       +    </head>
       +    <body>
       +        <div class='container'> 
       +          <div class='section friendly'> 
       +            <h1><strong>Heroku | Welcome to your new app!</strong></h1>
       +            <div class='article'> 
       +              <p>Refer to the <a href="//devcenter.heroku.com/">documentation</a> if you need help deploying.</p>
       +            </div> 
       +          </div> 
       +        </div>
       +    </body>
       +    </html>
     # ./spec/rubies_spec.rb:36:in `block (3 levels) in <top (required)>'
     # ./spec/rubies_spec.rb:34:in `block (2 levels) in <top (required)>'

Finished in 13 minutes 28.8 seconds
10 examples, 3 failures

Failed examples:

rspec ./spec/rubies_spec.rb:11 # Ruby Versions should deploy ruby 1.9.2 properly
rspec ./spec/rubies_spec.rb:25 # Ruby Versions should deploy ruby 1.9.3 properly
rspec ./spec/rubies_spec.rb:33 # Ruby Versions should deploy ruby 2.0.0 properly
@catsby

I get similar errors as well (on master, from a week ago): https://gist.github.com/ctshryock/2475c5475ada16b53bec

@dpiddy
Heroku member

I think this is just a race between the new app page going away and the app actually serving traffic. Not ideal but some retrying here might clear things up.

@dpiddy
Heroku member

Opened #87 to help with this.

@freeformz
Heroku member

#87 is now merged so ping.

@mikehale

bump

@hone hone merged commit 7939d58 into heroku:master Jun 19, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment