`cf push` with multiple buildpacks / non-default stack specified does not work

[tcdowney]

Please fill out the issue checklist below and provide ALL the requested information.

• I tried updating to the latest version of the CF CLI to see if it fixed my problem.
• I attempted to run the command with CF_TRACE=1 to help debug the issue.
• I am reporting a bug that others will be able to reproduce.

Describe the bug and the command you saw an issue with
This behavior was reported to CAPI last week. A user was trying to push an app using multiple buildpacks with a specific stack.

Example:

cf push bug -b ruby_buildpack -b php_buildpack -s cflinuxfs3

When those buildpacks exist for the default stack in the environment (in this case cflinuxfs2) it will actually stage with the cflinuxfs2 buildpacks. (This was wrong, my app got in a weird state) If there are no cflinuxfs2 buildpacks that match the requested names it will fail with a validation error.

See CAPI #160325184 for more information.

What happened
To get this to fail in a more obvious way, I first had to remove/rename the cflinuxfs2 Ruby buildpack:

1. cf curl /v2/buildpacks?q:name=ruby_buildpack
2. Find the one with "stack": "cflinuxfs2" and note its guid
3. cf curl -X PUT /v2/buildpacks/<guid> -d '{"name": "ruby_buildpack_backup"}'

My installed buildpacks now looked like this:

cf buildpacks
Getting buildpacks...

buildpack               position   enabled   locked   filename                                      stack
staticfile_buildpack    1          true      false    staticfile-buildpack-cflinuxfs2-v1.4.31.zip   cflinuxfs2
java_buildpack          2          true      false    java-buildpack-cflinuxfs2-v4.15.zip           cflinuxfs2
ruby_buildpack_backup   3          true      false    ruby-buildpack-cflinuxfs2-v1.7.22.zip         cflinuxfs2
dotnet_core_buildpack   4          true      false    dotnet-core-buildpack-cflinuxfs2-v2.1.4.zip   cflinuxfs2
nodejs_buildpack        5          true      false    nodejs-buildpack-cflinuxfs2-v1.6.30.zip       cflinuxfs2
go_buildpack            6          true      false    go-buildpack-cflinuxfs2-v1.8.26.zip           cflinuxfs2
python_buildpack        7          true      false    python-buildpack-cflinuxfs2-v1.6.20.zip       cflinuxfs2
php_buildpack           8          true      false    php-buildpack-cflinuxfs2-v4.3.59.zip          cflinuxfs2
binary_buildpack        9          true      false    binary-buildpack-cflinuxfs2-v1.0.25.zip       cflinuxfs2
staticfile_buildpack    10         true      false    staticfile-buildpack-cflinuxfs3-v1.4.31.zip   cflinuxfs3
java_buildpack          11         true      false    java-buildpack-cflinuxfs3-v4.15.zip           cflinuxfs3
ruby_buildpack          12         true      false    ruby-buildpack-cflinuxfs3-v1.7.22.zip         cflinuxfs3
dotnet_core_buildpack   13         true      false    dotnet-core-buildpack-cflinuxfs3-v2.1.4.zip   cflinuxfs3
nodejs_buildpack        14         true      false    nodejs-buildpack-cflinuxfs3-v1.6.30.zip       cflinuxfs3
go_buildpack            15         true      false    go-buildpack-cflinuxfs3-v1.8.26.zip           cflinuxfs3
python_buildpack        16         true      false    python-buildpack-cflinuxfs3-v1.6.20.zip       cflinuxfs3
php_buildpack           17         true      false    php-buildpack-cflinuxfs3-v4.3.59.zip          cflinuxfs3
binary_buildpack        18         true      false    binary-buildpack-cflinuxfs3-v1.0.25.zip       cflinuxfs3

Then I attempted to push an app:

cf push bug -b ruby_buildpack -b php_buildpack -s cflinuxfs3

The push failed and said it was not able to find the ruby_buildpack I asked for.

{
  "errors": [
    {
      "code": 10008,
      "detail": "Buildpack \"ruby_buildpack\" must be an existing admin buildpack or a valid git URI",
      "title": "CF-UnprocessableEntity"
    }
  ]
}

The actual API request that CLI made omitted the stack so CC was falling back to the system default stack.

REQUEST: [2018-09-12T16:45:16-07:00]
PATCH /v3/apps/0e48abcf-d260-4c5b-9ba2-2039e9a31a37 HTTP/1.1
Host: api.plume-unicorn.capi.land
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/6.39.0+607d4f8be.2018-09-11 (go1.10.3; amd64 darwin)
{
  "lifecycle": {
    "data": {
      "buildpacks": [
        "ruby_buildpack",
        "php_buildpack"
      ]
    },
    "type": "buildpack"
  }
}

Expected behavior
I would have expected it to find and use the cflinuxfs3 versions of the specified buildpacks to stage.

I was able to get it to work with a manual curl like this:

cf curl /v3/apps/0e48abcf-d260-4c5b-9ba2-2039e9a31a37 -X PATCH -d '{"lifecycle": {"data": {"buildpacks": ["ruby_buildpack","php_buildpack"], "stack": "cflinuxfs3"},"type": "buildpack"}}'

To Reproduce
You can pretty much follow the steps above, but to reiterate them (copy pasta'd from CAPI #160325184):

1. Deploy an environment with the cflinuxfs3 stack and cflinuxfs3-aware buildpacks: add-cflinuxfs3.yml.
2. Rename or remove one of the cflinuxfs2 builpacks for which there is a corresponding cflinuxfs3 buildpack
3. cf curl /v2/buildpacks?q:name=ruby_buildpack
4. Find the one with "stack": "cflinuxfs2" and note its guid
5. cf curl -X PUT /v2/buildpacks/<guid> -d '{"name": "ruby_buildpack_backup"}'
6. Attempt to push an application using multiple buildpacks (including ruby_buildpack) and stack cflinuxfs3: cf push bug -b ruby_buildpack -b php_buildpack -s cflinuxfs3
7. This results in the error: Buildpack "ruby_buildpack" must be an existing admin buildpack or a valid git URI

See CAPI #160325184 for more information on repro steps.

Provide more context

• Mac OS X 10.13 iTerm
• 6.39.0+607d4f8be.2018-09-11
• capi-release 1.68.0

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/160470027

The labels on this github issue will be updated when the story is started.

[zrob]

@sclevine

[tylerphelan]

When those buildpacks exist for the default stack in the environment (in this case cflinuxfs2) it will actually stage with the cflinuxfs2 buildpacks.

@tcdowney can you add the logs of this scenario? We are seeing this:

$ cf p ruby_go -b go_buildpack -b ruby_buildpack -s cflinuxfs3
Go Buildpack version 1.8.26
   -----> Installing godep 80
          Download [https://buildpacks.cloudfoundry.org/dependencies/godep/godep-v80-linux-x64-cflinuxfs3-b60ac947.tgz]
   -----> Installing glide 0.13.1
          Download [https://buildpacks.cloudfoundry.org/dependencies/glide/glide-v0.13.1-linux-x64-cflinuxfs3-6f19a57c.tgz]
   -----> Installing dep 0.5.0
          Download [https://buildpacks.cloudfoundry.org/dependencies/dep/dep-v0.5.0-linux-x64-cflinuxfs3-f9b0e8ec.tgz]
   -----> Installing go 1.8.7
          Download [https://buildpacks.cloudfoundry.org/dependencies/go/go1.8.7.linux-amd64-cflinuxfs3-db742637.tar.gz]
   -----> Ruby Buildpack version 1.7.22
   -----> Supplying Ruby
   -----> Installing ruby 2.3.7
          Download [https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby-2.3.7-linux-x64-cflinuxfs3-a72dc47f.tgz]
...

With cflinuxfs3 in the URLs:
Download [https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby-2.3.7-linux-x64-cflinuxfs3-a72dc47f.tgz]

[tcdowney]

@tylerphelan I think I accidentally got my app in a weird state when I noticed that happen and got the app's stack set back to cflinuxfs2. As long as the CLI successfully sets the stack on the app itself it will behave as you described above because of this fallback:

https://github.com/cloudfoundry/cloud_controller_ng/blob/ea0bf3387a069395377a9df0ef04cd29a482eb9b/lib/cloud_controller/diego/lifecycles/buildpack_lifecycle.rb#L39

[abbyachau]

Hi @tcdowney thanks for creating this issue, and for the great write-up. This issue has been fixed in CLI release v6.39.1. Please let us know if you have any feedback by creating a new issue or hitting us up in CF slack (#cli). Thanks again.