Compiling all project types with libcnb package #575

colincasey · 2023-06-12T14:30:37Z

This PR modifies the libcnb package command to:

Compile all Buildpacks in a project when run from the root of a Cargo Workspace
Compile a single buildpack or meta-buildpack when run from within a directory containing a buildpack.toml

Notable changes to how the command operates:

Buildpacks are compiled to a folder structure similar to target/buildpack/{debug|release}/{buildpack_id}
The target directory of each compiled Buildpack is written to stdout
Meta-buildpacks will detect any local buildpack dependencies that are referenced using a special uri format in package.toml (e.g.; uri = "libcnb:dependent_buildpack_id") and will ensure that those buildpacks are compiled first
Legacy buildpacks that contain just a bin/detect and bin/compile script will be left untouched and their source directory will be output

Closes #215, GUS-W-12731729

Example output

heroku/buildpacks-jvm

STDERR

🔍 Locating buildpacks...
📦 [1/7] Building heroku/java
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_java (0.00 MiB)
📦 [2/7] Building heroku/java-function
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_java-function (0.00 MiB)
📦 [3/7] Building heroku/scala
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_scala (0.00 MiB)
📦 [4/7] Building heroku/maven
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.14s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_maven (10.32 MiB)
📦 [5/7] Building heroku/jvm
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.14s
    Finished dev [unoptimized] target(s) in 0.14s
    Finished dev [unoptimized] target(s) in 0.14s
    Finished dev [unoptimized] target(s) in 0.14s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_jvm (16.40 MiB)
📦 [6/7] Building heroku/jvm-function-invoker
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.13s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_jvm-function-invoker (7.76 MiB)
📦 [7/7] Building heroku/sbt
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.14s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_sbt (7.12 MiB)
✨ Packaging successfully finished!

💡 To test your buildpack locally with pack, run:
pack build my-image-name \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_java \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_java-function \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_scala \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_maven \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_jvm \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_jvm-function-invoker \
  --buildpack /path/to/buildpacks-jvm/target/buildpack/debug/heroku_sbt \
  --path /path/to/application

STDOUT

/path/to/buildpacks-jvm/target/buildpack/debug/heroku_java
/path/to/buildpacks-jvm/target/buildpack/debug/heroku_java-function
/path/to/buildpacks-jvm/target/buildpack/debug/heroku_jvm
/path/to/buildpacks-jvm/target/buildpack/debug/heroku_jvm-function-invoker
/path/to/buildpacks-jvm/target/buildpack/debug/heroku_maven
/path/to/buildpacks-jvm/target/buildpack/debug/heroku_sbt
/path/to/buildpacks-jvm/target/buildpack/debug/heroku_scala

heroku/buildpacks-nodejs

STDERR

🔍 Locating buildpacks...
📦 [1/10] Building heroku/nodejs
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs (0.00 MiB)
📦 [2/10] Building heroku/nodejs-function
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs-function (0.00 MiB)
📦 [3/10] Building heroku/nodejs-test
Writing buildpack directory...
Successfully wrote buildpack directory: ../../../target/buildpack/debug/heroku_nodejs-test (0.00 MiB)
📦 [4/10] Building heroku/nodejs-function-test
Writing buildpack directory...
Successfully wrote buildpack directory: ../../../target/buildpack/debug/heroku_nodejs-function-test (0.00 MiB)
📦 [5/10] Building heroku/nodejs-corepack
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.15s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs-corepack (6.44 MiB)
📦 [6/10] Building heroku/nodejs-pnpm-install
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.14s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs-pnpm-install (6.35 MiB)
📦 [7/10] Building heroku/nodejs-yarn
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.16s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs-yarn (10.00 MiB)
📦 [8/10] Building heroku/nodejs-engine
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.16s
    Finished dev [unoptimized] target(s) in 0.16s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs-engine (10.92 MiB)
📦 [9/10] Building heroku/nodejs-function-invoker
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.15s
Writing buildpack directory...
Successfully wrote buildpack directory: ../../target/buildpack/debug/heroku_nodejs-function-invoker (6.43 MiB)
📦 [10/10] Building heroku/nodejs-npm
Not a libcnb buildpack, nothing to compile...
✨ Packaging successfully finished!

💡 To test your buildpack locally with pack, run:
pack build my-image-name \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-function \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-test \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-function-test \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-corepack \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-pnpm-install \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-yarn \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-engine \
  --buildpack /path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-function-invoker \
  --buildpack /path/to/buildpacks-nodejs/buildpacks/npm \
  --path /path/to/application

STDOUT

/path/to/buildpacks-nodejs/buildpacks/npm
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-corepack
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-engine
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-function
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-function-invoker
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-function-test
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-pnpm-install
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-test
/path/to/buildpacks-nodejs/target/buildpack/debug/heroku_nodejs-yarn

heroku/buildpacks-python

STDERR

🔍 Locating buildpacks...
📦 [1/1] Building heroku/python
Determining automatic cross-compile settings...
Building binaries (x86_64-unknown-linux-musl)...
    Finished dev [unoptimized] target(s) in 0.11s
Writing buildpack directory...
Successfully wrote buildpack directory: target/buildpack/debug/heroku_python (8.00 MiB)
✨ Packaging successfully finished!

💡 To test your buildpack locally with pack, run:
pack build my-image-name \
  --buildpack /path/to/buildpacks-python/target/buildpack/debug/heroku_python \
  --path /path/to/application

STDOUT

/path/to/buildpacks-python/target/buildpack/debug/heroku_python

* Compile all Buildpacks in a project when run from the root of a Cargo Workspace * Compile a single buildpack or meta-buildpack when run from within a directory containing a `buildpack.toml` Notable changes to how the command operates: * Buildpacks are compiled to a folder structure similar to `target/buildpack/{buildpack_id}` * The target directory of each compiled Buildpack is written to `stdout` * Meta-buildpacks will detect any local buildpack dependencies that are referenced via file paths in `package.toml` (e.g.; `uri = “libcnb:buildpack_id”`) and will ensure that those buildpacks are compiled first

* main: Update fastrand requirement from 1.8.0 to 2.0.0 (#573) Fix lint failures with Rust 1.70 (#574) Bump Swatinem/rust-cache from 2.3.0 to 2.4.0 (#572) Bump Swatinem/rust-cache from 2.2.1 to 2.3.0 (#571) Bump buildpacks/github-actions from 5.1.0 to 5.2.0 (#569) Prepare 0.12.0 release (#568) Support Buildpack API `0.9` (#567) Add Clone for Scope (#566) Improve Env::get, add Env::get_string_lossy (#565) Bump buildpacks/github-actions from 5.0.1 to 5.1.0 (#564)

Malax

Wow, this has come a long way since the first review we did some time ago, good job! :) There are still some kinks to iron out, but nothing that requires re-architecting the code-base. LMK when I didn't to a great job explaining what I mean in my comments so that I have a chance to fix them :)

.editorconfig

libcnb-cargo/src/exit_code.rs

libcnb-cargo/src/package/command.rs

libcnb-cargo/src/package/error.rs

libcnb-data/src/buildpackage.rs

Malax

I think we should do another polish pass over this before reviewing since:

Code doesn't pass linting checks
Many identifiers still have their old names from when the graph stuff was buildpack specific which isn't accurate anymore.
We said we wanted to change the naming for some things we introduced during our pairing session, TopoSort#deps for example. I leaning towards TopoSort itself not being a great name.
IntelliJ reported spelling mistakes (I fixed them via 1c2f423 already)
At least one function (lookup_buildpack_package_node_index) is no longer useful after we removed the buildpack specifics since it doesn't to much anymore - the cost of keeping it around is higher than coping the body to the locations it's being used.

From my previous review and the changes we made in our paring session, I think merging is fine after we did another polish run.

CHANGELOG.md

libcnb-data/src/buildpackage.rs

Malax

Some of my comments from the previous review weren't resolved, especially

Many identifiers still have their old names from when the graph stuff was buildpack specific which isn't accurate anymore.

I went ahead and did it myself to speed things up. See: 70f89c1.

I think we can merge this in this state now, after @edmorley's review comments have been addressed.

colincasey requested a review from a team as a code owner June 12, 2023 14:30

colincasey added 2 commits June 12, 2023 11:31

Update CHANGELOG.md

b9a24c7

Malax self-assigned this Jun 12, 2023

colincasey changed the title ~~This PR modifies the libcnb package command to:~~ Compiling all project types with libcnb package Jun 12, 2023

Malax requested changes Jun 13, 2023

View reviewed changes

colincasey and others added 4 commits June 14, 2023 17:07

Feedback from code review

b138ab7

Feedback from code review

4bb756a

Introduce TopoSort

2aaaeed

Feedback from code review

31e0b32

colincasey requested a review from Malax June 15, 2023 18:00

Fix spelling

1c2f423

Malax requested changes Jun 19, 2023

View reviewed changes

colincasey added 3 commits June 19, 2023 10:27

Feedback from code review

cce36cf

Feedback from code review

eaf3393

Feedback from code review

629351c

edmorley reviewed Jun 19, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

libcnb-data/src/buildpackage.rs Show resolved Hide resolved

Fix naming, docs and remove unnecessary types

70f89c1

Malax force-pushed the feature/W-12731729_libcnb_package branch from 51e9a5f to 70f89c1 Compare June 20, 2023 08:19

Malax approved these changes Jun 20, 2023

View reviewed changes

Feedback from code review

6ecbb10

colincasey requested a review from edmorley June 20, 2023 14:11

edmorley added enhancement New feature or request libcnb-cargo libcnb-data libcnb-package labels Jun 21, 2023

edmorley approved these changes Jun 21, 2023

View reviewed changes

colincasey merged commit 5a22147 into main Jun 21, 2023
8 checks passed

colincasey deleted the feature/W-12731729_libcnb_package branch June 21, 2023 12:10

This was referenced Jul 17, 2023

libcnb-cargo: Example pack build command no longer uses relative paths after #575 #599

Closed

libcnb-cargo: Prevent stdout spam when not packaging for automation #600

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compiling all project types with libcnb package #575

Compiling all project types with libcnb package #575

colincasey commented Jun 12, 2023

Malax left a comment

Malax left a comment •

edited

Malax left a comment

Compiling all project types with libcnb package #575

Compiling all project types with libcnb package #575

Conversation

colincasey commented Jun 12, 2023

Example output

STDERR

STDOUT

STDERR

STDOUT

STDERR

STDOUT

Malax left a comment

Choose a reason for hiding this comment

Malax left a comment • edited

Choose a reason for hiding this comment

Malax left a comment

Choose a reason for hiding this comment

Malax left a comment •

edited