Buildpack reusing module layer even when the value of env vars change

[arjun024]

It looks like the buildpack does not consider env variables in its decision to reuse the module layer or not. For e.g., a user should be able to pass NODE_ENV=development|production and influence the actions of the yarn-install command. If the env vars are not considered in the decision to reuse layer, yarn install wouldn't be run again with the new env vars context.

Logs while using this app with the nodejs metabuildpack. See relevant "Yarn Install" section. The buildpack is Reusing cached layer /layers/paketo-buildpacks_yarn-install/modules though the value of myenv has changed in the second build.

# First Build
pack build myapp --env myenv=foo -b gcr.io/paketo-buildpacks/nodejs

===> DETECTING
[detector] paketo-buildpacks/node-engine  0.1.2
[detector] paketo-buildpacks/yarn         0.0.2
[detector] paketo-buildpacks/yarn-install 0.2.0
[detector] paketo-buildpacks/yarn-start   0.0.2
===> ANALYZING
[analyzer] Previous image with name "index.docker.io/library/myapp:latest" not found
[analyzer] Restoring metadata for "paketo-buildpacks/yarn:yarn" from cache
===> RESTORING
[restorer] Restoring data for "paketo-buildpacks/yarn:yarn" from cache
===> BUILDING
[builder] Paketo Node Engine Buildpack 0.1.2
[builder]   Resolving Node Engine version
[builder]     Candidate version sources (in priority order):
[builder]                 -> ""
[builder]       <unknown> -> "*"
[builder]
[builder]     Selected Node Engine version (using ): 10.22.1
[builder]
[builder]   Executing build process
[builder]     Installing Node Engine 10.22.1
[builder]       Completed in 1.461s
[builder]
[builder]   Configuring environment
[builder]     NODE_ENV     -> "production"
[builder]     NODE_HOME    -> "/layers/paketo-buildpacks_node-engine/node"
[builder]     NODE_VERBOSE -> "false"
[builder]
[builder]     Writing profile.d/0_memory_available.sh
[builder]       Calculates available memory based on container limits at launch time.
[builder]       Made available in the MEMORY_AVAILABLE environment variable.
[builder]
[builder] Paketo Yarn Buildpack 0.0.2
[builder]   Reusing cached layer /layers/paketo-buildpacks_yarn/yarn
[builder]
[builder] Paketo Yarn Install Buildpack 0.2.0
[builder]   Resolving installation process
[builder]     Process inputs:
[builder]       yarn.lock -> Found
[builder]
[builder]     Selected default build process: 'yarn install'
[builder]
[builder]   Executing build process
[builder]     Running yarn install --ignore-engines --frozen-lockfile --modules-folder /layers/paketo-buildpacks_yarn-install/modules/node_modules
[builder]       Completed in 1.046s
[builder]
[builder]   Configuring environment
[builder]     PATH -> "$PATH:/layers/paketo-buildpacks_yarn-install/modules/node_modules/.bin"
[builder] Paketo Yarn Start Buildpack 0.0.2
[builder]   Assigning launch processes
[builder]     web: node server.js
[builder]
===> EXPORTING
[exporter] Adding layer 'paketo-buildpacks/node-engine:node'
[exporter] Adding layer 'paketo-buildpacks/yarn-install:modules'
[exporter] Adding 1/1 app layer(s)
[exporter] Adding layer 'launcher'
[exporter] Adding layer 'config'
[exporter] Adding layer 'process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Setting default process type 'web'
[exporter] *** Images (a11747b628bc):
[exporter]       index.docker.io/library/myapp:latest
[exporter] Reusing cache layer 'paketo-buildpacks/node-engine:node'
[exporter] Reusing cache layer 'paketo-buildpacks/yarn:yarn'
[exporter] Reusing cache layer 'paketo-buildpacks/yarn-install:modules'
Successfully built image myapp

# Now build again with a different value for the env var
$ pack build myapp --env myenv=bar -b gcr.io/paketo-buildpacks/nodejs

===> DETECTING
[detector] paketo-buildpacks/node-engine  0.1.2
[detector] paketo-buildpacks/yarn         0.0.2
[detector] paketo-buildpacks/yarn-install 0.2.0
[detector] paketo-buildpacks/yarn-start   0.0.2
===> ANALYZING
[analyzer] Restoring metadata for "paketo-buildpacks/node-engine:node" from app image
[analyzer] Restoring metadata for "paketo-buildpacks/yarn:yarn" from cache
[analyzer] Restoring metadata for "paketo-buildpacks/yarn-install:modules" from app image
===> RESTORING
[restorer] Restoring data for "paketo-buildpacks/node-engine:node" from cache
[restorer] Restoring data for "paketo-buildpacks/yarn:yarn" from cache
[restorer] Restoring data for "paketo-buildpacks/yarn-install:modules" from cache
===> BUILDING
[builder] Paketo Node Engine Buildpack 0.1.2
[builder]   Resolving Node Engine version
[builder]     Candidate version sources (in priority order):
[builder]                 -> ""
[builder]       <unknown> -> "*"
[builder]
[builder]     Selected Node Engine version (using ): 10.22.1
[builder]
[builder]   Reusing cached layer /layers/paketo-buildpacks_node-engine/node
[builder]
[builder] Paketo Yarn Buildpack 0.0.2
[builder]   Reusing cached layer /layers/paketo-buildpacks_yarn/yarn
[builder]
[builder] Paketo Yarn Install Buildpack 0.2.0
[builder]   Resolving installation process
[builder]     Process inputs:
[builder]       yarn.lock -> Found
[builder]
[builder]     Selected default build process: 'yarn install'
[builder]
[builder]   Reusing cached layer /layers/paketo-buildpacks_yarn-install/modules
[builder] Paketo Yarn Start Buildpack 0.0.2
[builder]   Assigning launch processes
[builder]     web: node server.js
[builder]
===> EXPORTING
[exporter] Reusing layer 'paketo-buildpacks/node-engine:node'
[exporter] Reusing layer 'paketo-buildpacks/yarn-install:modules'
[exporter] Reusing 1/1 app layer(s)
[exporter] Reusing layer 'launcher'
[exporter] Reusing layer 'config'
[exporter] Reusing layer 'process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Setting default process type 'web'
[exporter] *** Images (a11747b628bc):
[exporter]       index.docker.io/library/myapp:latest
[exporter] Reusing cache layer 'paketo-buildpacks/node-engine:node'
[exporter] Reusing cache layer 'paketo-buildpacks/yarn:yarn'
[exporter] Reusing cache layer 'paketo-buildpacks/yarn-install:modules'
Successfully built image myapp

[arjun024]

Similar issue in go: paketo-buildpacks/go-build#88
Maybe this should be taken care at the packit level

[ryanmoran]

Similar issue in go: paketo-buildpacks/go-build#88
Maybe this should be taken care at the packit level

How would packit know which environment variables should impact the reuse of a layer. Ideally, we would not invalidate all layers because of the change of an environment variable, which means we would need an API that is tailored to enable each layer to be invalidated based on some specific variable criteria. What would such an API look like?

[arjun024]

See paketo-buildpacks/npm-install#153 (comment) for ideas

[ForestEckhardt]

In particular yarn config list could be a good command to investigate when it comes to determining yarn configuration.

[arjun024]

For posterity, here's a sample output of yarn config list:

$ yarn config list
yarn config v1.22.10
info yarn config
{
  'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-commit-hooks': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'bin-links': true,
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/1.22.10 npm/? node/v12.18.4 darwin x64',
  lastUpdateCheck: 1614289138425
}
info npm config
{
  python: '/usr/bin/python',
  'build-from-source': true
}

See https://classic.yarnpkg.com/en/docs/envvars/, it includes basic yarn config + npm config passed down via npm_config_* environment variables.