-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implements the `bin/report` build report API, as a replacement for log based metrics (which stopped working some time ago with the change in internal observability vendor). This will help with varies upcoming Python buildpack changes, such as the EOL of Python 3.8, assessing usage levels of different package managers (wrt Poetry addition, and future of Pipenv for CNB), removal of the legacy sqlite feature etc. This implementation is based on that used by the Node.js buildpack, which is currently the only buildpack that implements `bin/report`. There are a number of rough edges with the Node.js implementation that in an ideal world we'd improve upon. However, given that that implementation works, is actively used in production, and CNBs are our priority moving forwards, I've chosen not to make many changes to that design. See: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/report https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/metadata.sh https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/kvstore.sh GUS-W-8047826.
- Loading branch information
Showing
19 changed files
with
387 additions
and
126 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| #!/usr/bin/env bash | ||
| # Usage: bin/report <build-dir> <cache-dir> <env-dir> | ||
|
|
||
| # Produces a build report containing metadata about the build, that's consumed by the build system. | ||
| # This script is run for both successful and failing builds, so it should not assume the build ran | ||
| # to completion (e.g. Python or other tools may not even have been installed). | ||
| # | ||
| # Metadata must be emitted to stdout as valid YAML key-value pairs. Any fields that should always | ||
| # be typed as a string must be explicitly quoted. | ||
| # | ||
| # Example valid stdout: | ||
| # python_version: 'X.Y.Z' | ||
| # python_install_duration: 1.234 | ||
| # | ||
| # Failures in this script don't cause the overall build to fail (and won't appear in user | ||
| # facing build logs) to avoid breaking builds unnecessarily / causing confusion. To debug | ||
| # issues check the internal build system logs for `buildpack.report.failed` events, or | ||
| # when developing run `make compile` in this repo locally, which runs `bin/report` too. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| CACHE_DIR="${2}" | ||
|
|
||
| # The absolute path to the root of the buildpack. | ||
| BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd) | ||
|
|
||
| # The build system doesn't source the `export` script before running this script, so we have to do | ||
| # so manually (if it exists) so that buildpack Python/Pip can be found (if the build succeeded). | ||
| # We have to disable Bash error on undefined variables for now, since not all env vars used in the | ||
| # export script will be set by default (eg `LIBRARY_PATH`). | ||
| EXPORT_FILE="${BUILDPACK_DIR}/export" | ||
| if [[ -f "${EXPORT_FILE}" ]]; then | ||
| set +u | ||
| # shellcheck source=/dev/null | ||
| source "${EXPORT_FILE}" | ||
| set -u | ||
| fi | ||
|
|
||
| source "${BUILDPACK_DIR}/lib/metadata.sh" | ||
| meta_init "${CACHE_DIR}" "python" | ||
|
|
||
| # Emit the key / value pair unquoted to stdout. Skips if the value is empty. | ||
| # Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/report | ||
| kv_pair() { | ||
| local key="${1}" | ||
| local value="${2}" | ||
| if [[ -n "${value}" ]]; then | ||
| echo "${key}: ${value}" | ||
| fi | ||
| } | ||
|
|
||
| # Emit the key / value pair to stdout, safely quoting the string. Skips if the value is empty. | ||
| # Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/report | ||
| # (Though instead uses single quotes instead of double to avoid escaping issues.) | ||
| kv_pair_string() { | ||
| local key="${1}" | ||
| local value="${2}" | ||
| if [[ -n "${value}" ]]; then | ||
| # Escape any existing single quotes, which for YAML means replacing `'` with `''`. | ||
| value="${value//\'/\'\'}" | ||
| echo "${key}: '${value}'" | ||
| fi | ||
| } | ||
|
|
||
| STRING_FIELDS=( | ||
| django_collectstatic | ||
| failure_reason | ||
| nltk_downloader | ||
| package_manager | ||
| pip_version | ||
| pipenv_version | ||
| python_version_major | ||
| python_version_reason | ||
| python_version | ||
| setuptools_version | ||
| wheel_version | ||
| ) | ||
|
|
||
| # We don't want to quote numeric or boolean fields. | ||
| ALL_OTHER_FIELDS=( | ||
| dependencies_install_duration | ||
| django_collectstatic_duration | ||
| nltk_downloader_duration | ||
| pipenv_has_lockfile | ||
| post_compile_hook | ||
| post_compile_hook_duration | ||
| pre_compile_hook | ||
| pre_compile_hook_duration | ||
| python_install_duration | ||
| setup_py_only | ||
| sqlite_install_duration | ||
| total_duration | ||
| ) | ||
|
|
||
| for field in "${STRING_FIELDS[@]}"; do | ||
| kv_pair_string "${field}" "$(meta_get "${field}")" | ||
| done | ||
|
|
||
| for field in "${ALL_OTHER_FIELDS[@]}"; do | ||
| kv_pair "${field}" "$(meta_get "${field}")" | ||
| done | ||
|
|
||
| # If the build failed, pip might not have been installed yet. | ||
| if command -v pip >/dev/null; then | ||
| # Determine pysqlite3 usage since it's the only package that requires the sqlite3 headers. | ||
| if pip show pysqlite3 &>/dev/null; then | ||
| kv_pair pysqlite3_installed true | ||
| else | ||
| kv_pair pysqlite3_installed false | ||
| fi | ||
|
|
||
| if pip show pysqlite3-binary &>/dev/null; then | ||
| kv_pair pysqlite3_binary_installed true | ||
| else | ||
| kv_pair pysqlite3_binary_installed false | ||
| fi | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,12 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| if [[ -f bin/post_compile ]]; then | ||
| post_compile_hook_start_time=$(nowms) | ||
| meta_set "post_compile_hook" "true" | ||
| echo "-----> Running post-compile hook" | ||
| chmod +x bin/post_compile | ||
| sub_env bin/post_compile | ||
| fi | ||
| meta_time "post_compile_hook_duration" "${post_compile_hook_start_time}" | ||
| else | ||
| meta_set "post_compile_hook" "false" | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,12 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| if [[ -f bin/pre_compile ]]; then | ||
| pre_compile_hook_start_time=$(nowms) | ||
| meta_set "pre_compile_hook" "true" | ||
| echo "-----> Running pre-compile hook" | ||
| chmod +x bin/pre_compile | ||
| sub_env bin/pre_compile | ||
| fi | ||
| meta_time "pre_compile_hook_duration" "${pre_compile_hook_start_time}" | ||
| else | ||
| meta_set "pre_compile_hook" "false" | ||
| fi |
Oops, something went wrong.