Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e518e14
feat(composer): install PECL extension
EtienneM Dec 27, 2022
eb2a08f
feat(composer): make use of the cache for PECL extensions
EtienneM Dec 28, 2022
2b4f51a
fix(pecl): allows to specify zend_extension configuration as env var
SCedricThomas Jan 12, 2023
a2b288a
fix(pecl): correct zend typo
SCedricThomas Jan 12, 2023
7a0c232
fix(pecl): allows to configure build args
SCedricThomas Jan 12, 2023
afd1a1a
fix(pecl): allows to substitute $BUILD_DIR
SCedricThomas Jan 13, 2023
c5aea70
fix(pecl): change sed delimiter to allows path substitution
SCedricThomas Jan 13, 2023
078bddc
fix(pecl): invert zend condition
SCedricThomas Jan 13, 2023
17ab35f
feat(pecl): allows to retrieve the latest version of an extension usi…
SCedricThomas Jan 13, 2023
db3894b
feat(oci8); oci8 is now available, some code should be 'cleaned'
QuentinEscudierScalingo Jan 23, 2023
1ee469d
fix(oci8): oracle client installed in /app/vendor/oracle-client and l…
QuentinEscudierScalingo Jan 24, 2023
7a5a59c
fix(oci8): clean code
QuentinEscudierScalingo Jan 25, 2023
2e8f674
fix(oci8): set -e in compile
QuentinEscudierScalingo Jan 25, 2023
336de5f
Correctly export LD_LIBRARY_PATH from sub-APT buildpack when installi…
leo-scalingo Jan 26, 2023
b6b2566
Rever 'set -e'
leo-scalingo Jan 26, 2023
190e897
Fix cache directory in pecl installation
leo-scalingo Jan 26, 2023
d253f4d
Merge pull request #291 from Scalingo/feat/290/support_oci8_extension
Soulou Jan 26, 2023
83b3b1d
Polish codebase, make code more generic between internal packages and…
leo-scalingo Jan 27, 2023
7903a16
Review + readme
leo-scalingo Feb 2, 2023
e49d20a
Update lib/apt - better logging
Soulou Feb 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 54 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ support.
* Lightweight stack compared to Apache-ModPHP
* Composer support
* Various frameworks support out of the box (no configuration)
* Dynamic installing of [supported extensions](support/ext) listed as `ext-` requirements in `composer.json`.
* Dynamic installing of PHP extensions, listed as `ext-` requirements in `composer.json`.
* Interfacing with PECL platform to download and build extensions dynamically

## How to use it

Expand Down Expand Up @@ -415,6 +416,58 @@ A tail on each unique log file will be run at application startup
],


### PHP Extensions

Different types of extensions can be added to the PHP runtime, with different
levels of compatibility. All these extensions can be added in the `require`
block of the `composer.json` and this buildpack will try to install them.

```
{
...
"require": {
"ext-sodium": "*",
"ext-calendar": "*",
}
}
```

#### Internal PHP Extensions

The PHP binary is not built with all modules embedded. To reduce build size and
attack surface, these extensions have to be enabled manually. The supported list
can be found here: https://github.com/Scalingo/php-buildpack/tree/master/support/ext-internal

#### PECL PHP Extensions

PECL extensions are registered on the https://pecl.php.net platform. This
buildpack will try to download and build them during deployments.

Some extensions may require system dependencies absent from the stack your
application is based on. In this case you're encouraged to prepend the [APT
Buildpack](https://doc.scalingo.com/platform/deployment/buildpacks/apt) and
install the missing package according to the extension documentation.

You may need to add extra compilation flags, in this case define them through
environment variables with the following pattern:

```
# replace $extension_name by the right extension name
PHP_PECL_EXTENSION_CONFIGURE_ARGS_$extension_name="--flag-to-add"
```

If the extension you are trying to install is a Zend extension, please define

```
# replace $extension_name by the right extension name
PHP_PECL_EXTENSION_IS_ZEND_$extension_name=true
```

#### Third Party PHP Extensions

Some third party extensions can also be installed, list can be found here
https://github.com/Scalingo/php-buildpack/tree/master/support/ext

## Node.js

If your app contains a `package.json`, Node.js and its dependencies will be
Expand Down
50 changes: 7 additions & 43 deletions bin/compile
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,14 @@ shopt -s dotglob
basedir="$( cd -P "$( dirname "$0" )" && pwd )"
source "$basedir/../conf/buildpack.conf"
source $basedir/common.sh
source $basedir/../lib/package
source $basedir/../lib/composer
source $basedir/../lib/datadog
source $basedir/../lib/scout
source $basedir/../lib/newrelic
source $basedir/../lib/apt
source $basedir/../lib/pecl
source $basedir/../lib/pecl_oci8

if [ "$PHP_BUILDPACK_NO_NODE" != "true" ] ; then
source $basedir/../lib/nodejs
Expand All @@ -23,54 +27,13 @@ fi

BUILD_DIR="$1"
CACHE_DIR="$2"
ENV_DIR="$3"

cd "$BUILD_DIR"
mkdir -p "$CACHE_DIR/package"

export APP_ENV=${APP_ENV:-prod}

function fetch_engine_package() {
local engine="$1"
local version="$2"
local location="$3"

local base_url="$PHP_BASE_URL"
if [ "$engine" = "nginx" ] ; then
base_url="$NGINX_BASE_URL"
fi

fetch_package "$base_url" "${engine}-${version}" "$location"
}

function fetch_package() {
local base_url="$1"
local package="$2"
local location="$3"

mkdir -p "$location"

local checksum_url="${base_url}/package/${package}.md5"
local package_url="${base_url}/package/${package}.tgz"
local checksum=$(curl --fail --retry 3 --retry-delay 2 --connect-timeout 3 --max-time 30 "$checksum_url" 2> /dev/null)
local cache_checksum=""

if [ -f "$CACHE_DIR/package/${package}.md5" ]; then
local cache_checksum=$(cat "$CACHE_DIR/package/${package}.md5")
fi

mkdir -p "$CACHE_DIR/package/$(dirname "$package")"

if [ "$cache_checksum" != "$checksum" ]; then
curl --fail --retry 3 --retry-delay 2 --connect-timeout 3 --max-time 30 "$package_url" -L -s > "$CACHE_DIR/package/${package}.tgz"
echo "$checksum" > "$CACHE_DIR/package/${package}.md5"
else
echo "Checksums match. Fetching from cache."
fi

mkdir -p "$location"
tar xzf "$CACHE_DIR/package/${package}.tgz" -C "$location"
}

function detect_framework() {
BUILD_DIR=$1
for f in "$basedir/../frameworks/"*; do
Expand Down Expand Up @@ -416,7 +379,7 @@ if [ "$PHP_BUILDPACK_NO_NODE" != "true" ] ; then
install_node_deps "$BUILD_DIR"
fi

install_composer_deps "$BUILD_DIR"
install_composer_deps "${BUILD_DIR}" "${CACHE_DIR}" "${ENV_DIR}"

# Detect PHP framework
# Set FRAMEWORK if not set in environment by user
Expand Down Expand Up @@ -474,6 +437,7 @@ mv /app/vendor/php vendor/php
[ -d "/app/vendor/libtidy" ] && mv /app/vendor/libtidy vendor/libtidy
[ -d "/app/vendor/libsodium" ] && mv /app/vendor/libsodium vendor/libsodium
[ -d "/app/vendor/libwebp" ] && mv /app/vendor/libwebp vendor/libwebp
[ -d "/app/vendor/oracle-client" ] && mv /app/vendor/oracle-client vendor/oracle-client

mkdir -p "bin" "vendor/bin"

Expand Down
28 changes: 28 additions & 0 deletions lib/apt
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
function apt_install() {
local apt_deps="${1}"
local build_dir="${2}"
local cache_dir="${3}"
local env_dir="${4}"

apt_manifest="$(mktemp "${build_dir}/Aptfile.php-XXX")"
echo "${apt_deps}" | tr ' ' '\n' > "${apt_manifest}"

apt_deps_buildpack_dir=$(mktemp /tmp/apt_buildpack_XXXX)
rm "${apt_deps_buildpack_dir}"

readonly apt_buildpack_url="https://github.com/Scalingo/apt-buildpack.git"
git clone --quiet --depth=1 "${apt_buildpack_url}" "${apt_deps_buildpack_dir}" >/dev/null 2>&1

readonly apt_log_file=$(mktemp /tmp/apt_log_XXXX.log)
APT_FILE_MANIFEST="$(basename ${apt_manifest})" "${apt_deps_buildpack_dir}/bin/compile" "${build_dir}" "${cache_dir}" "${env_dir}" >"${apt_log_file}" 2>&1
if [ $? -ne 0 ] ; then
tail -n 30 "${apt_log_file}" | indent
echo
warn "Fail to install apt packages $apt_deps"
echo
exit 1
fi

# Source new libs for future buildpack (update of LD_LIBRARY_PATH)
echo "${apt_deps_buildpack_dir}/export"
}
37 changes: 30 additions & 7 deletions lib/composer
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ function install_composer_deps() {
chmod a+x "$CACHE_DIR/composer.phar"
echo "$checksum" > $CACHE_DIR/composer.phar.md5
else
echo "Checksums match. Fetching from cache."
echo "Checksums match. Fetching Composer from cache." |indent
fi

cp "$CACHE_DIR/composer.phar" "$target/vendor/composer/bin/"
Expand All @@ -63,9 +63,25 @@ function install_composer_deps() {
if [ -n "$required_extensions" ]; then
status "Bundling additional extensions"
for ext in $required_extensions; do
echo "$ext" | indent
# TODO: Find a better way to ignore extensions which were not found in S3
if [ "$ext" = "memcached" ] ; then
local apt_deps=""
local ext_version=$(jq --raw-output ".require | .[\"ext-${ext}\"]" < "${BUILD_DIR}/composer.json")

if [ "$ext" = "oci8" ] ; then
apt_deps="libaio-dev"
fi

if [ -n "$apt_deps" ] ; then
echo "Installing dependencies for ${ext}: ${apt_deps}" | indent
install_env_file="$(apt_install "${apt_deps}" "${BUILD_DIR}" "${CACHE_DIR}" "${ENV_DIR}")"
# Source environment to export LD_LIBRARY_PATH
source "${install_env_file}"
rm "${install_env_file}"
fi

# Install third-party dependencies after ubuntu packages have been added
if [ "$ext" = "oci8" ] ; then
install_oracle_client_extension "${BUILD_DIR}" "${CACHE_DIR}"
elif [ "$ext" = "memcached" ] ; then
fetch_package "$PHP_BASE_URL" "libmemcached-${libmemcached_version}" "/app/vendor/libmemcached" | indent
elif [ "$ext" = "gmp" ] ; then
fetch_package "$PHP_BASE_URL" "gmp-${gmp_version}" "/app/vendor/gmp" | indent
Expand All @@ -77,12 +93,20 @@ function install_composer_deps() {
fi
fetch_package "$PHP_BASE_URL" "libsodium-${sodium_version}" "/app/vendor/libsodium" | indent
fi
fetch_package "$PHP_BASE_URL" "ext/$(php_api_version)/php-${ext}" "/app/vendor/php" 2>/dev/null || true | indent

local extension_package_path="ext/$(php_api_version)/php-${ext}"
package_found="$(has_package "${PHP_BASE_URL}" "${extension_package_path}")"
if [ "${package_found}" = "true" ] ; then
echo "Installing PHP extension: ${ext}" | indent
fetch_package "${PHP_BASE_URL}" "${extension_package_path}" "/app/vendor/php"
else
install_pecl_extension "${ext}" "${ext_version}" "${BUILD_DIR}" "$CACHE_DIR"
fi
done
fi

if [ -n "$COMPOSER_GITHUB_TOKEN" ]; then
status "Configuring the github authentication for Composer"
status "Configuring the GitHub authentication for Composer"
php "vendor/composer/bin/composer.phar" config -g github-oauth.github.com "$COMPOSER_GITHUB_TOKEN" --no-interaction
fi

Expand All @@ -101,4 +125,3 @@ function install_composer_deps() {
cd "$cwd"
} | indent
}

55 changes: 55 additions & 0 deletions lib/package
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
function fetch_engine_package() {
local engine="$1"
local version="$2"
local location="$3"

local base_url="$PHP_BASE_URL"
if [ "$engine" = "nginx" ] ; then
base_url="$NGINX_BASE_URL"
fi

fetch_package "$base_url" "${engine}-${version}" "$location"
}

function has_package() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

best practice: I'm not sure about what this function does. Maybe add a comment or rename it so that it's clearer? If I understand correctly it makes a HEAD request to our object storage to check if the given package exists, which means that we pre-compiled the extension, isn't it?

local base_url="${1}"
local package="${2}"

local package_url="${base_url}/package/${package}.tgz"

curl --output /dev/null --silent --head --location --fail "${package_url}"
if [ $? -eq 0 ] ; then
echo "true"
else
echo "false"
fi
}

function fetch_package() {
local base_url="$1"
local package="$2"
local location="$3"

mkdir -p "$location"

local checksum_url="${base_url}/package/${package}.md5"
local package_url="${base_url}/package/${package}.tgz"
local checksum=$(curl --fail --retry 3 --retry-delay 2 --connect-timeout 3 --max-time 30 "$checksum_url" 2> /dev/null)
local cache_checksum=""

if [ -f "$CACHE_DIR/package/${package}.md5" ]; then
local cache_checksum=$(cat "$CACHE_DIR/package/${package}.md5")
fi

mkdir -p "$CACHE_DIR/package/$(dirname "$package")"

if [ "$cache_checksum" != "$checksum" ]; then
curl --fail --retry 3 --retry-delay 2 --connect-timeout 3 --max-time 30 "$package_url" --location --silent > "$CACHE_DIR/package/${package}.tgz"
echo "$checksum" > "$CACHE_DIR/package/${package}.md5"
else
echo "Checksums match. Fetching from cache." | indent
fi

mkdir -p "$location"
tar xzf "$CACHE_DIR/package/${package}.tgz" --directory "$location"
}
87 changes: 87 additions & 0 deletions lib/pecl
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
function install_pecl_extension() {
local extension_name="${1}"
local version="${2}"
local cache_dir="${3}"

if [[ $version = '*' ]]; then
local version=$(curl --silent "https://pecl.php.net/rest/r/$extension_name/latest.txt")
fi
local ext_dir=/app/vendor/php/lib/php/extensions/no-debug-non-zts-$(php_api_version)

local cache_extension_file="${cache_dir}/${extension_name}-${version}.so"
if [ -f "${cache_extension_file}" ]; then
echo "Installing PECL extension ${extension_name} version ${version} from the cache" | indent
cp "${cache_extension_file}" "${ext_dir}/${extension_name}.so"
enable_pecl_extension ${extension_name}
return
fi

local build_dir=$(pwd)
local temp_dir=$(mktmpdir "pecl-extension")

echo "Installing PECL extension ${extension_name} version ${version}" | indent

pushd "${temp_dir}" > /dev/null

curl --silent --location "https://pecl.php.net/get/${extension_name}-${version}.tgz" | tar xz

pushd ${extension_name}-${version} > /dev/null
(
set +e
readonly phpize_log_file=$(mktemp "/tmp/pecl-phpize-${extension_name}-XXXX.log")
/app/vendor/php/bin/phpize > "${phpize_log_file}" 2>&1
[ $? -eq 0 ] || install_pecl_error "${extension_name}" "package" "${phpize_log_file}"

local configure_extension_args_var_name="PHP_PECL_EXTENSION_CONFIGURE_ARGS_$extension_name"
local configure_extension_args=$(printenv $configure_extension_args_var_name)
local flags=$(echo $configure_extension_args | sed "s|\$BUILD_DIR|$build_dir|")

if [[ $extension_name = 'oci8' ]]; then
flags="--with-oci8=instantclient,${ORACLE_HOME}"
fi

readonly configure_log_file=$(mktemp "/tmp/pecl-configure-${extension_name}-XXXX.log")
./configure --with-php-config=/app/vendor/php/bin/php-config $flags > "${configure_log_file}" 2>&1
[ $? -eq 0 ] || install_pecl_error "${extension_name}" "configure build" "${configure_log_file}"

readonly make_log_file=$(mktemp "/tmp/pecl-make-${extension_name}-XXXX.log")
make -j 2 > "${make_log_file}" 2>&1
[ $? -eq 0 ] || install_pecl_error "${extension_name}" "compile" "${make_log_file}"
)

cp modules/${extension_name}.so "${ext_dir}/${extension_name}.so"
cp modules/${extension_name}.so "${cache_extension_file}"
enable_pecl_extension ${extension_name}

popd > /dev/null
popd > /dev/null
}

function enable_pecl_extension() {
local extension_name="${1}"
local is_zend_extension_var_name="PHP_PECL_EXTENSION_IS_ZEND_$extension_name"
local is_zend_extension=$(printenv $is_zend_extension_var_name)

if [[ $is_zend_extension = "true" ]] ; then
echo "zend_extension=${extension_name}.so" > "/app/vendor/php/etc/conf.d/${extension_name}.ini"
else
echo "extension=${extension_name}.so" > "/app/vendor/php/etc/conf.d/${extension_name}.ini"
fi
}

function install_pecl_error() {
local extension_name="${1}"
local action="${2}"
local log_file="${3}"

echo
tail -n 30 "${log_file}" | indent
echo
# This sleep prevents from having the following lines mixed up with the output
# of the above tail. Mystery of shellscripting.
sleep 0.1
warn "Fail to ${action} of PHP PECL extension: ${extension_name}"
echo "Read above logs to understand the source of the issue" | indent
echo
exit 1
}
Loading