Skip to content

chore(ci): skip packages with unsupported pre-release deps in import profiler #18033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hebaalazzeh merged 4 commits into from
Aug 8, 2026
+60 −11
Merged

chore(ci): skip packages with unsupported pre-release deps in import profiler #18033

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
55fb37a
chore(ci): skip packages with unsupported pre-release deps in import …
hebaalazzeh Aug 7, 2026
8165e12
add associated bug
hebaalazzeh Aug 7, 2026
6abdd0d
address feedback
hebaalazzeh Aug 7, 2026
cd2f6ca
address feedback
hebaalazzeh Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 60 additions & 11 deletions ci/run_single_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,13 +98,35 @@ case ${TEST_TYPE} in
;;
import_profile)
if [ -f setup.py ] || [ -f pyproject.toml ]; then
PACKAGE_NAME=$(basename $(pwd))

# TODO(https://github.com/googleapis/google-cloud-python/issues/18035):
# Remove this skip once Python 3.15 is officially released and upstream binary wheels
# (e.g. numpy, pyarrow, pandas, geopandas, pikepdf) are published on PyPI.
# Packages with heavy C/Rust dependencies attempt full source compilation on pre-release Python,
# taking 5-10+ minutes before failing due to unreleased CPython 3.15 C-API changes.
if [[ "${PY_VERSION}" == "3.15"* ]]; then
UNSUPPORTED_PRE_RELEASE_PACKAGES=(
"bigframes"
"pandas-gbq"
"google-cloud-documentai-toolbox"
"db-dtypes"
"bigquery-magics"
)
for unsupported in "${UNSUPPORTED_PRE_RELEASE_PACKAGES[@]}"; do
if [ "${PACKAGE_NAME}" = "${unsupported}" ]; then
echo "WARNING: Skipping import_profile for ${PACKAGE_NAME}: package has heavy C/Rust dependencies not yet supported on pre-release Python ${PY_VERSION}."
exit 0
fi
done
fi

echo "Creating temporary virtualenv for import profile..."
python3 -m venv .venv-profiler
source .venv-profiler/bin/activate
export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
python -m pip install --upgrade pip setuptools

PACKAGE_NAME=$(basename $(pwd))
PROFILER_TEMP_DIR=$(mktemp -d)
cp ../../scripts/import_profiler/profiler.py "${PROFILER_TEMP_DIR}/profiler.py"
PROFILER_SCRIPT="${PROFILER_TEMP_DIR}/profiler.py"
Expand All @@ -121,13 +143,26 @@ case ${TEST_TYPE} in
WORKTREE_DIR=$(mktemp -d)
rmdir "${WORKTREE_DIR}"
if git worktree add "${WORKTREE_DIR}" "${BASELINE_COMMIT}" 2>/dev/null; then
(
if ! (
cd "${WORKTREE_DIR}/${REPO_PREFIX}"
if [ -f setup.py ] || [ -f pyproject.toml ]; then
pip install -e .
python "${PROFILER_SCRIPT}" --package "${PACKAGE_NAME}" --iterations 11 --csv "${BASELINE_CSV}"
if pip install -e . ; then
echo "INFO: Successfully installed baseline dependencies for ${PACKAGE_NAME}."
python "${PROFILER_SCRIPT}" --package "${PACKAGE_NAME}" --iterations 11 --csv "${BASELINE_CSV}"
if [ $? -eq 0 ]; then
echo "INFO: Successfully ran baseline profiler for ${PACKAGE_NAME}."
fi
elif [[ "${PY_VERSION}" != "3.15"* ]]; then
exit 1
fi
Comment thread
hebaalazzeh marked this conversation as resolved.
Comment on lines +155 to +157

@ohmayr ohmayr Aug 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: exit 1 runs inside a subshell (...), so it will only exit the child process rather than failing the outer script if baseline pip install fails on non-3.15 Python versions (due to set +e).

Consider checking the subshell exit status in the parent shell or restructuring to if ! ( ... ); then exit 1; fi to ensure baseline build failures properly fail CI.

@hebaalazzeh hebaalazzeh Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed

fi
)
); then
git worktree remove -f "${WORKTREE_DIR}"
deactivate
rm -rf .venv-profiler
rm -rf "${PROFILER_TEMP_DIR}"
exit 1
fi
git worktree remove -f "${WORKTREE_DIR}"
else
echo "Failed to create git worktree for baseline. Skipping baseline generation."
Expand All @@ -137,14 +172,28 @@ case ${TEST_TYPE} in
fi
fi

pip install -e .

if [ -f "${BASELINE_CSV}" ]; then
python ${PROFILER_SCRIPT} --package ${PACKAGE_NAME} --iterations 11 --fail-threshold 5000 --diff-baseline "${BASELINE_CSV}" --diff-threshold 100
# TODO(https://github.com/googleapis/google-cloud-python/issues/18035):
# Clean up this fallback once Python 3.15 is officially released and upstream binary wheels are available on PyPI.
# On pre-release Python versions, packages with complex C/Rust dependencies (e.g. bigframes) fail during pip install due to missing pre-built wheels.
if ! pip install -e . ; then
if [[ "${PY_VERSION}" == "3.15"* ]]; then
echo "WARNING: Could not install dependencies for ${PACKAGE_NAME} on Python ${PY_VERSION} (missing pre-built binary wheels for pre-release Python). Skipping import_profile."
retval=0
else
retval=1
fi
else
python ${PROFILER_SCRIPT} --package ${PACKAGE_NAME} --iterations 11 --fail-threshold 5000
echo "INFO: Successfully installed dependencies for ${PACKAGE_NAME} on Python ${PY_VERSION}."
if [ -f "${BASELINE_CSV}" ]; then
python ${PROFILER_SCRIPT} --package ${PACKAGE_NAME} --iterations 11 --fail-threshold 5000 --diff-baseline "${BASELINE_CSV}" --diff-threshold 100
else
python ${PROFILER_SCRIPT} --package ${PACKAGE_NAME} --iterations 11 --fail-threshold 5000
fi
retval=$?
if [ $retval -eq 0 ]; then
echo "INFO: Successfully completed import_profile for ${PACKAGE_NAME}."
fi
fi
Comment thread
hebaalazzeh marked this conversation as resolved.
retval=$?
deactivate
rm -rf .venv-profiler
rm -rf "${PROFILER_TEMP_DIR}"
Expand Down
Loading