Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CI run for notebooks #300

Merged
merged 8 commits into from
Nov 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/notebooks_with_irdb_clone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,6 @@ jobs:
pip install .
pip install .[dev,test]
- name: Run notebooks
env:
TQDM_MININTERVAL: 10
run: ./runnotebooks.sh --checkout-irdb --delete
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ dependencies = [
"lxml>=4.5.0",
"pyyaml>5.1",
"more-itertools>=9.0",
"tqdm>=4.64",
"tqdm>=4.66.1",
"requests-cache>1.0",

"synphot>=1.1.0",
Expand Down
15 changes: 11 additions & 4 deletions runnotebooks.sh
Original file line number Diff line number Diff line change
@@ -1,41 +1,48 @@
#!/usr/bin/env bash

echo "# Running Notebooks Tests" >> $GITHUB_STEP_SUMMARY

if [[ "x${1}" == "x--clone-irdb" ]] ; then
# Cloning IRDB
if [[ ! -e irdb ]] ; then
echo "_Cloning IRDB_" >> $GITHUB_STEP_SUMMARY
git clone https://github.com/AstarVienna/irdb.git
fi

echo "## Symlinks" >> $GITHUB_STEP_SUMMARY
# https://github.com/koalaman/shellcheck/wiki/SC2044
find . -iname "*.ipynb" -printf '%h\0' | sort -z | uniq -z | while IFS= read -r -d '' dirnotebooks; do
echo "${dirnotebooks}"
echo "- ${dirnotebooks}" >> $GITHUB_STEP_SUMMARY
dirinstpkgs="${dirnotebooks}/inst_pkgs"
if [[ (! -e ./docs/source/examples/inst_pkgs) && (! -L ./docs/source/examples/inst_pkgs) ]] ; then
echo "Creating symlink to irdb: ${dirinstpkgs}"
echo "Creating symlink to irdb: ${dirinstpkgs}" >> $GITHUB_STEP_SUMMARY
ln -s irdb "${dirinstpkgs}"
else
echo "Directory exists, not creating symlink: ${dirinstpkgs}"
echo "Directory exists, not creating symlink: ${dirinstpkgs}" >> $GITHUB_STEP_SUMMARY
fi

# Comment out any download_package[s] in the notebooks.
pusd "${dirnotebooks}" || exit 1
pushd "${dirnotebooks}" || exit 1
sed -i -E 's|"(.*\.download_package)|"#\1|g' -- *.ipynb
popd || exit 1
done
fi


echo "## Notebooks tested" >> $GITHUB_STEP_SUMMARY
# https://github.com/koalaman/shellcheck/wiki/SC2044
find . -iname "*.ipynb" -print0 | while IFS= read -r -d '' fnnotebook
do
echo "Testing ${fnnotebook}"
echo "Testing ${fnnotebook} ..."
fnpy="${fnnotebook%.ipynb}.py"

# Convert .ipynb file to .py.
jupytext --to py "${fnnotebook}"

# Run the python script and quit on first error.
python "${fnpy}" || exit 1
echo "- ${fnnotebook}" >> $GITHUB_STEP_SUMMARY

# Delete generated files if --delete is specified.
# By default do not delete any files.
Expand Down