Feature/asv workflow #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

prasad-sawantdesai wants to merge 38 commits into iterorganization:develop from prasad-sawantdesai:feature/asv_workflow

.github/workflows/asv.yml

-Original file line number
+Diff line change
@@ -0,0 +1,136 @@
+    name: ASV Benchmarks
+    on:
+      push:
+        branches: [main, develop]
+      pull_request:
+        types: [opened, synchronize, reopened]
+      workflow_dispatch:
+    jobs:
+      benchmark:
+        runs-on: ubuntu-22.04
+        permissions:
+          contents: write       # needed to push HTML report to gh-pages branch
+        steps:
+          - name: Checkout code
+            uses: actions/checkout@v4
+            with:
+              fetch-depth: 0
+          - name: Set up Python
+            uses: actions/setup-python@v5
+            with:
+              python-version: '3.11'
+          - name: Display Python version
+            run: python -c "import sys; print(sys.version)"
+          - name: Install dependencies
+            working-directory: backend
+            run: |
+              python -m pip install --upgrade pip
+              # virtualenv required by asv to create per-commit benchmark environments
+              pip install asv virtualenv
+          - name: Download test data from Zenodo
+            run: |
+              mkdir -p test_data
+              # iter_scenario_53298_seq1_DD4: ~14 MB, DD 4.0.0, contains core_profiles + equilibrium
+              # (required by all benchmark node paths). The 123364 file is a SOLPS edge
+              # simulation with no core_profiles/equilibrium so is not used for benchmarks.
+              wget -q -O test_data/iter_scenario_53298_seq1_DD4.nc \
+                "https://zenodo.org/records/17062700/files/iter_scenario_53298_seq1_DD4.nc?download=1"
+              echo "Downloaded files:"
+              ls -lh test_data/
+          - name: Configure ASV regression threshold for CI
+            run: |
+              # On GitHub Actions: only flag regressions >2x slower in the HTML report,
+              # matching the --factor 2.0 used in asv continuous below.
+              # On Bamboo/local, asv.conf.json is used as-is (default ASV threshold of 5%).
+              python - <<'EOF'
+              import pathlib
+              conf_path = pathlib.Path('asv.conf.json')
+              conf = conf_path.read_text()
+              # Insert threshold entry before the final closing brace
+              assert conf.rstrip().endswith('}'), "Unexpected asv.conf.json ending"
+              patched = conf.rstrip()[:-1].rstrip().rstrip(',') + ',\n    "regressions_thresholds": {".*": 2.0}\n}\n'
+              conf_path.write_text(patched)
+              print("CI: regressions_thresholds set to 2.0")
+              EOF
+          - name: Run ASV continuous benchmarks
+            id: asv
+            # continue-on-error so we can post the PR comment before re-failing the job
+            continue-on-error: true
+            env:
+              # Tells benchmarks/__init__.py where the test data lives
+              IBEX_TEST_DATA_DIR: ${{ github.workspace }}/test_data
+            run: |
+              asv machine --yes
+              if [ "${{ github.event_name }}" = "pull_request" ]; then
+                # Compare base branch vs PR head — both on the same runner, same hardware
+                BASELINE="origin/${{ github.base_ref }}"
+                CONTENDER="${{ github.sha }}"
+              else
+                # Compare previous commit vs current commit on push/manual trigger
+                # Guard: skip if there is no previous commit (e.g. initial branch push)
+                if ! git rev-parse HEAD~1 >/dev/null 2>&1; then
+                  echo "No previous commit to compare against — skipping benchmark comparison."
+                  exit 0
+                fi
+                BASELINE="HEAD~1"
+                CONTENDER="HEAD"
+              fi
+              # Resolve to full SHAs for clarity in the log
+              BASELINE_SHA=$(git rev-parse "${BASELINE}" 2>/dev/null || echo "${BASELINE}")
+              CONTENDER_SHA=$(git rev-parse "${CONTENDER}" 2>/dev/null || echo "${CONTENDER}")
+              echo "Comparing commits:"
+              echo "  BASELINE  : ${BASELINE} → ${BASELINE_SHA}"
+              echo "  CONTENDER : ${CONTENDER} → ${CONTENDER_SHA}"
+              # --factor 2.0: only flag as regression if a benchmark is >2x slower
+              # (filters out GitHub runner hardware noise, ~20-50% typical variance)
+              asv continuous --quick --show-stderr --factor 2.0 "${BASELINE_SHA}" "${CONTENDER_SHA}" \
+>&1 | tee asv_output.txt
+              # Print a direct before/after comparison table (ratio per benchmark).
+              # This makes it easy to see which benchmarks changed and by how much.
+              echo ""
+              echo "=== Before vs After comparison ==="
+              asv compare --factor 2.0 --split "${BASELINE_SHA}" "${CONTENDER_SHA}" 2>&1 | tee -a asv_output.txt
+          - name: Fail job if regression detected
+            if: steps.asv.outcome == 'failure'
+            run: |
+              echo "FAILED: Performance regression detected (>2x threshold). See benchmark output above."
+              exit 1
+          - name: Generate HTML results
+            # Only meaningful on push to develop/main — asv publish builds the branch
+            # timeline, so PR commits (not yet on any branch) are simply skipped with
+            # "Couldn't find <sha> in branches". Run only after a merge to avoid noise.
+            if: github.event_name != 'pull_request'
+            run: |
+              # asv publish resolves branch names from asv.conf.json via git rev-list.
+              # On PR checkouts only the PR branch is a local ref. Using
+              # refspec <remote>:<local> creates local branches directly so that
+              # git rev-list develop / main succeeds (plain fetch only makes
+              # remotes/origin/develop which asv does not resolve).
+              git fetch origin develop:develop 2>/dev/null || true
+              git fetch origin main:main 2>/dev/null || true
+              asv publish
+          - name: Deploy HTML report to GitHub Pages
+            # Only run on push to develop/main, same reasoning as Generate HTML above.
+            if: github.event_name != 'pull_request'
+            uses: peaceiris/actions-gh-pages@v3
+            with:
+              github_token: ${{ secrets.GITHUB_TOKEN }}
+              publish_dir: .asv/html
+              destination_dir: benchmarks
+              keep_files: false

.readthedocs.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,17 @@
+    version: 2
+    build:
+      os: ubuntu-24.04
+      tools:
+        python: "3.11"
+    sphinx:
+      configuration: docs/source/conf.py
+      fail_on_warning: true
+    python:
+      install:
+        - method: pip
+          path: backend
+          extra_requirements:
+            - docs

README.md

-Original file line number
+Diff line change
@@ Expand Up @@
     > [!NOTE]
     > This project is under active development, important changes may occur including in the backend endpoint API.
+    **Documentation:** https://imas-ibex.readthedocs.io/
     [Frontend readme](frontend/README.md)
@@ Expand Down @@

asv.conf.json

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -7,7 +7,7 @@
  
        "project": "IBEX",

        // The project's homepage

        "project_url": "https://git.iter.org/projects/IMEX/repos/ibex",

        "project_url": "https://github.com/iterorganization/IBEX",

        // The URL or local path of the source code repository for the

        // project being benchmarked

    @@ -20,10 +20,10 @@
  
        // Customizable commands for building the project.

        // See asv.conf.json documentation.

        // To build the package using pyproject.toml (PEP518), uncomment the following lines

           // To build the package using pyproject.toml (PEP518), uncomment the following lines

        "build_command": [

            "python -m pip install build",

            "python -m build --wheel -o {build_cache_dir} {build_dir}",

            "python -m pip install build setuptools>=61 setuptools_scm[toml]>=6.2 versioneer[toml] wheel",

            "python -m build --no-isolation --wheel -o {build_cache_dir} {build_dir}",

            //"python -m pip wheel --no-deps -w {build_cache_dir} {build_dir}"

        ],

        // Customizable commands for installing and uninstalling the project.

backend/ci/build_docs.sh

-Original file line number
+Diff line change
@@ Expand Up / @@ -19,9 +19,8 @@ python -m venv venv @@
     echo "PWD: " `pwd`
     # PREPARE THE ENVIRONMENT
-    pip install --upgrade pip setuptools wheel build pytest-cov
-    pip install --upgrade sphinx sphinx-autosummary-accessors sphinx_immaterial
+    pip install --upgrade pip setuptools wheel
     pip install --upgrade .[docs]
-    # RUN PYTEST
+    # BUILD DOCS
     make -C ../docs html

backend/ci/configure_env.sh

-Original file line number
+Diff line change
@@ Expand Up / @@ -7,7 +7,8 @@ source /etc/profile.d/modules.sh @@
     module purge
     # Set up environment
-    module load IDStools/2.1.0-intel-2023b IMAS-Python/2.0.0-intel-2023b
+    module load Python
+    module load IMAS-Core
     # Debuggging:
     echo "Done loading modules"

backend/ci/run_benchmarks.sh

-Original file line number
+Diff line change
@@ -1,21 +1,26 @@
     #!/bin/bash
-    # Bamboo CI script for linting
+    # Bamboo CI script for benchmarks
+    set -e -o pipefail
-    # Fetch develop and main for `ASV compare`
-    git fetch origin develop
-    git fetch origin main
+    # Always fetch develop/main from the canonical upstream repo, not from origin
+    UPSTREAM="https://github.com/iterorganization/IBEX"
+    git remote remove upstream 2>/dev/null || true
+    git remote add upstream "${UPSTREAM}"
+    git fetch upstream
-    # Debuggging:
-    set -e -o pipefail
+    git checkout -B develop upstream/develop
+    git checkout -B main upstream/main
+    # Go back to the triggering branch
+    git checkout "${bamboo_planRepository_branch}"
     # Set up environment s
     BACKEND_ROOT_DIR=$(realpath "$(dirname "$(realpath "${BASH_SOURCE[0]}")")/..")
     source ${BACKEND_ROOT_DIR}/ci/configure_env.sh
     BENCHMARKS_DIR=$(realpath "$PWD/ibex_benchmarks")
     if [[ "$(uname -n)" == *"bamboo"* ]]; then
-        set -e -o pipefail
         # create
         BENCHMARKS_DIR=$(realpath "/mnt/bamboo_deploy/ibex/benchmarks/")
     fi
@@ Expand Down Expand Up / @@ -47,8 +52,8 @@ asv run --skip-existing-successful develop^! @@
     asv run --skip-existing-successful main^!
     # Compare results
-    if [ `git rev-parse --abbrev-ref HEAD` == develop ]
-    then
+    CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
+    if [ "${CURRENT_BRANCH}" = "develop" ]; then
         asv compare main develop --machine $(hostname) || echo "asv compare failed"
     else
         asv compare develop HEAD --machine $(hostname) || echo "asv compare failed"
@@ Expand Down @@

backend/ci/run_pytest.sh

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -19,8 +19,8 @@ python -m venv venv
  
    echo "PWD: " `pwd`

    # PREPARE THE ENVIRONMENT

    pip install --upgrade pip setuptools wheel pytest-cov

    pip install --upgrade .

    pip install --upgrade pip setuptools wheel

    pip install --upgrade .[test]

    # RUN PYTEST

    python -m pytest -n=auto --cov=ibex --cov-report=term-missing --cov-report=html --junit-xml=test-reports/pytest-report.xml ${BACKEND_ROOT_DIR}/tests
      
    python -m pytest -n auto --cov=ibex --cov-report=term-missing --cov-report=html --junit-xml=test-reports/pytest-report.xml ${BACKEND_ROOT_DIR}/tests

backend/ibex/core/ibex_service.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -34,7 +34,6 @@ class IMAS_URI: @@
         def __init__(self, full_uri):
             """
             IMAS_URI constructor
-            :param full_uri: pulsefile uri along with #fragment part
             """
             self.full_uri = full_uri
@@ Expand Down @@

backend/ibex/data_source/imas_python_source.py

-Original file line number
+Diff line change
@@ Expand Up @@
                 return ids_root
             except imas.exception.IDSNameError as e:
                 raise IdsNotFoundException(e) from None
+            except imas.exception.DataEntryException as e:
+                raise IdsNotFoundException(e) from None
         def data_entry_exists(self, uri: str) -> bool:
             """
@@ Expand Down @@

backend/pyproject.toml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -13,7 +13,7 @@ authors = [
  
    description = "Data visualization for IMAS IDS data"

    readme = {file = "README.md", content-type = "text/markdown"}

    requires-python = ">=3.7"

    license-files = ["LICENSE.txt"]

    license = {text = "LGPL-3.0-only"}

    classifiers = [

        "Development Status :: 3 - Alpha",

        "Environment :: Console",

    @@ -39,12 +39,10 @@ classifiers = [
  
        "Topic :: Scientific/Engineering",

        "Topic :: Scientific/Engineering :: Physics",

        "Topic :: Software Development :: Libraries",

        "Topic :: Software Development :: Libraries :: Python Modules",

        "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)"

        "Topic :: Software Development :: Libraries :: Python Modules"

    ]

    dynamic = ["version"]

    dependencies = [

        "imas-python",

        "imas-idstools",

        "click",

        "uvicorn",

    @@ -65,15 +63,17 @@ all = [
  
    test = [

        "pytest",

        "pytest-cov",

        "pytest-xdist",

    ]

    linting = [

        "ruff",

    ]

    docs =[

        "sphinx",

        "sphinx-autosummary-accessors",

        "sphinx>=7,<9",

        "sphinx-autosummary-accessors==2025.3.1",

        "sphinx_immaterial",

    ]

    @@ -84,6 +84,10 @@ benchmark = [
  
    [project.urls]

    homepage = "https://github.com/iterorganization/IBEX"

    [tool.setuptools]

    # license-files must be here, not in [project], for setuptools <77 compatibility

    license-files = ["LICENSE.txt"]

    [tool.setuptools.packages.find]

    where = ["."]

    include = ["ibex*"]

benchmarks/__init__.py

-Original file line number
+Diff line change
@@ -1,6 +1,20 @@
+    import os
+    from pathlib import Path
     uris_label = ["pulsefile uri"]
-    uris = [
-        "imas:mdsplus?user=public;database=ITER;pulse=134173;run=106;version=3",  # 871 slices
-        "imas:hdf5?user=public;database=ITER;pulse=134173;run=106;version=3",  # 871 slices
-        "imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/105070/2",  # 4183 slices
-    ]
+    if os.environ.get("GITHUB_ACTIONS") == "true":
+        # On GitHub Actions: use publicly available Zenodo datasets (CC-BY 4.0)
+        # https://zenodo.org/records/17062700
+        # Downloaded by the workflow into IBEX_TEST_DATA_DIR before benchmarks run.
+        _data_dir = Path(os.environ["IBEX_TEST_DATA_DIR"])
+        uris = [
+            str(_data_dir / "iter_scenario_53298_seq1_DD4.nc"),  # ~14 MB, DD 4.0.0, core_profiles+equilibrium
+        ]
+    else:
+        # Local / ITER infrastructure: use internal IMAS data sources directly
+        uris = [
+            "imas:mdsplus?user=public;database=ITER;pulse=134173;run=106;version=3",  # 871 slices
+            "imas:hdf5?user=public;database=ITER;pulse=134173;run=106;version=3",     # 871 slices
+            "imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/105070/2",               # 4183 slices
+        ]

benchmarks/endpoints_data.py

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -12,7 +12,7 @@ def setup(self, *args):
  
            self.test_client = TestClient(app)

        def time_field_value(self, uri, node_path):

            parameters = {"uri": f"{uri}/{node_path}"}

            parameters = {"uri": f"{uri}{node_path}"}

            self.test_client.get("/data/field_value", params=parameters)

        time_field_value.param_names = param_names + ["node path"]

    @@ -21,13 +21,13 @@ def time_field_value(self, uri, node_path):
  
            [

                "#core_profiles:0/time",  # LEAF

                "#core_profiles:0/profiles_1d[0]/t_i_average",  # LEAF IN AoS,

                "#core_profiles:0/profiles_1d[0:100]/t_i_average",  # LEAF IN AoS SLICE,

                "#core_profiles:0/profiles_1d[:]/t_i_average",  # LEAF IN AoS SLICE,

                "#equilibrium:0/time_slice[:]/profiles_2d[:]/psi",  # 2D QUANTITY,

            ],

        )

        def time_plot_data(self, uri, node_path):

            parameters = {"uri": f"{uri}/{node_path}"}

            parameters = {"uri": f"{uri}{node_path}"}

            self.test_client.get("/data/plot_data", params=parameters)

        time_plot_data.param_names = param_names + ["node path"]

    @@ -36,7 +36,7 @@ def time_plot_data(self, uri, node_path):
  
            [

                "#core_profiles:0/time",  # LEAF

                "#core_profiles:0/profiles_1d[0]/t_i_average",  # LEAF IN AoS,

                "#core_profiles:0/profiles_1d[0:100]/t_i_average",  # LEAF IN AoS SLICE,

                "#core_profiles:0/profiles_1d[:]/t_i_average",  # LEAF IN AoS SLICE,

                "#equilibrium:0/time_slice[:]/profiles_2d[:]/psi",  # 2D QUANTITY,

            ],

        )

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/asv workflow #41

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Feature/asv workflow #41

Are you sure you want to change the base?

Uh oh!

Feature/asv workflow #41

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!