Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace singularity with apptainer #137

Merged
merged 9 commits into from
Jul 5, 2023
Merged

Replace singularity with apptainer #137

merged 9 commits into from
Jul 5, 2023

Conversation

sverhoeven
Copy link
Member

@sverhoeven sverhoeven commented Mar 6, 2023

  • Apptainer installation ships with symlink to /usr/bin/singularity, so creating models with BmiClientSingularity still works.
  • Depending on ewatercylce python package version a slightly different config file is generated.

TODO

Testable by

  1. spinning up a VM in SRC or with vagrant locally
  2. Login to JupyterHub
  3. Open a terminal
  4. Run apptainer run docker://ghcr.io/apptainer/lolcow, it should produce ascii art of a cow when apptainer is installed correctly.
  5. Run ls -l $(which singularity), singularity should be symbolic link to apptainer.
  6. Run pip show ewatercycle should have version 1.4.1 installed
  7. Verify /etc/ewatercycle.yaml has container_engine: apptainer.

Created role with `cd roles && molecule init role ewatercycle.apptainer`
Except where used in combination with python ewatercycle package
As eWaterCycle/ewatercycle#324 has not been merged/released yet
@@ -102,4 +101,11 @@ parameter_sets:
target_model: wflow
supported_model_versions: !!set {2020.1.1: null, 2020.1.2: null, 2020.1.3: null}
parameterset_dir: {{ parameterset_dir }}
singularity_dir: {{ singularity_image_root }}
{# ewatercycle>1.4.1 will use apptainer instead of singularity #}
{% if pyewatercycle_version == '1.4.1' %}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO make more future proof.
change so <1.4.1 uses singularity and else uses apptainer.

@sverhoeven
Copy link
Member Author

on SRC creation failed
TASK [Run Ansible plugin] ******************************************************
Monday 05 June 2023  11:27:46 +0000 (0:00:01.196)       0:04:07.036 *********** 
fatal: [145.38.187.131]: FAILED! => {"ansible_job_id": "495034654240.12475", "changed": true, "cmd": ["ansible-playbook", "--connection=local", "-b", "--extra-vars={'alt_home_location': '/data/volume_2', 'dcache_ro_token': '***', 'rclone_cache_dir': '/data/volume_3', 'rclone_max_gsize': '45'}", "/rsc/plugins/39795e81-c3e0-408b-9e80-a936c42048d1/research-cloud-plugin.yml"], "delta": "0:00:00.671938", "end": "2023-06-05 11:27:48.058763", "finished": 1, "msg": "non-zero return code", "rc": 2, "start": "2023-06-05 11:27:47.386825", "stderr": "[WARNING]: No inventory was parsed, only implicit localhost is available\n[WARNING]: provided hosts list is empty, only localhost is available. Note that\nthe implicit localhost does not match 'all'\n[WARNING]: Reset is not implemented for this connection", "stderr_lines": ["[WARNING]: No inventory was parsed, only implicit localhost is available", "[WARNING]: provided hosts list is empty, only localhost is available. Note that", "the implicit localhost does not match 'all'", "[WARNING]: Reset is not implemented for this connection"], "stdout": "\nPLAY [Install and configure eWaterCycle Jupyter] *******************************\n\nTASK [Wait for system to become reachable] *************************************\nok: [localhost]\n\nTASK [Gather facts for first time] *********************************************\nfatal: [localhost]: FAILED! => {\"changed\": false, \"module_stderr\": \"/bin/sh: 1: powershell: not found\\n\", \"module_stdout\": \"\", \"msg\": \"The module failed to execute correctly, you probably need to set the interpreter.\\nSee stdout/stderr for the exact error\", \"rc\": 127}\n\nPLAY RECAP *********************************************************************\nlocalhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   ", "stdout_lines": ["", "PLAY [Install and configure eWaterCycle Jupyter] *******************************", "", "TASK [Wait for system to become reachable] *************************************", "ok: [localhost]", "", "TASK [Gather facts for first time] *********************************************", "fatal: [localhost]: FAILED! => {\"changed\": false, \"module_stderr\": \"/bin/sh: 1: powershell: not found\\n\", \"module_stdout\": \"\", \"msg\": \"The module failed to execute correctly, you probably need to set the interpreter.\\nSee stdout/stderr for the exact error\", \"rc\": 127}", "", "PLAY RECAP *********************************************************************", "localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   "]}

@Peter9192
Copy link
Collaborator

vagrant@default:~$ apptainer run docker://ghcr.io/apptainer/lolcow
INFO:    Using cached SIF image
INFO:    underlay of /etc/localtime required more than 50 (77) bind mounts
 ______________________________
< Wed Jun 14 12:58:23 UTC 2023 >
 ------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

@Peter9192
Copy link
Collaborator

Apptainer works.
However when trying to run a generated notebook I get

ModuleNotFoundError                       Traceback (most recent call last)
File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:9
      8 try:
----> 9     import ESMF as esmpy  # noqa: N811
     10 except ImportError:

ModuleNotFoundError: No module named 'ESMF'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 import ewatercycle.parameter_sets
      2 import ewatercycle.forcing
      3 import ewatercycle.models

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/__init__.py:1
----> 1 from .config import CFG
      2 from .version import __version__

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/config/__init__.py:87
      1 """Config
      2 ******
      3 
   (...)
     84     wflow.docker_images: ewatercycle/wflow-grpc4bmi:2020.1.1
     85 """
---> 87 from ._config_object import CFG, DEFAULT_CONFIG, SYSTEM_CONFIG, USER_HOME_CONFIG, Config
     89 __all__ = ["CFG", "Config", "DEFAULT_CONFIG", "SYSTEM_CONFIG", "USER_HOME_CONFIG"]

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/config/_config_object.py:11
      7 from typing import Optional, TextIO, Union
      9 from ruamel.yaml import YAML
---> 11 from ewatercycle.util import to_absolute_path
     13 from ._validated_config import ValidatedConfig
     14 from ._validators import _validators

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/util.py:9
      7 import xarray as xr
      8 from dateutil.parser import parse
----> 9 from esmvalcore.experimental.recipe_output import RecipeOutput
     10 from shapely import geometry
     13 def find_closest_point(
     14     grid_longitudes: Iterable[float],
     15     grid_latitudes: Iterable[float],
     16     point_longitude: float,
     17     point_latitude: float,
     18 ) -> Tuple[int, int]:

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/experimental/__init__.py:9
      6 from esmvalcore.config import CFG
      8 from ._warnings import warnings  # prints experimental API warning
----> 9 from .recipe import Recipe
     10 from .utils import RecipeList, get_all_recipes, get_recipe
     12 logging.basicConfig(format='%(message)s',
     13                     level=logging.INFO,
     14                     stream=sys.stdout)

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/experimental/recipe.py:12
      8 from typing import Dict, Optional
     10 import yaml
---> 12 from esmvalcore._recipe.recipe import Recipe as RecipeEngine
     13 from esmvalcore.config import CFG, Session
     15 from ._logging import log_to_dir

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/_recipe/recipe.py:24
     22 from esmvalcore.config._config import TASKSEP, get_project_config
     23 from esmvalcore.config._diagnostics import TAGS
---> 24 from esmvalcore.dataset import Dataset
     25 from esmvalcore.exceptions import (
     26     ESMValCoreDeprecationWarning,
     27     InputFilesNotFound,
     28     RecipeError,
     29 )
     30 from esmvalcore.local import (
     31     _dates_to_timerange,
     32     _get_multiproduct_filename,
   (...)
     35     _truncate_dates,
     36 )

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/dataset.py:18
     15 from iris.cube import Cube
     17 from esmvalcore import esgf, local
---> 18 from esmvalcore._recipe import check
     19 from esmvalcore._recipe.from_datasets import datasets_to_recipe
     20 from esmvalcore.cmor.table import _get_mips, _update_cmor_facets

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/_recipe/check.py:17
     15 from esmvalcore.exceptions import InputFilesNotFound, RecipeError
     16 from esmvalcore.local import _get_start_end_year, _parse_period
---> 17 from esmvalcore.preprocessor import TIME_PREPROCESSORS, PreprocessingTask
     18 from esmvalcore.preprocessor._multimodel import STATISTIC_MAPPING
     19 from esmvalcore.preprocessor._supplementary_vars import (
     20     PREPROCESSOR_SUPPLEMENTARIES,
     21 )

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py:27
     25 from ._bias import bias
     26 from ._cycles import amplitude
---> 27 from ._derive import derive
     28 from ._detrend import detrend
     29 from ._io import (
     30     _get_debug_filename,
     31     cleanup,
   (...)
     35     write_metadata,
     36 )

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_derive/__init__.py:32
     28         derivers[short_name] = getattr(module, 'DerivedVariable')
     29     return derivers
---> 32 ALL_DERIVED_VARIABLES = _get_all_derived_variables()
     34 __all__ = list(ALL_DERIVED_VARIABLES)
     37 def get_required(short_name, project):

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_derive/__init__.py:26, in _get_all_derived_variables()
     24 for path in Path(__file__).parent.glob('[a-z]*.py'):
     25     short_name = path.stem
---> 26     module = importlib.import_module(
     27         f'esmvalcore.preprocessor._derive.{short_name}')
     28     derivers[short_name] = getattr(module, 'DerivedVariable')
     29 return derivers

File /opt/conda/envs/ewatercycle/lib/python3.11/importlib/__init__.py:126, in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_derive/sispeed.py:6
      3 import logging
      4 from iris import Constraint
----> 6 from .._regrid import regrid
      8 from ._baseclass import DerivedVariableBase
     10 logger = logging.getLogger(__name__)

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid.py:24
     22 from ..cmor._fixes.shared import add_altitude_from_plev, add_plev_from_altitude
     23 from ..cmor.table import CMOR_TABLES
---> 24 from ._regrid_esmpy import ESMF_REGRID_METHODS
     25 from ._regrid_esmpy import regrid as esmpy_regrid
     26 from ._supplementary_vars import add_ancillary_variable, add_cell_measure

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:11
      9         import ESMF as esmpy  # noqa: N811
     10     except ImportError:
---> 11         raise exc
     12 import iris
     13 import numpy as np

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:5
      2 """Provides regridding for irregular grids."""
      4 try:
----> 5     import esmpy
      6 except ImportError as exc:
      7     # Prior to v8.4.0, `esmpy`` could be imported as `ESMF`.
      8     try:

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/__init__.py:112
    108 __obsoletes__ = msg["obsoletes"]
    110 #### IMPORT LIBRARIES #########################################################
--> 112 from esmpy.api.esmpymanager import *
    113 from esmpy.api.grid import *
    114 from esmpy.api.mesh import *

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/api/esmpymanager.py:9
      3 """
      4 The ESMPyManager API
      5 """
      7 #### IMPORT LIBRARIES #########################################################
----> 9 from esmpy.interface.cbindings import *
     11 from esmpy.api.constants import *
     12 from esmpy.util.exceptions import *

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/interface/cbindings.py:13
     11 import esmpy.api.constants as constants
     12 from esmpy.util.decorators import *
---> 13 from esmpy.interface.loadESMF import _ESMF
     16 def copy_struct(src):
     17     dst = type(src)()

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/interface/loadESMF.py:28
     26     esmfmk = os.environ["ESMFMKFILE"]
     27 except:
---> 28     raise ImportError('The ESMFMKFILE environment variable is not available.')
     30 #### INVESTIGATE esmf.mk ######################################################
     31 
     32 # TODO: look for various dependecies in the ESMF build log
   (...)
     37 #       use this information to set variables that can be checked at beginning
     38 #       of the routines that require an ESMF build with these dependencies
     40 with open(esmfmk, 'r') as MKFILE:
     41     
     42     # investigate esmf.mk


ImportError: The ESMFMKFILE environment variable is not available.

@Peter9192
Copy link
Collaborator

Lot's of issues on ESMValCore and Iris about this, e.g.

ESMValGroup/ESMValCore#1876
SciTools-incubator/iris-esmf-regrid#231

Copy link
Collaborator

@Peter9192 Peter9192 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving as this fixes the issue it was intended for. Shall we open a new issue for the ESMValTool fix?

@sverhoeven sverhoeven merged commit 9276164 into main Jul 5, 2023
0 of 2 checks passed
@sverhoeven sverhoeven deleted the apptainer branch March 6, 2024 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants