Singularity as executable provider

[vsoch]

Hi @douglasjacobsen! It looks like Singularity is hiding in ramble as (what I'm guessing is) a spack containerize output? I'm wondering if we might consider having a container provider that knows how to provision a container. E.g., right now I'm going to provide a container binary directly to internals->custom executables, but I wonder if there would be an easy way to define a URI alongside the config, and then have ramble check for Singularity and do the pull of the binary (just to the Singularity cache would be fine) if it doesn't exist.

I'm good just pulling on my own for now and then defining as a custom executable, but wanted to get your thoughts. For some context, I'm working with Olga to see if we can add some apps from our performance study, all of which were run with containers (not spack). Thanks, and happy Monday!

[douglasjacobsen]

Hey @vsoch!

We actually have modifiers to make workflows around using containers easier. See the apptainer modifier here: https://github.com/GoogleCloudPlatform/ramble/blob/develop/var/ramble/repos/builtin/modifiers/apptainer/modifier.py

As an example, if you wanted to run NeMo with containers, you could use something like the following config:

ramble:
  variants:
    package_manager: spack
  modifiers:
  - name: apptainer
  env_vars:
    set:
      OMP_NUM_THREADS: '{n_threads}'
  variables:
    mpi_command: ''
    container_uri: docker://nvcr.io/nvidia/nemo:{nemo_version}
    container_name: nemo-{nemo_version}
    batch_submit: '{execute_experiment}'
    processes_per_node: 1
    gpus_per_node: 1
    apptainer_run_args: --nv --bind {container_mounts} --writable-tmpfs 
  applications:
    py-nemo:
      workloads:
        pretraining:
          experiments:
            test:
              variables:
                env_name: apptainer
                nemo_stage: training
                nemo_model: gpt3
                nemo_config_name: 5b
                n_nodes: 1
               
                nemo_launcher_tag: 24.07
                nemo_version: 24.07

                exp_manager.checkpoint_callback_params.model_parallel_size: 1

                max_steps: 5
                model.optim.grad_sync_dtype: bf16
                model.data.data_impl: mock
                model.data.data_prefix: []
                trainer.max_steps: '{max_steps}'
                trainer.val_check_interval: '{max_steps}'
                trainer.limit_val_batches: '0.0'
                exp_manager.create_checkpoint_callback: False
                model.optim.name: distributed_fused_adam
                exp_manager.exp_dir: '{experiment_run_dir}'
  software:
    packages:
      apptainer:
        pkg_spec: apptainer
    environments:
      py-nemo:
        packages:
        - apptainer

Here, spack is only used to manage the apptainer install. If apptainer is installed outside of spack, and always available in your path then you can remove the software definitions and the varitants block.

[vsoch]

That's perfect! I should have thought to search for "apptainer" but I only did "singularity"... the name "apptainer" is just so generic (and it's still singularity) so I always forget. 😆

I'll test this out today, thanks @douglasjacobsen !

[douglasjacobsen]

No problem! I have other examples of using this with NVIDIA HPL and HPCG too if that's helpful. But, generally if you disable the variants:package_manager block, you'll not use spack for software. And then the apptainer modifier will pull the container for you, and set up some generic variable definitions (like {container_mounts} and {container_env_vars}) for use in the workflow.