v1.1.0

b2luigi v1.1.0

Summary

The latest b2luigi updates introduce several enhancements. The new XrootDTarget feature simplifies grid file management by using XRootD-based targets. Apptainer support allows tasks to run in containers both locally and in batch modes. Batch system discovery is now automated when set batch_system="auto". Task-specific propagation of gbasf2 settings is supported for proxy groups, project paths, and proxy lifetimes. The removal of the git_hash parameter from Basf2Task ensures stable task IDs and output paths, with the option to reintroduce it manually if needed. Finally, the new B2LUIGI_SETTINGS_JSON environment variable allows users to specify a custom path for the settings.json file, providing more flexibility in configuration management.

Changes

• XrootDTarget

  b2luigi simplifies working with grid files by offering XRootD-based targets. The XRootD client, pre-installed in basf2 environments on CVMFS, requires a valid VOMS proxy for access. To use this feature, override the _get_output_file_target function in your task and set up a XrootDSystem object with the server address (root://<server_address>). Optionally, define a scratch_dir to handle temporary files using the temporary_path context. Files are first stored in the scratch directory and copied to their final XRootD location after execution, streamlining grid-based file management.

• Apptainer Support

  b2luigi now supports the execution of jobs in an Apptainer container. Tasks can be run both locally and in batch mode for the lsf and the htcondor batch systems. However, for the latter batch system, it is not recommended to use Apptainer images since HTCondor provides its own mechanisms for containerised execution (steerable with the htcondor_settings). To achieve the execution of tasks, an apptainer exec command is built within this class and executed in a subprocess. To steer the execution, one can steer the following settings:

  • apptainer_image: The image to use for the Apptainer container.
  • apptainer_mounts: A list of directories to mount into the Apptainer container.
  • apptainer_mount_defaults: Boolean parameter to mount log_dir and result_dir by default.
  • apptainer_additional_params: Additional parameters to pass to the apptainer exec command.

• Automatic Batch System Discovery

  It is now possible to set batch_system="auto" which will check if bsub or condo_submit commands are available on the system (in that order).

• gbasf2 settings propagation

  The gbasf2 settings for gbasf2_proxy_group, gbasf2_project_lpn_path and gbasf2_proxy_lifetime are now propagated on the task level, introducing the possibility to set these settings task-specific.

• Removal of git_hash

  This release removes the git_hash parameter from the Basf2Task significant parameters. Task IDs and automatic output paths created with the add_to_output will no longer contain the git_hash parameter. This change was implemented to avoid the task ID changing every time the basf2 git hash changes. If the git_hash is needed for the current pipeline's execution or output structure, please add git_hash = b2luigi.Parameter(default=get_basf2_git_hash()) to Basf2Task.

• Settings

  The newly introduced environment variable B2LUIGI_SETTINGS_JSON can be used to steer the path to the settings.json file.

Compatibility

• Workflows including Basf2Task will no longer use the git_hash for building the task graph.
• Workflows including Basf2Task which use the add_to_output method will no longer store the output in directories containing the git_hash. Current workflows might be affected.
• Workflows including Basf2Task will no longer store the log output in directories containing the git_hash.

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v1.0.2...v1.1.1