Big Data Bag Utilities
bdbag utilities are a collection of software programs for working with
BagIt packages that conform to the BDBag and Bagit/RO profiles.
bdbag profiles specify the use of the
file, require serialization, and specify what manifests must be provided with a
bdbag utilities incorporate functions from various other Python-based
bagit components (such as the
Bagit-Python bag creation utility and the
utility) and wraps them in a single, easy to use software package with additional features.
Enhanced bag support includes:
- Update-in-place functionality for existing bags.
- Automatic archiving and extraction of bags using ZIP, TAR, and TGZ formats.
- Automatic generation of file manifest entries and
fetch.txtfor remote files via configuration file.
- Automatic file retrieval based on the contents of a bag's
fetch.txtfile with multiple protocol support. Transport handlers for
globusare provided, along with an extensibility mechanism for adding externally developed transports.
- Built-in support for creation of bags with Bagit/RO profile compatibility.
An experimental Graphical User Interface (GUI) for
bdbag can be found here.
"I'll take that to go: Big data bags and minimal identifiers for exchange of large, complex datasets" explains the motivation for BDBags and the related Minid construct, provides details on design and implementation, and gives examples of use.
"Reproducible big data science: A case study in continuous FAIRness" presents a data analysis use case in which BDBags and Minids are used to capture a transcription factor binding site analysis.
- Python 2.7 is the minimum Python version required.
- The code and dependencies are also compatible with Python 3, versions 3.5 through 3.9.
bdbag release is available on PyPi and can be installed using
pip install bdbag
Note that the above command will install
bdbag with only the minimal dependencies required to run.
If you wish to install
bdbag with the extra fetch transport handler support provided by
boto (for AWS S3)
globus (for Globus Transfer) packages, use the following command:
pip install bdbag[boto,globus]
Installation from Source
You can use
pip to install
bdbag directly from GitHub:
sudo pip install git+https://github.com/fair-research/bdbag
pip install --user git+https://github.com/fair-research/bdbag
You can also download the current
bdbag source code from GitHub or
alternatively clone the source from GitHub if you have git installed:
git clone https://github.com/fair-research/bdbag
From the root of the
bdbag source code directory execute the following command:
sudo pip install .
pip install --user .
Note that if you want to install the extra dependencies from a local source directory you would use the following command:
pip install .[boto,globus]
The unit tests can be run by invoking the following command from the root of the
bdbag source code directory:
python setup.py test
This software can be used from the command-line environment by running the
bdbag script. For detailed usage
instructions, see the CLI Guide.
Some components of the
bdbag software can be configured via JSON-formatted configuration files.
See the Configuration Guide for further details.
Application Programming Interface
It is also possible to use
bdbag from within other Python programs via an API.
See the API Guide for further details.
A CLI utility module is provided for various ancillary tasks commonly involved with authoring bdbags. See the Utility Guide for further details.
The change log is located here.