Big Data Bag Utilities
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
bdbag Bump version. Feb 11, 2019
doc Add monkeypatch for hashlib.algorithms_guaranteed for unpatched Pytho… Feb 7, 2019
profiles Change GitHub repo paths referenced in source code and documentation. Mar 19, 2018
test Add monkeypatch for hashlib.algorithms_guaranteed for unpatched Pytho… Feb 7, 2019
.travis.yml Revert inclusion of OSX and Windows in TravisCI due to Travis/Python … Oct 18, 2018 Add monkeypatch for hashlib.algorithms_guaranteed for unpatched Pytho… Feb 7, 2019
LICENSE Update Oct 29, 2018
setup.cfg Update setup.cfg May 23, 2018


Build Status Coverage Status PyPi Version PyPi Wheel Python Versions License

Big Data Bag Utilities

The bdbag utilities are a collection of software programs for working with BagIt packages that conform to the BDBag and Bagit/RO profiles.

The bdbag profiles specify the use of the fetch.txt file, require serialization, and specify what manifests must be provided with a bdbag.

These utilities combine various other components such as the Bagit-Python bag creation utility and the Bagit-Profiles-Validator utility into a single, easy to use software package.

Enhanced bag support includes:

  • Update-in-place functionality for existing bags.
  • Automatic archiving and extraction of bags using ZIP, TAR, and TGZ formats.
  • Automatic generation of remote file manifest entries and fetch.txt via configuration file.
  • Automatic file retrieval based on the contents of a bag's fetch.txt file with multiple protocol support.
  • Built-in profile validation.
  • Built-in support for creation of bags with Bagit/RO profile compatibility.

An experimental Graphical User Interface (GUI) for bdbag can be found here.

Technical Papers

"I'll take that to go: Big data bags and minimal identifiers for exchange of large, complex datasets" explains the motivation for BDBags and the related Minid construct, provides details on design and implementation, and gives examples of use.

"Reproducible big data science: A case study in continuous FAIRness" presents a data analysis use case in which BDBags and Minids are used to capture a transcription factor binding site analysis.


  • Python 2.7 is the minimum Python version required.
  • The code and dependencies are also compatible with Python 3, versions 3.3 through 3.6.


The latest bdbag release is available on PyPi and can be installed using pip:

pip install bdbag

Note that the above command will install bdbag with only the minimal dependencies required to run. If you wish to install bdbag with the extra fetch transport handler support provided by boto (for AWS S3) and globus (for Globus Transfer) packages, use the following command:

pip install bdbag[boto,globus]

Installation from Source

You can use pip to install bdbag directly from GitHub:

sudo pip install git+


pip install --user git+

You can also download the current bdbag source code from GitHub or alternatively clone the source from GitHub if you have git installed:

git clone

From the root of the bdbag source code directory execute the following command:

sudo pip install .


pip install --user .

Note that if you want to install the extra dependencies from a local source directory you would use the following command:

pip install .[boto,globus]


The unit tests can be run by invoking the following command from the root of the bdbag source code directory:

python test


This software can be used from the command-line environment by running the bdbag script. For detailed usage instructions, see the CLI Guide.


Some components of the bdbag software can be configured via JSON-formatted configuration files. See the Configuration Guide for further details.

Application Programming Interface

It is also possible to use bdbag from within other Python programs via an API. See the API Guide for further details.


A CLI utility module is provided for various ancillary tasks commonly involved with authoring bdbags. See the Utility Guide for further details.

Change Log

The change log is located here.