-
Notifications
You must be signed in to change notification settings - Fork 0
Data Versioning
The MLFramework Template utilizes Data Version Control (DVC) to manage large datasets, model weights, and artifacts. Git is inherently unsuited for tracking large binary files; DVC mitigates this by tracking pointers (metadata) in Git while storing the actual raw data in a dedicated remote storage backend.
If the repository was newly instantiated from the template, DVC might require local initialization. A dedicated Makefile target is provided to streamline this process safely:
make dvc_initWhen introducing new large datasets (e.g., GPR scans, high-resolution imagery, or processed tensors) into the data/ directory, they must be tracked via DVC rather than Git.
- Tell DVC to track the dataset:
dvc add data/raw_datasetThis command computes the MD5 hash of the dataset, moves the actual files to the local .dvc/cache, and generates a data/raw_dataset.dvc tracking file.
- Commit the tracking metadata to Git:
git add data/raw_dataset.dvc data/.gitignore
git commit -m "Track raw dataset via DVC"The project environment is pre-configured with the dvc-gdrive dependency, allowing the use of Google Drive as a remote backend for artifact storage. This is highly effective for academic and independent research setups.
To configure a Google Drive remote, obtain the specific Folder ID from your target Google Drive folder URL and execute:
dvc remote add -d storage gdrive://<YOUR_FOLDER_ID>Note: OAuth authentication will be requested in the terminal upon the first push or pull operation.
Once the remote is configured and data is locally tracked, you must synchronize the artifacts with the remote backend.
To upload local artifacts to the remote storage:
dvc pushTo retrieve artifacts on a new machine, or after a repository clone:
dvc pull- Never commit raw data to Git: Ensure the
data/directory remains appropriately configured in.gitignore. Only commit the lightweight.dvcfiles. - Git-DVC Parity: DVC is designed to track data state exactly alongside Git branches. When switching branches (
git checkout <branch>), always executedvc checkoutto ensure your localdata/directory instantly matches the specific dataset version expected by the code on that branch.
If you utilize this framework template in your research or engineering workflows, please consider citing it to support ongoing development:
@software{MLFramework_Template_2026,
author = {Danylo Chystiakov},
title = {MLFramework Template: A Reproducible MLOps Environment},
year = {2026},
url = {[https://github.com/allllpina/MLTemplate](https://github.com/allllpina/MLTemplate)}
}This template is built upon the philosophies and architectural patterns established by the following foundational projects:
- Hydra for hierarchical configuration.
- PyTorch Lightning for hardware-agnostic training abstractions.
- Data Version Control (DVC) for Git-integrated data management.
Contact & Support For inquiries regarding architectural decisions, bug reports, or feature requests, please open an issue on the GitHub repository.
Maintained by Danylo Chystiakov.