ldv

Light Data Versioning

Ldv is a tool for version tracking big files on a remote storage and storing digest files together with code that uses the big files. A digest file is a small file containing information about the file to version track. The information includes file size in bytes, timestamp of version, hexdigest (the version number), and local and remote filepaths.

Ldv can be run from the command line or from code, using the APIs of the package.

Ldv has no external dependencies on other programs, e.g. Git.

Currently, only AWS is supported as remote storage.

Getting started

Installation

This will install the package to version track on AWS remote storage.

pip install ldv

Usage

1. Authentication

Authenticate with remote storage provider.

ldv auth aws -rn <region_name> [-pn <profile name>] [-ec]

• -rn is the region name where the bucket is located.
• -pn is the name of the profile specified in ~/.aws/config
• -ec Use credentials stored in environment variables. LDV_AWS_ACCESS_KEY_ID and LDV_AWS_SECRET_ACCESS_KEY must be set. Will not be used if profile is provided.

You either use -pn or -ec, not both.

Example:

ldv auth aws -rn eu-west-1 -pn dev-profile

2. Initialize

Initialize the local base path you want to version track files under. Also specify the remote storage url.

ldv init -p <path> -u <url> [-rta] [-du]

• -p specifies the local path that contains files to version track.
• -u is the remote storage url, e.g. s3://my_bucket/sub_folder.
• -rta is an optional flag will run version tracking of all files under the path.
• -du is an optional flag will prevent files from being uploaded to remote storage every time you version track them. Use this if you only want to create a digest files locally but not upload the files to remote storage.

Example:

ldv init -p local-data-folder -u s3://my_s3_bucket/sub_folder

Version track file

ldv versioning track <file path> [-du]

• file path to version track. Can be absolute or relative path but must be under path provided when running 'init' command.
• -du is optional flag if you don't want to upload the file.

Example:

ldv versioning track local-data-folder/first_file.txt

Version track all files

Version track all files under path specified in the 'init' command.

ldv versioning track-all [-du]

• -du is optional flag if you don't want to upload the files.

Download file

Download file from remote storage to local storage.

ldv versioning download <file path to digest file> [-v <version>]

• -v is the optional version. If not provided, the version in the .digest file will be used.

Download all files

Download all files under folder specified in 'init' command.

ldv versioning download-all

List versions of file

ldv versioning list-versions <path to digest file>

Example:

ldv versioning list-versions local-data-folder/first_file.txt.digest

Contact

Martin Norin, ldv@martinnorin.se