Skip to content

UT‐StorR Archival Storage

Jump to bottom
Ryan edited this page Mar 5, 2025 · 1 revision

UT-StorR Archival Storage

UT now has a tape-based archival system in place (link).

Preparing data for UT-StorR

OIT guide

Recommended steps to archive a project:

  1. Try to structure directory so most of content is contained in directories at the top level. The idea is each directory will be a reasonable size to tar compress. This may not always be the case, see OIT guidance for optimal file sizes and use best judgement.
  2. Remove .git directories (they won't be used in archive and add to the checksums).
  3. Perform checksums on all files (recommend untarring directories so the checksum is exhaustive) and save it as a file.
  4. Compress top-level directories using tar.zst (ideal) or tar.gz.
  5. Confirm the integrity of the tar archives.
  6. Remove the source top-level directories (if you're 100% certain they're archived properly).
  7. Perform a checksum on the tar archives and save it as a file.
  8. Populate a README.txt file with the OIT recommendations (number of files, total size, etc.).
  9. Copy files to UT-StorR (one or two copy) using scp over DTN2.
  10. Confirm the file transfer was a success (use the post-tar checksums file).
  11. Cross your fingers and toes and delete the source directory.

Notes:

  • 20-200GB range is ideal for speed of transfer

1. Document all files present as a checksum

Checksum all files:

find <directory> -type f -exec sha1sum {} \; > checksums.sha1

This will give you a full list of all files in the directory and their checksums. This is also a useful guide for retrieval of your files in the future.

To get a list of common file suffices to summarize your filetypes, you can run the following:

cat checksums.sha1 | tr -s " " | cut -d " " -f 2 | rev | cut -d "." -f 1 | rev | sort | uniq

2. Compress directories such that they approximate the 20-200GB file sizes

Zstandard compressed tar files are the most optimal form of compression for this size range and are recommended by OIT.

An example of how to create a zstandard tarball (.tar.zst) from a given directory.

tar --use-compress-program zstd -cf directory.tar.zst directory/

You can check the integrity of the file by running the following:

tar -I 'zstd' -dpf <directory>.tar.zst

This assumes there is a "directory" matching the name (sans .tar.zst). If any files are different it should print to the screen.

3. Create a documented README for ease of file retrieval

Here is the OIT template for a README.txt

README
Directory:
Project Name:
Project Description:

Number of Files:
File Formats:
Volume of Data:
PI name:
Researcher name(s):
Related publication DOIs:
Selected retention scheme:
Other notes:

Example filled out:

README
Directory: readsynth_analysis
Project Name: readsynth_analysis
Project Description: Associated research files for readsynth publication

Number of Files:
File Formats: mixed (primarily fasta, fastq, png, csv)
Volume of Data:
PI name: Margaret Staton
Researcher name(s): Margaret Staton, Ryan Kuster
Related publication DOIs: https://doi.org/10.1186/s12859-024-05809-3
Selected retention scheme: 1copy
Other notes:

Helpful UT-StorR commands from /lustre/utstorr/examples:

#Checksum all the files in a directory and output to one file:
find <directory> -type f -exec sha1sum {} \; > checksums.sha1

#Checksum all the files in a directory and output to one file using 4 threads (useful if #files >1,000,000)
find <directory> -type f -print0 | xargs -0 -P4 sha1sum > checksums.sha1

#Compress a directory using gzip compression
tar -czpf my_archive.tar.gz <directory>

#Compress a directory using zstd compression
tar -I zstd -cpf my_archive.tar.zst <directory>

#Compress a directory with zstd using multiple threads
tar -I 'zstd -T4' -cpf my_archive.tar.zst <directory>

#Compress a directory with zst using multiple threads and split the resulting archive into chunks of no more than 8TB
#(note, each part of the resulting archive will have a name like "my_archive.tar.zstAA", "my_archive.tar.zstAB", etc
tar -I 'zstd -T8' -cpf - <directory> | split --bytes=8T - my_archive.tar.zst

#Uncompress a gzipped archive
tar -xzpf my_archive.tar.gz

#Uncompress a zstd archive
tar -I zstd -xpf my_archive.tar.zst 

#Uncompress an archive that has been split into multiple files
cat my_archive.tar.zst* | tar -I zstd -xpf -

Clone this wiki locally