Skip to content

Ifdh commands

Jump to bottom
Marc Mengel edited this page Feb 13, 2025 · 1 revision

ifdh commands

Note that many of these commands are new in ifdh v1_3_x and/or v1_7_x.

Many commands implicitly use a State Directory.

Environment variables

See Environment Variable Reference

Commands

  • ifdh cp args

general file copy using cpn locks dd, gridftp, or srmcp
supports:

* cp src1 dest1 [';' src2 dest2 [';'...]] -- basic source/dest filenames
* cp -r src1 dest1 [';' src2 dest2 [';'...]] -- recursive directory copies
* cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]] -- copies to dest. directory
* cp -f file_with_src_space_dest_lines -- copies to a list file
* any of the above can take ---force={cpn,gridftp,srmcp,rootd,http}
* any of the file/dest arguments can be URIs

Users might consider utilizing these environment variables to control the response of "ifdh cp" commands:

* export IFDH_GRIDFTP_EXTRA="-st N" which sets the stall timeout at N seconds IFDH_XROOTD_EXTRA
* export IFDH_CP_MAXRETRIES=N, we find 2 is usually a good number
* note that IFDH_GRIDFTP_EXTRA can be used to pass many arguments to globus-url-copy http://toolkit.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html\ * note that IFDH_XROOTD_EXTRA can be used to pass many arguments to xrootd/xrdcp actions https://linux.die.net/man/1/xrdcp\ * full environment variables here: [[Environment Variable Reference]]

  • ifdh fetchInput src_uri

get input file to local scratch, return scratch location

  • ifdh localPath src_uri

return scratch location fetchInput would give, without copying

  • ifdh addOutputFile filename

add output file to set

  • ifdh copyBackOutput dest_dir

copy output file set to destination with cpn or srmcp

  • ifdh log args

logging

  • ifdh enterState state

log entering/leaving states

  • ifdh leaveState state 
<!-- -->
  • ifdh createDefinition name dims user group

make a named dataset definition from a dimension string

  • ifdh deleteDefinition name

remove data set definition

  • ifdh describeDefinition name

describe a named dataset definition

  • ifdh translateConstraints dims

give file list for dimension string

  • ifdh locateFile name

locate a file

  • ifdh getMetadata name

get a files metadata

  • ifdh dumpStation name what

give a dump of a SAM station status

  • ifdh startProject name station defname_or_id user group

start a new file delivery project

  • ifdh findProject name station

find a started project

  • ifdh establishProcess projecturi appname appversion location user appfamily description filelimit schemas

set yourself up as a file consumer process for a project

  • ifdh getNextFile projecturi processid

get the next file location from a project

  • ifdh updateFileStatus projecturi processid filename status

update the file status (use: transferred, skipped, or consumed)

  • ifdh endProcess projecturi processid

end the process

  • ifdh dumpProject projecturi

say what the sam station knows about your process

  • ifdh setStatus projecturi processid status

set process status

  • ifdh endProject projecturi

end the project

  • ifdh cleanup

clean up any tmp file stuff

  • ifdh renameOutput how

give output files reported with addOutputFile a unique name

  • ifdh mv args

general file rename using mvn or srmcp

  • ifdh ls loc recursion_depth force

Get a list of directory contents, or check existence of files

  • ifdh mkdir loc force

make a directory (i.e. for file destination)

  • ifdh rm loc force

remove files

  • ifdh rmdir loc force

remove directories

  • ifdh more loc

view text files

  • ifdh pin loc seconds

pin file

  • ifdh chmod mode loc force

change file permissions

  • ifdh rename loc loc2 force

atomic rename items in same directory/fs

  • ifdh ll loc recursion_depth force

list files with long listing
use recursion_depth== 0 to check directory without contents

  • ifdh lss loc recursion_depth force

list files with sizes
use recursion_depth== 0 to check directory without contents

  • ifdh findMatchingFiles path glob

find filenames and sizes matching pattern

  • ifdh fetchSharedFiles args

filenames and sizes matching pattern moved locally enough to be seen

  • ifdh locateFiles args

locate multiple files

  • ifdh checksum loc

cheksum file

Clone this wiki locally