Tool 2 Usage
Tool 2 builds on the basic idea of Tool1. Tool 2 allows the usage data monitored by IOV monitoring of ROOT files stored on the HADOOP Servers. More information can be found here. To see a list of all of the command line and other options simply run python HCCTool2.py or if the file has been given executable permissions HCCTool2.py. By default this program will create 3 files for each file close even found in the monitoring data from each file. The files generated include a color coded graphic displaying all file reads, a display of the basket layouts similar to Tool 1's output with file read information overlaid, and a legend for the second graphic produced.
The following are the command line options available to the tool and the values that they require. The One required arguments (input file ) is listed first.
Options:
-h, --help show this help message and exit
-f FILE, --file=FILE ROOT input file REQUIRED
-t TREENAME, --tree=TREENAME Tree to parse, Default Event
-s READ_CUTOFF, --sizeCutoff=READ_CUTOFF Read size cutoff, Default 100
-o OUTPUT, --output_directory=OUTPUT The name of the output file
-r FILENAMEREGEX, --regex=FILENAMEREGEX Regular expression to filter files
-w WINDOW_SIZE, --window_size=WINDOW_SIZE Size of the displayed data graphic in mb Default:20
-b BEGINNING, --begin_read=BEGINNING Beginning read to display in the file graphic
--raw_only Create only the raw read graphics and not overlaid file usage
-c CHECK_FUN_FILE, --even_check=CHECK_FUN_FILE File to import that contains checkEvent method to validate any event
-n TOPN, --top_n=TOPN Color only the top N branches in the file
-p PREFIX, --prefix=PREFIX Prefix to append to files to search for file on system
Below is some sample output from the tool and then a rundown of all the options from the tool.
A valid ROOT file is required to run the program and is specified using the -f flag in the output. This file must contain a valid Xrd monitoring tree named XrdFar. If no file is specified the program will display an error message and exit.
This argument specifies which tree will be parsed and displayed on the usage and read graphs in Tool2. By default this argument is 'Events.' -t will change this behavior.
The -o flag allows the user to specify the name of the output directory. This directory must exist on the system and the user must have permission to write to this directory. Default is the current directory. This program may produce a large amount of output files so it is desirable to place them all in a relevant directory.
Since many read event contain less than 100Mb and are not useful the program by default filters out any file read event that contains less than 100Mb read from the file. This default cutoff size can be changed using the -s command.
The program automatically searches for the file to parse to generate the file layout that the read data will be overlaid. Often this files are in one location on disk. The prefix flag allows the user to specify where to begin the search for the .ROOT files.
By default all files that are contained within the file close even ROOT file and can be located on the system will be parsed and output will be produced for each file. The -r option allows a user to filter which files are parsed using a python regular expression. Any legal python regular expression is allowed. For more information on python regular expressions see the Python documenation.
This option allows the program to only output the raw read graphics that are shown below.
This advanced option allows even more filtering if desired by users. If this command line option is followed by a valid python file it will be imported and every file encountered in the file monitoring file will be checked using a call to the check_event(event) method in the python file that is imported. This method has access to all branches in the XrdFar tree contained in the the file read information root files. For more information on what is contained in this tree see this link.
The raw reads displays every read on the file color coded. Each time data is read from a file that information is stored in the xrd root files. Tool 2 uses this data to create a graphic where each read is depicted on the graphic. Locations where the file reads backtrack within a file are highlighted by a star. Below is an example file.
The file usage information is next overlaid on the file structure. The file is parsed and a graphic of the file layout is produced. However, an additional line of data is added for each mb within the file. All reads on the file are colored on the additional line according to chronological order. Below is an example of the two files created using overlaid reads.
This option enables the program to begin the file layout on the graph after it has encountered this many reads within the file instead of starting at the first read encountered as normal. The following is an example of the overlaid reads with the 1500th read encountered in the file.
This option enables a change to the default window size of 20mb on the usage overlay plots. The following example is the file used in this example with a window size of 60 mb and another below with a window size of 30 mb.
This option is exactly like the option in tool1. It changes the number of branches colored in the usage overlay plots. An example of the same file used in this layout with only the top 3 branches colored is displayed.
