Skip to content

User Guide

Jump to bottom Edit New page
Jens Janssen edited this page May 27, 2015 · 50 revisions

User Guide

This guide is written to help new users of pyBAR. The focus is put on the data structures and the analysis flow.

Data Structure

PyBAR uses three different file formats. The configuration files are stored in human readable text files similar to the RCE system. The FPGA code is stored in a binary file and the arising data from scans and analysis is compressed and stored in HDF5 files (using PyTables).

FPGA Files

There are two binary FPGA files for different adapter cards. One with single chip card / burn in card support

  • mio.bit) supporting Single Chip Card / Burn-in Card (Quad Module Adapter Card)
  • mio_gpac.bit for GPAC support

pyBAR Configuration Files

TODO

The configuration.yaml can be extended to also include run configuration for each type of run (e.g. Analog Scan, Digital Scan, …) and run configuration for all runs.
The data output folder (will be created on first run) contains raw data generated during a run, and analyzed data, PDF output files, and the configuration files, which are generated at the end of each run. In addition one can find the run.cfg containing information about each run (exit status, run name, start and stop time), and the crash.log containing traceback information if the exit status was “CRASHED”.

Additional help is given inside the configuration.yaml from the repository.

FE-I4 Configuration Files

The standard config parameters are loaded from the fei4_defines.py (equivalent configuration files are located here).

Configuration files are generated during the first run. They are placed inside the data output folder given by the module_id parameter in configuration.yaml. The flavor of the initial configuration file is given by the fe_flavor parameter in configuration.yaml.
The FE-I4 configuration is splitted up into several files. The configs subfolder contains the main configuration files. They contain the global configuration, chip parameters, calibration parameters and the paths to the pixel configurations.
The masks subfolder contains all masks (C_high, C_low, Enable, DigInj, Imon). The column number is given on top and the row number on the left:


###  1     6     11    16     21    26     31    36     41    46     51    56     61    66     71    76
  1  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111
  2  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111
  3  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111
  4  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111  11111-11111

  ...

The TDAC and FDAC settings are placed in the tdacs and fdacs subfolder, respectively. To make the content fit on a computer screen, one line represents only the half of a FE column. The indices on the left represents the row number and a stands for columns 1-40 and b for columns 41-80:


###    1  2  3  4  5  6  7  8  9 10   11 12 13 14 15 16 17 18 19 20   21 22 23 24 25 26 27 28 29 30   31 32 33 34 35 36 37 38 39 40
###   41 42 43 44 45 46 47 48 49 50   51 52 53 54 55 56 57 58 59 60   61 62 63 64 65 66 67 68 69 70   71 72 73 74 75 76 77 78 79 80
  1a   8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8
  1b   8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8
  2a   8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8
  2b   8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8    8  8  8  8  8  8  8  8  8  8

  ...

Once, the data output path is filled with first data, the configuration for the FE-I4 can be specified. There are different methods how to do it which involve changing the fe_configuration parameter in the configuration.yaml:

  • Leave it blank. The configuration with the highest run number and run staus FINISHED will be taken from the data output folder.
  • Select configuration by run number:
    fe_configuration : 42
  • Specify configuration file by full or relative path:
    fe_configuration : ../42_scan_analog.cfg
  • Specify configuration from data output file (HDF5 file) by full or relative path:
    fe_configuration : ../42_scan_analog.h5

Note: the configuration files (.cfg) are created after the data analysis at the very end of a run. They contain modifications made to the FE-I4 configuration. In contrast, the data output files (.h5) contain the FE-I4 configuration at the very beginning of a run.

Data File

The data files are using the HDF5 file format (.h5). HDF5 designed to store and organize large amounts of numerical data. HDF5 is using nodes and leafs to organize the data. The PyTables is a package for managing HDF5 files. It is built on top of the HDF5 library and is providing additional features like BLOSC filter.

One can distinguish between data files at two different stages:

  • the data output file containing the data generated during a run (raw data, meta data, parameter data, configuration data)
  • the interpreted data file containing interpreted/analyzed data (histograms, hit and cluster tables)

The interpreted data is usually generated at the end of a run during the analyze step.

Scan File

The data output file is created at the beginning of a scan and data is added at every readout. The file contains four different nodes: the FE-I4 raw data, the meta data, the configuration, and the scan parameter data (optional).

  1. Raw data
    The raw data contains all FE words + FPGA trigger words + FPGA TDC words in an one dimensional array.
     

     
    Every word is a 32-bit unsigned int. The format is: 82444303 = 0 000 0100 1110 1010 0000 0000 0000 1111
     
    If the first bold bit is 1 the word is a trigger word followed by a trigger number. If it is 0 the following bits indicate the FE number. This number is usually 5 when you use the single chip adapter card. With other adapter cards (e.g. burn-in card) this number identifies the different FE connected to it. The remaining 24-bits hold the FE word. For more information (e.g. TDC word) please look here.
     
  2. Meta data
    The meta data stores read out related infos:
     

     
    One row is added for every read out. It contains the word index at the first / last+1 word of the read out and the total numbers of words. To allow time based analysis it also has a timestamp and an error code indicating read out errors (like out of sync). The error code is not used so far.
     
  3. Configuration data
    All configuration parameters needed to be able to repeat the scan are stored here in different nodes.
    This includes the Front-End configration, the specific run configuration etc.
     
  4. Parameter Data
    In the parameter table the parameters that is changed during a scan is monitored (e.g. PlsrDAC setting during a threshold scan). At every readout the acutal parameter settings is added to the table. One column is reserved for one parameter.

Interpreted Data File

The raw data is interpreted, analysed and plotted with the raw data converter class. The output HDF5 file can hold hit data, cluster data, meta data and histograms at different HDF5 notes.

  1. Hit data
    The hit data table stores the information for every hit.
     

     
    The columns are:
    • event_number: The absolute event number. Only events with hits appear here.
    • trigger_number: The TLU trigger number. If no TLU is used the trigger number is always 0.
    • relative_BCID: The relative BCID is the number of data headers of one event when the hit occured.
    • LVL1ID: The LVL1ID is the internal LVL1 trigger counter value of the FE.
    • column: the column position of the hit
    • row: the row position of the hit
    • tot: the tot code of the hit
    • BCID: the absolute BCID value
    • TDC: the TDC value. If TDC was not used this values is always 0.
    • TDC_time_stamp: the TDC time stamp value. If not used it is always 255.
    • trigger_status: the trigger status of the event where the hit occured. The trigger status error code is a sum of the different binary coded trigger error codes that occured during the event. Please take a look at the histogram section for the error code explanation.
    • service_record: the service records that occured in the event of the hit. The service record code is a sum of the different binary coded serivce records that occured during the event.
    • event_status: the status of the event of the hit. The event error code is a sum of the different binary coded event error codes that occured during the event. Please take a look at the histogram section for the error code explanation.
       
  2. Meta data
    There are two output tables for meta data. The table called MetaData is a copy of the input meta table (see Scan data section above), but has event numbers instead of raw data word indices and additional scan parameter columns for each scan parameter.
     

     
    The table EventMetaData stores the start and stop raw data word indices for each event. This table is usually not created and only needed for in depth raw data analysis. It can act as a dictionary to get the connection between event number and raw data words of the event.
     

     
  3. Hit Histograms
    During raw data interpretation a lot of different histograms can be created on the fly. Usually single hit information is less interesting while histogram data provides sufficient information. The most common histograms are listed below, including there plotting representation created by the pyBAR plotting module:
    1. HistOcc: The occupancy histogram stores the FE hits for each scan parameter in a three dimensional map. The first two dimensions reflect the pixel alignment (col x row) and the third dimension is the scan parameter.
       


       
      The occupancy per pixel and scan parameter is also plotted with a 2D heat map with logarithmic color coding. The data of a threshold scan is show in the following plot with scan parameter == PlsrDAC.
    2. HistThreshold: A two dimensional histogram (col x row) storing the threshold of each pixel. The threshold is given in pulser DAC (1 DAC ~ 55 e).
       

       
    3. HistNoise: A two dimensional histogram (col x row) storing the noise of each pixel. . The threshold is given in pulser DAC (1 DAC ~ 55 e).
       

       
    4. HistRelBcid: A one dimensional histogram [0:16[ storing the relative BCID of the hits.
       

       
    5. HistTot: A one dimensional histogram [0:16] storing the tot codes of the hits.
       

       
    6. HistErrorCounter: A one dimensional histogram [0:16] counting the event error codes that occured in events.
       

       
      The error codes are:
      • No error (error code 0)
      • Service record occurred in the event (1). This can happen if service records are activated and does not mean that there is a serious error. One should check the service record error code if this occurs to often.
      • There is no external trigger (2). Usual case if no TLU is used.
      • The LVL1 ID is not constant during the event (4). Serious error that should not be ignored. Indicates a bad Front-End / power-up.
      • The number of BCIDs in the event is wrong (8). Serious error that should not be ignored.
      • The event has at least one unknown word (16). Should not happen in normal operation mode (reasonable threshold, no noisy pixels). Serious error if occuring too often.
      • The BCID counter is not increasing by one between two data headers (32). Error that can be ignored. Indicates a bad power-up. Power cycling usually helps.
      • The event is externally triggered and a trigger error occured (64). Please contact the pyBAR developers if you see this.
      • The software buffers are too small to store all hits of one event. The event is truncated (128). This indicates that there are too many data headers for TLU triggered events or too many hits per event for the analysis (__MAXHITBUFFERSIZE = 4000000). More than __MAXHITBUFFERSIZE per event is extreme and can only be created in very rare use cases (e.g. stop-mode readout).
      • The event has one TDC word. Is expected if the FPGA TDC is used.
      • The event more than one TDC word. Is expected if the source is very hot. These events cannot be used for TDC analysis. Thus there should be only a few events with more than one TDC word.
      • The TDC value is too big and therefore meaningless.
  1. HistTriggerErrorCounter: A one dimensional histogram [0:8] counting the trigger error codes that occured in events.
     

     
    The error codes are:
    • No error (error code 0)
    • Trigger number does not increase by 1 (1)
    • One event has two trigger number words (2)
    • not used yet (4)
    • not used yet (8)
    • not used yet (16)
  1. HistServiceRecord: A one dimensional histogram [0:32] counting the service record codes that occured in events.
     

     
    Please check the FE manual for error code explanations.

Clustered Data

Clustered data is usually stored as additional nodes in the analyzed data file or in a new file. The nodes are cluster hits, cluster or histograms.

  1. Cluster hits
    The table ClusterHits is a copy of the Hits table (see above) but with four additional columns:
    • cluster_id: The cluster ID of the hit [0:number of cluster in the event]
    • is_seed: Is one if the hit is the seed hit, otherwise 0.
    • cluster_size: The size of the cluster the hit belongs to.
    • n_cluster: The number of cluster of the event the hit belongs to.
       

       
  2. Cluster
    The table Cluster stores the infos for each cluster.
     

     
    The columns are:
    • event_number: The absolute event number. Only events with cluster appear here.
    • id: The cluster ID of the cluster [0:number of cluster in the event].
    • size: The number of hits that belong to the cluster.
    • tot: The tot sum of all hits.
    • charge: The charge sum of all hits (not used yet).
    • seed_column: The column of the seed pixel.
    • seed_row: The row of the seed pixel.
    • event_status: The status of the event of the cluster. The event error code is a sum of the different binary coded event error codes that occured during the event. Please take a look at the histogram section for the error code explenation.
  3. Cluster Histograms
    Different histograms can be created on the fly while interpreting the data. Usually single cluster information are not needed while the cluster histograms provide sufficient information. The different histograms are:
    • HistClusterSize: A one dimensional histogram [0:1024] filled with the cluster sizes.
       

       
    • HistClusterTot: A two dimensional histogram (tot x cluster size) storing the size of the cluster for different tot values. The cluster size = 0 column is the sum of the other columns.
       

       

Online Monitor

PyBAR has a real time online monitor that can be used to show the data that is stored during a scan. This is very usefull to understand what is going on! The online monitor runs in a seperate process and does not influence the data taking. Thus it is save to have an instance of the online monitor always open.
A stand alone example that simulates the online monitor by replaying any raw data file can be found in the examples folder. Just run replay_data.py.

To use the online monitor for any scan, just start the script online_monitor.py in the pybar/ folder. For example open a console, goto the pybar/ folder and type

 python online_monitor.py

The online monitor window look like this:

With the right mouse button pressed you can zoom the histograms. You can switch the data by pressing on the blue tabs and reorder the windows. No everything is mentioned here, just play with it! The top, right number in the interface is the integration time in read-outs for the hit histograms. Zero means infinite integration time. 1 is a refresh for every readout and thus a fast real time plot.

To enable the online monitor, just add this line to the _default_run_conf in the run script.

'send_data': 'tcp://127.0.0.1:5678'

This will enable the online monitor just for this run script. To enable the online monitor for all run scripts, add these lines to the configuration.yaml:


run_conf:
    send_data : 'tcp://127.0.0.1:5678'

This is the standard address to send the data to (localhost with port 5678). Change this address according to your needs. The data can be sent even to another PC.

Data Analysis

If the histograms mentioned above are not sufficient and there is no special analysis script in /pybar/scans or no fitting method in analysis.py that does what you want (like analyze_source_scan_tdc_data or def analyze_hits_per_scan_parameter) well… than you have to do your analysis on your own. The following sections explain the analysis details.
 
The pyBAR readout is designed in a way that every data word from the Front-End is stored and analyzed. There is no data reduction done in the FPGA, the user is in full control. Since the amount of data words can be large (e.g. threshold scan: 15000000 words = 450 Mb data) fast and optimized c++ classes were written to cope with the data efficiently. These classes are compiled via cython and all needed functions are wrapped in one python class called AnalyzeRawData. This class was written to hide the complexity of the conversion process. Additional helper function for the analysis are analysis_utils. The compiled C++ code used in the background is intended to work and not to be changed on a frequent basis. To analyze the raw data is straight forward. An example script can be found here.

Warning the following section is only for interested people and does not have to be understood to do a self written analysis:
There are three objects in the AnalyzeRawData class for data analysis:

  • PyDataInterpreter:
    The PyDataInterpreter takes the raw data words (see data structures → Scan data file → raw data) and interprets them. This includes hit and event finding as well as error checks on the raw data structure and the counting of service records. For more information on the FE-I4 data structure please take a look at the FE manual.
  • PyDataHistograming:
    The PyDataHistograming takes the hit infos from the PyDataInterpreter and histograms them. It also provides a fast algorithm to deduce the threshold and the noise from the occupancy data without S-Curve fitting.
  • PyDataClusterizer:
    The PyDataClusterizer takes the hit and event infos from the PyDataInterpreter and clusters them.

A in depth view what the three different classes (PyDataInterpreter, PyDataHistograming,PyDataClusterizer) do and the input/output of the AnalyzeRawData shows the following figure.
 

 

Scans

How-to: Setting up and Running an Analog Scan

TODO

How-to: Setting up and Running an External Trigger Scan

TODO

Advanced Scans and Analysis

For some special applications you might find our Application notes useful.

Add a custom footer
Add a custom sidebar

Clone this wiki locally