Data format requirements: Displacement data file formats

TL;DR summary:

Chorus can import a full matrix of displacement data from five different displacement data file formats:

($d$= detection index ($0$ to $n-1$), $g$=generation index ($0$ to $n-1$), $t$=time index ($0$ to $m-1$), $n$=number of elements in array, $m$=number of temporal samples per A-scan)

1. .npy 3D: .npy file containing a 3D NumPy ndarray in NumPy[d, g, t] format. $\color{lime}{\textsf{(Recommended)}}$

2. .npy 2D: .npy file containing a 2D NumPy ndarray in NumPy[dn+g, t] format.

3. .txt 2D: .txt file containing rows of whitespace-separated values, where A-scans form rows. When loaded with numpy.loadtxt using default parameters, a 2D ndarray in NumPy[dn+g, t] format is returned.

4. .mat 3D: Version 7.2 or lower .mat file containing a single 3D MATLAB array in MATLAB(t, g+t, d+1) format.

5. .mat 2D: Version 7.2 or lower .mat file containing a single 2D MATLAB array in MATLAB(t, dn+g+1) format.

After the user supplies a path to their chosen file, the format is automatically detected out of the list of five formats above based on the file extension (.npy, .txt or .mat) and, for .npy and .mat files, the number of dimensions of the array stored inside (3D or 2D).

To import a new full matrix, select File > Open full matrix....

In the dialog that opens, enter the path to your displacement data file, and the other required information.

Introduction

Chorus requires multiple pieces of information regarding a 1D, periodic full matrix dataset in order to display and process it.

Many of these peices of information (such as the time basis of the A-scans, the material properties of the sample etc.) can be input manually by the user. The largest single set of data Chorus requires is a file containing a full matrix of displacement measurements (the output of the ultrasound sensors used during the full matrix capture).

As illustrated in the Array Requirements page of this wiki, the displacements measured during full matrix capture using a 1D, periodic array can be organised into a 3D data array (the full matrix), where values of displacement are stored in a 3D data volume with dimensions of generation index $g$, detection index $d$, and time index $t$.

Following a Python-ready zero-indexed system:

• Both $g$ and $d$ range in value from $0$ to $(n-1)$ where $n$ is the number of elements in the 1D periodic array.

• $t$ varies from $0$ to $(m-1)$ where $m$ is the number of displacement samples acquired per A-scan.

The number of individual displacement measurements forming a full matrix is therefore equal to $n^2m$. For a 64-element array, capturing 1000 displacement measurements per A-scan, the full matrix would contain 4,096,000 values of displacement!

Full matrices of displacement measurements may be imported into Chorus from files following one of five different formats. The purpose of this wiki article is to describe these five compatible file formats. The five formats are:

1. .npy 3D: .npy file containing a 3D NumPy ndarray in NumPy[d, g, t] format. $\color{lime}{\textsf{(Recommended)}}$

2. .npy 2D: .npy file containing a 2D NumPy ndarray in NumPy[dn+g, t] format.

3. .txt 2D: .txt file containing rows of whitespace-separated values, where A-scans form rows. When loaded with numpy.loadtxt using default parameters, a 2D ndarray in NumPy[dn+g, t] format is returned.

4. .mat 3D: Version 7.2 or lower .mat file containing a single 3D MATLAB array in MATLAB(t, g+t, d+1) format.

5. .mat 2D: Version 7.2 or lower .mat file containing a single 2D MATLAB array in MATLAB(t, dn+g+1) format.

The following figure summarises these five compatible displacement data file formats, and illustrates that, no matter the external file format, all imported displacement data will be converted to the internal format used within Chorus: a 3D numpy ndarray in numpy[d,g,t] format.

The Chorus internal format for the full matrix of displacement values

The format used inside the Python code of Chorus to store the full matrix of displacement values (U) is illustrated in the figure above. The displacement values are stored in a 3D NumPy ndarray, which may be indexed by NumPy[d,g,t].

When U is an ndarray in NumPy[d,g,t] format:

• Individual A-scans form rows. An A-scan can be accessed as U[d, g, :] or simply U[d, g].
• An iso-detection plane can be accessed as U[d, :, :] or simply U[d].
• An iso-generation plane can be accessed as U[:, g, :] or simply U[:, g].
• An iso-time plane can be accessed as U[:, :, t].

The author considered this format to be the most logical and performant arrangement of the measurement as a NumPy ndarray (which is a widely used and well documented python class). Documentation for the NumPy ndarray class can be found here. The notation for indexing on NumPy ndarrays is documented here.

The smallest logical sub-group of displacement values within the full matrix is a single A-scan. This is the series of displacement measurements recorded over time for a fixed combination of generation and detection element. Chorus performs a variety of processing operations on individual A-scans (such as de-trending, frequency-domain filtering & 1D interpolation). These can be performed most rapidly if the displacement values for a single A-scan are stored contiguously in memory. Since NumPy stores array data in row-major (C-contiguous) order by default, it is performant to store A-scans as the rows of a NumPy ndarray. With A-scans stored as rows in an ndarray, they can be accessed without striding. Many functions that operate on ndarrays (in NumPy and other Python libraries) operate along the last axis (rows) by default for this reason.

.npy displacement data formats:

The .npy file format is a simple format for saving a single NumPy ndarray. Documentation for this file format can be found here.

.npy files can be produced using the numpy.save function on an ndarray created in a Python session.

Chorus can import a full matrix of displacement data from .npy files containing an ndarray in one of two array formats:

1. 3D: The ndarray is 3-dimensional, and indexable by numpy[d,g,t]. This is identical to the internal representation of the full matrix within Chorus, and is hence the $\color{lime}{\textsf{recommended}}$ file format.

2. 2D: The ndarray is 2-dimensional, and indexable by numpy[dn+g,t]. With A-scans stored as rows in the 2D array, rows $0$ to $n-1$ contain the set of A-scans with detection index $d=0$, and increasing generation index $g$. The detection index $d$ increments to $d=1$ at row $n$, which contains A-scan $d=1$, $g=0$.

Internally, Chorus uses the numpy.load function to import data from .npy files. See the function load_full_matrix_from_npy_file in the Chorus codebase.

.txt displacement data format: