End to End Example

In addition to the DetEdit examples in Getting Started, the following is an additional example showing a complete end-to-end application of DetEdit, from audio file to annotate and remove undesired detections.

The following steps will guide you through an example of acoustic data with a beaked whale encounter, and how to detect beaked whale echolocation clicks applying a generic energy detector, create files required for the DetEdit interface, annotate the data, and remove false detections.

Download the EndToEnd_Example_Test_Set folder from Test datasets Drive folder, and learn how to download and set up DetEdit repository in the quick setup section.

Content

mkLtsa: Create LTSA file from a collection of acoustic recordings
Create TPWS files
1. Edetect: Apply generic energy detector to detect signals from acoustic recordings
2. make_TPWS: Provide detection times to create TPWS file
mkLTSAsessions: Create LTSA sessions files
detEdit: Launch DetEdit interface to annotate detections
modDet: Clean up of TPWS files

1. `mkLtsa`: Create LTSA file from a collection of acoustic recordings

An example audio file ( DCPP01C_DL67_130205_054500.x.wav) is provided in EndToEnd_Example_Test_Set folder. Use the Command Window in MATLAB to create the .ltsa file of the example audio file as follows:

>> mkLtsa

and specify the audio file type: press 1 for WAV files or 2 for XWAV files.

filetype .

It will prompt you to specify a folder containing a collection of audio files, in this case, select the EndToEnd_Example_Test_Set folder.

After, set up the LTSA parameters:

Time Average: length of time for each spectral average
Frequency Bin Size for LTSA

LTSAparams

Since these parameters are data sample rate dependent, setup with caution. The default parameters for broadband data sampled at 200 kHz are usually 5s averages and 100 Hz frequency bins. For other sample rates, such as 48 kHz or 96 kHz, 5s averages and 48 Hz frequency bins are recommended.

Then, select the folder to save the .ltsa file.

It is recommended to choose the same folder as the audio files so that the LTSA can provide a link and index to these fine-scale data files. The amount of time required to create the LTSA is over several hours for multiple audio files.

While running, the user will be presented with progress information.

Click here to visualize printed information in Command Window

>> mkLtsa
1 data files for LTSA
Total number of raw files: 30
LTSA version 4
Number of samples for fft: 1000
XWAV to LTSA Compression Factor: 1000
  
Opened File: 
G:\EndToEnd_Example_Test_Set\DCPPC01.ltsa
Completed Processing audio file 1
Time to calculate 450 spectra is 119.0585

2. Create TPWS files

i. `Edetect`: Apply generic energy detector to detect signals from acoustic recordings

Edetect applies a simple energy detector on audio files. User is required to select a folder or multiple folders containing audio files, provide a spreadsheet containing detector parameters, transfer function, and a spreadsheet containing the times to search for signals.

An example of each file required to execute the generic energy detector are provided in the EndToEnd_Example_Test_Set folder:

Detector parameter file:BeakedWhale_parameters_file.xlsx
Event time file: DCPP01C_event_times.xlsx
Transfer function file: 682_120919_invSensit.tf

Detector parameter files:

The detector parameters to specify:

Parameter	Value	Description
Sample rate (#/sec)	200000	Your data sample rate
Bpf low freq (Hz)	10000	Band-pass filter low frequency
Bpf high freq (Hz)	95000	Band-pass filter high frequency
Threshold (dB_pp re uPa)	121	Minimum received level threshold
Frequency for transfer function (Hz)	40200	Calibrate data corresponding to peak frequency of signal on-axis
Time lockout btw detections (ms)	30	Minimum time for next signal to be detected
Pulse length (ms)	1	Maximum signal length to be detected
Un-filtered data	0	Store unfiltered spectra of detections

Event time file:

A spreadsheet containing start and end times (in serial date number format) to search for signals. DCPP01C_event_times.xlsx contains the start and end times of the audio data. Or multiple start and end times can be given to localized the search.

Transfer function:

The transfer function file (682_120919_invSensit.tf) containing preamp gain at different frequencies based on the individual hydrophone calibration.

Execute generic energy detector:

You can execute Edetect providing the files and folders directly in the function call as follows:

>> Edetect('paramFile','G:\EndToEnd_Example_Test_Set\BeakedWhale_parameters_file.xlsx',...
    'tfFile','G:\EndToEnd_Example_Test_Set\682_120919_invSensit.tf',...
    'timeFile','G:\EndToEnd_Example_Test_Set\DCPP01C_event_times.xlsx',...
    'audioDir','G:\EndToEnd_Example_Test_Set',...
    'outputFileName','DCPP01C_Cuviers',...
    'fileType',2,... 
    'channel',1)

Multiple folders can be specified to search for audio files by giving audioDir the multiple paths as 'audioDir',{'G:\EndToEnd_Example_Test_Set';'G:\Other_folder'},...

or through GUI tools that will prompt the user to select the files and folders:

>> Edetect

It would assume that audio files provided would be of extended wave file format (.x.wav). If you still want to execute it through GUI tools for .wav file format, call it as follows:

>> Edetect('fileType',1)

Click here to visualize printed information in Command Window

>> Edetect

Parameter File: G:\EndToEnd_Example_Test_Set\BeakedWhale_parameters_file.xlsx
Loading Parameter File
Transfer Function: G:\EndToEnd_Example_Test_Set\682_120919_invSensit.tf
Loading Transfer Function File

Event Time File: G:\EndToEnd_Example_Test_Set\DCPP01C_event_times.xlsx
Load event time file

No file type specified. Reading pseudo-wav (x.wav) files.

G:\EndToEnd_Example_Test_Set\DCPP01C_Cuviers_Edetect_output.txt

Number of Events to run Edetector : 1

 
1  audio data files in directory G:\EndToEnd_Example_Test_Set
 
Number of data chunks to evaluate 28
Detecting in chunk 1 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 2 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 3 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 31
Detecting in chunk 4 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 263
Detecting in chunk 5 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 218
Detecting in chunk 6 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 215
Detecting in chunk 7 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 246
Detecting in chunk 8 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 264
Detecting in chunk 9 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 268
Detecting in chunk 10 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 270
Detecting in chunk 11 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 248
Detecting in chunk 12 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 109
Detecting in chunk 13 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 42
Detecting in chunk 14 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 14
Detecting in chunk 15 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 102
Detecting in chunk 16 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 87
Detecting in chunk 17 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 2
Detecting in chunk 18 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 19 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 20 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 21 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 22 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 23 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 24 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 25 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 26 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 27 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 28 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
 
Number of Detections in Event = 2379

TPWS file saved in G:\EndToEnd_Example_Test_Set\DCPP01C_Cuviers_TPWS1.mat
Total Number of Detections for All Events = 2379
 
Runtime : 20.4387 seconds

Edetect stores the received levels and spectra in the TPWS file without the preamp gain. DetEdit, in this case, will ask the user to provide the transfer function.

The summary printed information in the Command Window will be stored automatically in a text file (e.g. DCPP01C_Cuviers_Edetect_output.txt).

ii. `make_TPWS`: Provide detection times to create TPWS file

The input to the DetEdit GUI is a Matlab binary file (*TPWS1.mat), which contains matrices of detection start times, peak-to-peak received levels, waveform, and spectra parameters from acoustic detections. These matrices can be created manually by the user or with make_TPWS. An example detection files with detections are provided in EndToEnd_Example_Test_Set folder to test and learn how to use make_TPWS.

Example file containing start times and waveform of each detection

The user is required to modify make_TPWS to match the variable names provided from a detection file (.mat) and the directory paths. The minimum required variables in the detection file to create a TPWS file are:

detection start times
waveform for each detection
transfer function: preamp gain per frequency

The detection file (DCPP01C_beakedwhale_detections_example.mat) contains 3 variables:

detTimes vector of detection start times
detWaveform is matrix of detection time series.
transferFunction matrix with frequency and gain values.

Spectra matrices and received levels (dB_pp re 1uPa) for each detection would be computed. The user is required to specify fftLength and sampleRate. It is recommended to specify bandPassEdges as well.

Modify make_TPWS as follows:

% Define input/output locations
inDir = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing detection files.
myFileFlag = 'DCPP01C_beaked_whale_detections_example1.mat'; % include a string to match for identifying the files to be processed.
iterationNum = 1; % Iteration number, usually 1 when creating TPWS files from scratch.
saveDir = 'G:\EndToEnd_Example_Test_Set';
outputFileName = 'DCPP01C_Cuviers';

%% Begin calculations
% Find all the files to be processed
detectionFileList = dir(fullfile(inDir,myFileFlag));

% Initialize variables
fnew = [];
MSP = [];
MSN = [];
MTT = [];
MPP = [];

% Iterate over each file, collecting and calculating the various
% parameters.
for iFile = 1:length(detectionFileList)
    % load each detection file
    thisDetFile = fullfile(detectionFileList(iFile).folder,...
        detectionFileList(iFile).name);
    load(thisDetFile)
    
    [MPPnew,MTTnew,MSPnew,MSNnew,fnew] = make_TPWS_vars('timeSeries',detWaveform,...
        'detectionTimes',detTimes,'fftLength',200,'sampleRate',200000,'bandPassEdges',[5 95],...
        'tfFunFrequency',transferFunction(:,1),'tfFunValues',transferFunction(:,2));
    
    MPP = [MPP;MPPnew];
    MTT = [MTT;MTTnew];
    MSN = [MSN;MSNnew];
    MSP = [MSP;MSPnew];
    f   = fnew;
    
    % Optionally can add logic in here for rolling into a new output file
    % if things get too large (>1.5 million detections is a rough estimate 
    % of max desirable file size). Preallocating space for MSP and MSN will
    % improve performance.
end

% Save output
fullOutputFileName = [outputFileName,'_TPWS',num2str(iterationNum),'.mat'];
save(fullfile(saveDir,fullOutputFileName),'MPP','MSP','MSN','MTT','f','-v7.3')

After executing the modified make_TPWS, a TPWS file (e.g. DCPP01C_Cuviers_TPWS1.mat) will be created in the specified folder.

3. `mkLTSAsessions`: Create LTSA sessions files

After DCPP01C_Cuviers_TPWS1.mat has been created, the following step is making the snippets of LTSAs corresponding to bouts of detections.

Create data Settings Script - EndToEnd_Example_Settings

A data settings script is required to create LTSAs snippets and annotate data. An example script (EndToEnd_Example_Settings.m) based on the beakedwhaleSettings file is created for this example and also provided in the EndToEnd_Example_Test_Set folder:

% EndToEnd_Example_Settings.m
% Based of beakedwhaleSettings

% Example script to define directories and parameter preference for the
% interface. You can make different versions of this, with different names
% for different species or sites.

% Define input/output locations.REQUIRED
filePrefix = 'DCPP01C'; % TPWS file name to match. 
% Optional, replace file prefix to a more generic name to specify settings 
% for mkLTSAsessions or modDet, it will run in multiple files.
% Example: GofMX_DT03 (will run modDet to all files matching the generic name) 
iterationNum = '1'; % iteration number
sampleRate = 200; % replace with your sample rate
sp = 'Zc'; % species code 
% Example:  '' (Unknown), 'De' (Dolphin), 'Pm' (sperm whale)
% (See comments in initSpParams.m for more species codes)
tpwsDir = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing TPWS files

% Specific input/output locations (comment them if not in use)
% tfName = 'E:\MyTFfolder'; % identify folder containing transfer function 
% files (.tf). Required if spectra has not been calculated peak to received levels 

ltsaDir = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing ltsa files (.ltsa)
% REQUIRED to run mkLTSAsessions.m


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Setting preferences to override defaults parameters for the interface
% (Uncomment these in as needed to override detEdit defaults defined at
% initDefaultParams.m and initSpParams.m)

%% Bout preferences
% paramsUser.threshRL = 130; % minimum RL threshold in dB peak-to-peak
paramsUser.tfSelect = 40200; % freq used for transfer function, leave at 0 if no adjustment
paramsUser.c4fd = 100; % Detections step size to estimate false detection rate
paramsUser.threshRMS = 58; % default for < command, RMS threshold cutoff

If amplitude has not been adjusted for the preamp gain to received levels in the TPWS files, the user has to specify in the EndToEnd_Example_Settings file where to find the transfer function file and the frequency to calculate the peak-to-peak amplitude to received levels. Spectra are adjusted for all the corresponding frequencies. Add the following lines in EndTo End_Example_Settings file as follows:

line 19: % Specific input/output locations (comment them if not in use)
line 20: tfName = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing transfer function 
line 21: % files (.tf). Required if spectra have not been calculated peak to received levels 

line 38: paramsUser.tfSelect = 40200; % freq used for transfer function, leave at 0 if no adjustment

Using EndToEnd_Example_Settings.m, the LTSA snippets for each detection bout is performed as follows:

>> mkLTSAsessions(@EndToEnd_Example_Settings)

This will work if EndToEnd_Example_Test_Set folder is added in the Matlab set path

or you can invoke the function directly

>> mkLTSAsessions

and it will prompt you with a window to select the data settings script.

While running, the user will be presented with progress information.

Click here to visualize printed information in Command Window

>> mkLTSAsessions
No TF Applied 
 Removed too low:148
reading ltsa headers, please be patient ...
done reading ltsa headers
Number Bouts : 1
Session: 1  Start: 05-Feb-2013 05:45:00  End:05-Feb-2013 06:22:30   Duration: 975.4317 sec
Done with file G:\EndToEnd_Example_Test_Set\DCPP01C_Cuviers_TPWS1.mat
Elasped Time : 13.6579 s

4. `detEdit`: Launch DetEdit interface to annotate detections

Following the previous three steps, the user should have the following files required to invoke the DetEdit interface:

DCPP01C_Cuviers_TPWS1.mat
DCPP01C_Cuviers_LTSA1.mat
EndToEnd_Example_Settings.m

Invoke the interface as follows:

>> detEdit(@EndToEnd_Example_Settings)

Automatically the annotation files (DCPP01C_Cuviers_FD1.mat, DCPP01C_Cuviers_TD1.mat, DCPP01C_Cuviers_ID1.mat) are created in the same folder, where the annotations will be store accordingly.

The user will be asked to select the starting session, press 1 to start with the first bout.

Click here to visualize printed information in Command Window

>> detEdit(@EndToEnd_Example_Settings)
No TF Applied
Made new FD file
Made new ID file
 Removed too low:148
 Removed 1 detections that do not match detection times
Number Bouts : 1
Loading LTSA Sessions, please wait ...
Done Loading LTSA Sessions
 Make new TD file
Starting Session:

beaked_whale_example_results_make_TPWS

A batch of detections are annotated as false positives using the MATLAB's brushing tool as follows:

activate the brush by selecting Brush button .
select detections by flagging the area of interest (highlighted in yellow).
press r to annotate selection as false positive, detections will be displayed in red as well as averaged sound pressure waveform and power spectra of the annotated detections. Detection times will be automatically saved in the corresponding *FD.mat file.

brushing_colors_numbers

The selection performed here is just for the purpose of showing how signals are annotated as false detections.

5. `modDet`: Clean up of TPWS files

The user can remove the annotated detections as false positive detections stored in DCPP01C_Cuviers_FD1.mat file from the DCPP01C_Cuviers_TPWS1.mat files as follows:

>> modDet(@EndToEnd_Example_Settings)

Click here to visualize printed information in Command Window

>> modDet(@EndToEnd_Example_Settings)
Make new folder: G:\EndToEnd_Example_Test_Set\TPWS2
No TF Applied
 Removed too low:148
Number of Starting Detections = 2231
Number of Final Detections = 2203
Save G:\EndToEnd_Example_Test_Set\TPWS2\DCPP01C_Cuviers_ID2.mat
Done Modifying File
Number Bouts : 1
Save G:\EndToEnd_Example_Test_Set\TPWS2\DCPP01C_Cuviers_TPWS2.mat
No parameters calculated

This will create DCPP01C_Cuviers_TPWS2.mat file in a new folder (TPWS2 folder) within the current folder.

Visualize cleaned up TPWS in DetEdit

The user can visualize the new TPWS2 files by changing the iteration number to 2 in the EndToEnd_Example_Settings as follows:

line 12: iterationNum = 2;

After, create the LTSA snippets for the DCPP01C_Cuviers_TPWS2.mat file:

>> mkLTSAsessions(@EndToEnd_Example_Settings)

Click here to visualize printed information in Command Window

>> mkLTSAsessions(@EndToEnd_Example_Settings)
No TF Applied 
 Removed too low:0
reading ltsa headers, please be patient ...
done reading ltsa headers
Number Bouts : 1
Session: 1  Start: 05-Feb-2013 05:45:00  End:05-Feb-2013 06:22:30   Duration: 975.4317 sec
Done with file G:\EndToEnd_Example_Test_Set\TPWS2\DCPP01C_Cuviers_TPWS2.mat
Elasped Time : 7.5421 s

Invoke the interface again:

>> detEdit(@EndToEnd_Example_Settings)

Click here to visualize printed information in Command Window

>> detEdit(@EndToEnd_Example_Settings)
No TF Applied
Made new FD file
 Removed too low:0
 Removed 1 detections that do not match detection times
Number Bouts : 1
Loading LTSA Sessions, please wait ...
Done Loading LTSA Sessions
 Make new TD file
Starting Session:  1
Press 'b' key to go backward 
Press any other key to go forward
 BEGIN SESSION: 1
 Detection times:2203
 Test False Detection times:23
 No False Detections
 No identified detections (ID)
 True Detections: 2203
END SESSION: 1 Start: 05-Feb-2013 05:48:18 End:05-Feb-2013 06:04:33

beaked_whale_example_results_itr2

Notice that the annotated false detections have been removed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

End to End Example

Content

1. `mkLtsa`: Create LTSA file from a collection of acoustic recordings

2. Create TPWS files

i. `Edetect`: Apply generic energy detector to detect signals from acoustic recordings

Detector parameter files:

Event time file:

Transfer function:

Execute generic energy detector:

ii. `make_TPWS`: Provide detection times to create TPWS file

Example file containing start times and waveform of each detection

3. `mkLTSAsessions`: Create LTSA sessions files

Create data Settings Script - EndToEnd_Example_Settings

4. `detEdit`: Launch DetEdit interface to annotate detections

5. `modDet`: Clean up of TPWS files

Visualize cleaned up TPWS in DetEdit

Clone this wiki locally

End to End Example

Content

1. mkLtsa: Create LTSA file from a collection of acoustic recordings

2. Create TPWS files

i. Edetect: Apply generic energy detector to detect signals from acoustic recordings

Detector parameter files:

Event time file:

Transfer function:

Execute generic energy detector:

ii. make_TPWS: Provide detection times to create TPWS file

Example file containing start times and waveform of each detection

3. mkLTSAsessions: Create LTSA sessions files

Create data Settings Script - EndToEnd_Example_Settings

4. detEdit: Launch DetEdit interface to annotate detections

5. modDet: Clean up of TPWS files

Visualize cleaned up TPWS in DetEdit

Clone this wiki locally

1. `mkLtsa`: Create LTSA file from a collection of acoustic recordings

i. `Edetect`: Apply generic energy detector to detect signals from acoustic recordings

ii. `make_TPWS`: Provide detection times to create TPWS file

3. `mkLTSAsessions`: Create LTSA sessions files

4. `detEdit`: Launch DetEdit interface to annotate detections

5. `modDet`: Clean up of TPWS files