Skip to content
Code for our paper "Acoustic Features Fusion using Attentive Multi-channel Deep Architecture" in Keras and tensorflow
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Results Delete .~lock.CLASSIFICATION.ppt# Oct 14, 2018
architecture Updated the architecture Oct 14, 2018
LICENSE Added License template Oct 14, 2018
README.md Added paper link Nov 23, 2018
environment.yaml Initial code upload 💥 Oct 12, 2018
feature_extraction.py Updated features file Oct 14, 2018
mainfile.py Final commit 💥 Oct 17, 2018
mainfile_classify.py Final commit 💥 Oct 17, 2018
models.py Added description Oct 14, 2018
prepare.py
utils.py Final commit 💥 Oct 17, 2018
utils_classify.py Final commit 💥 Oct 17, 2018

README.md

Acoustic-Feature-Fusion @Chime18

Code for the paper accepted at Interspeech-Chime' 2018

Setup and Dependencies

Our code is implemented in Keras (v2.0.8 with CUDA). To setup, do the following:

If you do not have any Anaconda or Miniconda distribution, head over to their downloads' site before proceeding further.

Clone the repository and create an environment.

git clone https://github.com/DeepLearn-lab/Acoustic-Feature-Fusion_Chime18.git
conda env create -f environment.yaml

This creates an environment named kerasenv2 with all the dependencies installed.

Download the Dataset

Download the CHiME-Home, Acoustic Scene and LITIS-Rouen datasets and unzip the required datasets.

Code Usage

In the following steps I am going to show how to run the code on the given dataset. (This is what you would need to do for your own audio classification and tagging dataset as well.) The scripts that I am going to mention in each step, please open them and see if you need to the paths.

Extracting Audio features

To extract audio features, run the following:

python feature_extraction.py

The extracted features will get saved at Fe/ and Fe_eva/ folders.

Training

Training with defaut parameters works in the following way

1. Audio Tagging
- python mainfile.py

2. Classification
- python mainfile_classify.py

You can add your own module in models.py

Evaluation

  1. Audio Tagging
y=[]
y_pred_new = []
class_eer=[]
p_y_pred = lrmodel.predict([X0,X1,X2]) # probability, size: (n_block,label)

for i in range(0,len(p_y_pred),12):
    y_pred = np.mean(p_y_pred[i:i+12],axis=0)
    y_pred_new.append(y_pred)
    y.append(te_y2[i].tolist())
y_pred_new, y = np.array(y_pred_new), np.array(y)
print y_pred_new.shape
print y.shape

n_out = y.shape[1]
print y_pred_new.shape
print y.shape
for k in xrange(n_out):
    eer = EER(y[:, k],y_pred_new[:, k])
    print "Class ",k,'ERR ',eer
    class_eer.append(eer)
    

EER1=np.mean(class_eer)
print("EER",EER1)

Class  0 ERR  0.14377162629757783
Class  1 ERR  0.20401913875598086
Class  2 ERR  0.2398412698412699
Class  3 ERR  0.12727272727272726
Class  4 ERR  0.1588235294117647
Class  5 ERR  0.06666666666666667
Class  6 ERR  0.03994082840236687
('EER', 0.14019082666405057)
  1. Classification
y=[]
y_pred_new = []
p_y_pred = lrmodel.predict([X0,X1,X2]) # probability, size: (n_block,label)
for i in range(0,len(p_y_pred),258):
    y_pred = np.mean(p_y_pred[i:i+258],axis=0)
    y_pred_new.append(y_pred)
    y.append(te_y2[i].tolist())
y_pred_new, y = np.array(y_pred_new), np.array(y)
preds = np.argmax( y_pred_new, axis=-1 )     # size: (n_block)
b = scipy.stats.mode(preds)  # Finding the maximum value out of the array for predicting the label
pred = int( b[0] )
truth = open(cfg.eval_txt,'r').readlines()
pred.sort()
truth.sort()
pred = [i.split('\t')[1].split('\n')[0]for i in pred]
truth = [i.split('\t')[1]for i in truth]
met = metrics.classification_report(truth,pred)
val = met.split('\n')
print '\n\n'
for i in val:
    if len(i)>0:
        print i
print '\n\n'
pos,neg=0,0 
for i in range(0,len(pred)):
    if pred[i] == truth[i]:
        pos = pos+1
    else:
        neg = neg+1
print "\n\n ********************* Accuracy ********************* \n"

print "correctly classified     --> ",pos
print "not correctly classified -->",neg
print "percentage(%) accuracy   --> ",(float(pos)/float(len(pred)))*100
acc=(float(pos)/float(len(pred)))*100

Visualizing Results

We have integrated Tensorboard_logger to visualize training and mean squared error. To install tensorboard logger use :

pip install tensorboard_logger

The Training curves for vanilla model and proposed architecture for Tagging Task,

The training loss and Mean Squared Error(MSE) for the attention based system shows a steep decrease in the loss function as compared to the vanilla model.

Performance of the proposed model on Dcase, LITIS-Rouen and CHiME-Home.

Techniques DCASE ROUEN (F1) CHiME-Home
Vanilla 85.50 96.85 15.6
EF 86.1 96.36 15.0
LF 87.00 96.80 14.6
EF+LF 88.70 98.25 14.0

Contributors

License

BSD

You can’t perform that action at this time.