Voice Flickering during RT Use

[vinod1234567890]

Hi,

Thank you for this great repo, the model and pre-trained weights.

I tried to use the model you provided (dtln_saved_model) for real-time denoising on my laptop.
It has successfully removed the background noise, but the resultant speech signal has a lot of flickering.

Please check the audio files in below links.

Using Windows OS, CPU, TF 2.2. pyaudio for real time processing.

Can you suggest what is causing this and how to avoid it?

noisy - clyp.it/opnfnagd?token=1f91ac255bf94fce0dac66f2fe2cc36c
cleaned - clyp.it/55rwflcn?token=cec87a02aa62634e7ff5da4d9e43d5c2

[breizhn]

Hi,

I added some code to illustrate, how real time processing with overlap add works. I hope that removes the flickering. I will add the script to the repo.

Best,
Nils

import soundfile as sf
import numpy as np
import tensorflow as tf

block_len = 512
block_shift = 128

# load model
model = tf.saved_model.load('.pretrained_model/dtln_saved_model')
infer = model.signatures["serving_default"]

# load audio file (please change)
audio,fs = sf.read('audioset_realrec_airconditioner_2TE3LoA2OUQ.wav')
# preallocate output audio
out_file = np.zeros((len(audio)))
# calculate number of blocks
num_blocks = (audio.shape[0] - (block_len-block_shift)) // block_shift
# iterate over the number of blcoks        
for idx in range(num_blocks):
    # take the block
    in_block = audio[idx*block_shift:(idx*block_shift)+block_len]
    # create a batch dimension of one
    in_block = np.expand_dims(in_block, axis=0).astype('float32')
    # process one block
    out_block= infer(tf.constant(in_block))['conv1d_1']
    # write block to output file
    out_file[idx*block_shift:(idx*block_shift)+block_len] = out_file[idx*block_shift:(idx*block_shift)+block_len]  + out_block
    
sf.write('out.wav', out_file, fs) 
print('Test_finished.')

[breizhn]

It could also maybe come from low level input. Try a model with normalization of the STFT features from the pretrained_model folder.

[vinod1234567890]

It is working now. Thank you! The overlap code did the trick.

However, I want to add, while processing live microphone feed, flicker is not completely gone. If I'm taking chunks of size 512, the previous issue is re-occurring. If I increase the chunk size, the flicker is far less frequent - almost negligible for an average word, but can be heard when pronouncing fairly long words or like humming a song. This issue persists even after using the norm_500h_saved_model

[breizhn]

The block length and block shift are fixed. If you would like to use another length and shift, you have to retrain the model.

Another reason for flickering can be the sound card and the library for interacting with the sound card. But I can’t deliver support on that one. Did you check if a pass through, copying the block from the microphone directly to the output, is flickering free?

[vinod1234567890]

Yes, the pass through, directly copying the input block to the output without any processing, is flickering free.

[breizhn]

Are you using 16k sampling frequency?

The model needs some CPU time and so there is maybe not enough for the audio driver. Real time audio is always tricky handling all the resources.

I added a file for real time processing today, check if you doing it similar.

After which time occurs the flickering?

The model can handle silence relatively well, but maybe integrate a gate, so the processing is only performed if the energy in the current block coming from the mic is above a threshold.

[shilsircar]

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

[vinod1234567890]

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

directly playing back after the model inference

[breizhn]

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

directly playing back after the model inference

Does it have flickering, if you write it directly to a file?

[vinod1234567890]

Are you using 16k sampling frequency?

The model needs some CPU time and so there is maybe not enough for the audio driver. Real time audio is always tricky handling all the resources.

I added a file for real time processing today, check if you doing it similar.

After which time occurs the flickering?

The model can handle silence relatively well, but maybe integrate a gate, so the processing is only performed if the energy in the current block coming from the mic is above a threshold.

Yes, using 16k sampling rate and the real-time code you posted, which made the flicker sparse, but unfortunately not totally negligible.

No issues during silence.

I think the flicker is due to the process delay before each block is glued back into the output stream. Because, it is happening at a fixed interval, and the length of this interval is directly proportional to the block size.

[vinod1234567890]

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

directly playing back after the model inference

Does it have flickering, if you write it directly to a file?

Yes. Exaclty similar to the live playback

[breizhn]

Please post the code of your loop.
Did you check against the code I added to the repo? It is a bit different to the code here in the issue.

[breizhn]

Does it work now?

[vinod1234567890]

Does it work now?

No. The earlier code, which was posted here in the issue, was giving lesser flicker. The one in the repo is resulting in a chopped kind of feel.

Here is what I'm currently using and working better than the one in the repo:

        #out_put container
        self.out_file = np.zeros((4096),dtype='float32')
        # iterate over the number of blcoks        
        for idx in range(self.num_blocks):
            # take the block
            in_block = audio[idx*self.block_shift:(idx*self.block_shift)+self.block_len] #here the audio size is 4096
            # create a batch dimension of one
            in_block = tf.expand_dims(in_block,axis=0)
            # process one block
            out_block= self.infer(in_block)['conv1d_1']
            # write block to output file
            self.out_file[idx*self.block_shift:(idx*self.block_shift)+self.block_len] = self.out_file[idx*self.block_shift:(idx*self.block_shift)+self.block_len]  + out_block
        out_bytes = self.out_file.tobytes()

[breizhn]

I added a file: real_time_dtln_audio.py. Check it out. The audio works perfectly on my really old Macbook.