Conversion for TF op 'RFFT' not implemented

[longluu]

• Name of layer type: rfft
• Is this a PyTorch or a TensorFlow layer type: TensorFlow
• Your version of coremltools: 6.01b
• Your version of TensorFlow: 2.8.0
• Impact of supporting this layer type: I'm trying to convert a pretrained yamnet model, which is a hugely popular sound classification model (https://tfhub.dev/google/yamnet/1). This is the error I encountered:

NotImplementedError: Conversion for TF op 'RFFT' not implemented.

name: "yamnet_frames/tf.signal.stft/stft/rfft"
op: "RFFT"
input: "yamnet_frames/tf.signal.stft/stft/rfft/Pad"
input: "yamnet_frames/tf.signal.stft/stft/rfft/fft_length"
attr {
key: "Tcomplex"
value {
type: DT_COMPLEX64
}
}
attr {
key: "Treal"
value {
type: DT_FLOAT
}
}

Thanks for helping.

[TobyRoseman]

Related to (and possible a duplicate of) #941. @longluu - are you trying to convert a TensorFlow 1 model or a TensorFlow 2 model?

Also related to #1311. As mentioned in that issue, I don't think we're going to be able to support RFFT without new MIL ops (i.e. changes to the Core ML Framework).

[longluu]

Thanks @TobyRoseman for the prompt reply. I'm trying to convert a TensorFlow 2 model. Issue #941 is a similar issue but for TensorFlow 1 so I just open a new one here in case it is different for Tensorflow 2.

Thanks for referring to issue #1311.

So is there a function in coremltools that can provide log mel spectrogram from an audio waveform?

[KostyaAtarik]

I've faced the same problem of missing support for rfft/irfft ops and complex numbers in coreml while trying to convert spleeter model from deezer to coreml. Worked it around by implementing rfft, irfft and abs for complex numbers as custom ops and representing complex numbers as additional dimention of size 2 in float-typed tensor.

[RahulBhalley]

I've faced the same problem of missing support for rfft/irfft ops and complex numbers in coreml while trying to convert spleeter model from deezer to coreml. Worked it around by implementing rfft, irfft and abs for complex numbers as custom ops and representing complex numbers as additional dimention of size 2 in float-typed tensor.

That's awesome @KostyaAtarik!!! Could you please share the code, if possible? Many (including me) going through this struggle from a long time will benefit from this.

Thanks,
Rahul Bhalley

[RahulBhalley]

Following scipy.fft.fft docs I'm implementing it in NumPy for simplicity:

import numpy as np

def my_fft(x):
  y = np.array([0] * n, dtype=complex)
  for k in range(n):
    y[k] = np.sum(x * np.exp(-2j * np.pi * k * np.arange(n) / n))
  return y

Let's compare the results with torch.fft.fft:

import torch

n = 10
x = np.random.randn(n)

print(my_fft(x))
print(torch.fft.fft(torch.from_numpy(x)))

Prints

array([-3.71954511+0.00000000e+00j,  4.91026958+4.07203035e+00j,
        2.13874859-2.39957690e+00j,  2.3043132 +1.56328625e-02j,
       -0.6578261 -3.33422061e+00j, -0.19899026-1.69282808e-15j,
       -0.6578261 +3.33422061e+00j,  2.3043132 -1.56328625e-02j,
        2.13874859+2.39957690e+00j,  4.91026958-4.07203035e+00j])

tensor([-3.7195+0.0000j,  4.9103+4.0720j,  2.1387-2.3996j,  2.3043+0.0156j,
        -0.6578-3.3342j, -0.1990+0.0000j, -0.6578+3.3342j,  2.3043-0.0156j,
         2.1387+2.3996j,  4.9103-4.0720j], dtype=torch.complex128)

We get correct results with our custom NumPy implementation.

Next goal is to code torch.fft.rfft and torch.fft.irfft in NumPy.

[roimulia2]

@longluu I might have a solution for it, can you send me the conversion code you used? (CoreML)

[longluu]

Thanks @roimulia2, here's the code I used for conversion:

import tensorflow as tf # TF 2.6.0
import coremltools as ct
import src.params as yamnet_params
import src.yamnet as yamnet_model
# from src.yamnet_utils import *

# Set the sample rate, window length and step length
params = yamnet_params.Params(sample_rate=16000, patch_window_seconds=0.6, patch_hop_seconds=0.1)

# Set up the YAMNet model.
yamnet = yamnet_model.yamnet_frames_model(params)
yamnet.load_weights('../models/yamnet.h5')

# Load class labels (from a separate file)
class_names = yamnet_model.class_names('../models/yamnet_class_map.csv')

I also attach the model and class labels files here.

Archive.zip

Thanks a lot for your help.

[junpeiz]

TensorFlow RFFT is supported in coremltools 6.2.