problem downsampling and computing NNLS chroma

[PRamoneda]

HI!! This library is awesome compared with other alternatives as meyda.

I have a problem downsampling audio before calculating the nnls chroma.

ERROR

Uncaught (in promise) 6094632 - Exception catching is disabled, this exception cannot be caught. Compile with -s DISABLE_EXCEPTION_CATCHING=0 or DISABLE_EXCEPTION_CATCHING=2 to catch.

Error is propaged after async function onClickFeatureExtractor() function.

CODE

function handleFiles(event) {
  var files = event.target.files;
  $("#src").attr("src", URL.createObjectURL(files[0]));
  document.getElementById("audio").load();
}

document.getElementById("upload").addEventListener("change", handleFiles, false);


let essentia

/* "https://freesound.org/data/previews/328/328857_230356-lq.mp3"; */

let audioData;
// fallback for cross-browser Web Audio API BaseAudioContext
const AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext();
let plotChroma;
let plotContainerId = "plotDiv";

let isComputed = false;


// callback function which compute the frame-wise HPCP chroma of input audioURL on a button.onclick event
async function onClickFeatureExtractor() {
  let audioURL = document.getElementById("audio").currentSrc;
  console.log(audioURL);
  let chromagram = [];
  // load audio file from an url

  audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
  audioData = essentia.arrayToVector(audioData);
  audioData = essentia.Resample(audioData, 44100, 8000).signal; // sample rate
  console.log(audioData);
}

$(document).ready(function() {
  
  // create EssentaPlot instance
  plotChroma = new EssentiaPlot.PlotHeatmap(
    Plotly, // Plotly.js global 
    plotContainerId, // HTML container id
    "chroma", // type of plot
    EssentiaPlot.LayoutChromaPlot // layout settings
  );

  // Now let's load the essentia wasm back-end, if so create UI elements for computing features
  EssentiaModule().then(async function(WasmModule) {

    essentia = new Essentia(WasmModule);

    // essentia version log to html div
    $("#logDiv").html(
      "<h5> essentia-" + essentia.version + " wasm backend loaded ... </h5>"
    );

    $("#logDiv").append(
      '<button id="btn" class="ui white inverted button">Compute HPCP Chroma </button>'
    );

    var button = document.getElementById("btn");

    // add onclick event handler to comoute button
    button.addEventListener("click", () => onClickFeatureExtractor(), false);
  });
});

Thank you so much!!!!

[albincorreya]

Hi @PRamoneda, thanks for reporting the issue.

The Resample algorithm uses the external dependency of libsamplerate library. But the current Essentia WASM builds were not linked with any of these third-party dependencies to make it lightweight. Hence it won't work with the current builds. Somehow we made a mistake by including it with the current JS API. We are working on it and will include support for this algorithm in the next version release.

Meanwhile, for downsampling, you could do that using native JS by directly downsampling the JS typed array from the getChannelData method of the Web Audio API. An example of this can be found here.

Hope it helps

[PRamoneda]

Thank you so much!!!.

I have another question about how can I create a vectorFloatFloat to feed ChomaNNLS.

If I call ArrayToVector with an array of VectorFloat. Should it return a vectorFloatFloat??

I get later Expected null or instance of VectorVectorFloat, got an instance of VectorFloat because logFreqSpectrum is VectorFloat

This code is inspired in https://github.com/MTG/essentia/blob/9bca80eb331efa550975d00a353e9928815a2b3f/test/src/unittests/tonal/test_nnlschroma.py

let audioURL = document.getElementById("audio").currentSrc;
  console.log(audioURL);
  // load audio file from an url

  let   audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
  // let audioData2 =  essentia.arrayToVector(audioData);
  audioData = downsample(audioData, 44100, 8000); // sample rate
 
  if (isComputed) { plotChroma.destroy(); };
  
  let frames = essentia.FrameGenerator(audioData, 1024, 512);

  // compute for overlapping frames
 let logFreqSpectrum = new Array(frames.length);
  let meanTuning = 0;
  let localTuning = 0;

  for (var i=0; i<frames.size(); i++) {
    let log_spectrum =  essentia.LogSpectrum(essentia.Spectrum(frames.get(i)).spectrum,
                                                            3, // bins per semitone
                                                            1024, //frameSize
                                                            0, // rollon
                                                            8000);// sample rate
   
    // let c = essentia.vectorToArray(log_spectrum.logFreqSpectrum);
    console.log(log_spectrum.logFreqSpectrum);
    logFreqSpectrum.push(log_spectrum.logFreqSpectrum);

    meanTuning = log_spectrum.meanTuning;
    localTuning = log_spectrum.meanTuning;
  }
  logFreqSpectrum = essentia.arrayToVector(logFreqSpectrum);
  console.log(logFreqSpectrum)

[albincorreya]

Removed the redundant comment:)

You can create VectorVectorFloat type on the JS side using

let vecvecFloat = new essentia.module.VectorVectorFloat();

arrayToVector and vectorToArray methods only works for 1D arrays/vector. For 2D vector/arrays, you need to manually convert by iterating through it.

The below example should work. Haven't tested it though!

for (var i=0; i<frames.size(); i++) {
    let log_spectrum =  essentia.LogSpectrum(essentia.Spectrum(frames.get(i)).spectrum,
                                                            3, // bins per semitone
                                                            1024, //frameSize
                                                            0, // rollon
                                                            8000);// sample rate
   
    vecvecFloat.push_back(log_spectrum.logFreqSpectrum);

    meanTuning = log_spectrum.meanTuning;
    localTuning = log_spectrum.meanTuning;

   let nnlsChroma = essentia. NNLSChroma(vecvecFloat, meanTuning, localTuning);

  // you need manually resize the 2D vector after its use
  vecvecFloat.resize(0, 1);

  console.log(nnlsChroma);
  }

[PRamoneda]

Why I need to manually resize the 2D vector after using it??? Is It due to memory allocation, isnt it??

This,

let nnlsChroma = essentia. NNLSChroma(vecvecFloat, meanTuning, localTuning);

  // you need manually resize the 2D vector after its use
  vecvecFloat.resize(0, 1);

  console.log(nnlsChroma);

have to be called after the loop. I understand. As https://github.com/MTG/essentia/blob/ba79be6515f2fd0cde75ee3f6fa98706a66f4c36/src/examples/standard_nnls.cpp#L125

However, I have imitated the cpp version and NNLSchroma compute all the pitch classes to zero.

example.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>essentia.js examples</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width,initial-scale=1" />
    <link
      rel="stylesheet"
      type="text/css"
      href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css"
    />
  </head>
  <center>
    <body style="background-color:  #000000!important;">
      <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
      <script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
      <script src="https://unpkg.com/essentia.js@0.0.9/dist/essentia-wasm.web.js"></script>
      <script src="https://unpkg.com/essentia.js@0.0.9/dist/essentia.js-core.js"></script>
      <script src="https://unpkg.com/essentia.js@0.0.9/dist/essentia.js-plot.js"></script>
      <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
      <script src="script.js" defer></script>
      <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>

      <div>
        <a>
          <h1> HARMONIC CHANGE DETECTION FUNCTION </h1>
        </a>
      </div>
      <h2 style="color: azure;">
        HPCP chroma example
      </h2>
      
      <div class="ui divider" style="height: 2px; width: 2px;"></div>

      <input type="file" id="upload" />
      <audio id="audio" controls>
        <source src="https://freesound.org/data/previews/328/328857_230356-lq.mp3" id="src" />
      </audio>

      <div id="logDiv" style="color: azure;"><br /></div>
      <div class="ui divider" style="width: 2px; height: 5px;"></div>
      <div id="plotDiv"></div>
      <br />
      <br />
    </body>
  </center>
</html>

script.js

function handleFiles(event) {
  var files = event.target.files;
  $("#src").attr("src", URL.createObjectURL(files[0]));
  document.getElementById("audio").load();
}

document.getElementById("upload").addEventListener("change", handleFiles, false);


let essentia

/* "https://freesound.org/data/previews/328/328857_230356-lq.mp3"; */

let audioData;
// fallback for cross-browser Web Audio API BaseAudioContext
const AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext();
let plotChroma;
let plotContainerId = "plotDiv";

let isComputed = false;

function downsample(buffer, old_sr, new_sr) {
    if (new_sr == old_sr) {
        return buffer;
    }
    if (new_sr > old_sr) {
        throw "downsampling rate show be smaller than original sample rate";
    }
    var sampleRateRatio = old_sr / new_sr;
    var newLength = Math.round(buffer.length / sampleRateRatio);
    var result = new Float32Array(newLength);
    var offsetResult = 0;
    var offsetBuffer = 0;
    while (offsetResult < result.length) {
        var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
        var accum = 0, count = 0;
        for (var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
            accum += buffer[i];
            count++;
        }
        result[offsetResult] = accum / count;
        offsetResult++;
        offsetBuffer = nextOffsetBuffer;
    }
    return result;
}


// callback function which compute the frame-wise HPCP chroma of input audioURL on a button.onclick event
async function onClickFeatureExtractor() {
  let audioURL = document.getElementById("audio").currentSrc;
  console.log(audioURL);
  // load audio file from an url

  let   audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
  // let audioData2 =  essentia.arrayToVector(audioData);
  audioData = downsample(audioData, 44100, 8000); // sample rate
 
  if (isComputed) { plotChroma.destroy(); };
  
  let frames = essentia.FrameGenerator(audioData, //
                                       1024, //
                                       512);

  // compute for overlapping frames
  let logFreqSpectrum = new essentia.module.VectorVectorFloat();
  let meanTuning = new essentia.module.VectorFloat();
  let localTuning = new essentia.module.VectorFloat();

  for (var i=0; i<frames.size(); i++) {
    let log_spectrum =  essentia.LogSpectrum(essentia.Spectrum(frames.get(i), 1024).spectrum,
                                                            3, // bins per semitone
                                                            1024, //frameSize
                                                            0, // rollon
                                                            8000);// sample rate

    console.log(essentia.vectorToArray(log_spectrum.logFreqSpectrum));
    logFreqSpectrum.push_back(log_spectrum.logFreqSpectrum);
    localTuning.push_back(log_spectrum.localTuning);
    //as in python script https://github.com/MTG/essentia/blob/9bca80eb331efa550975d00a353e9928815a2b3f/test/src/unittests/tonal/test_nnlschroma.py
    meanTuning = log_spectrum.meanTuning;

  }
  
  console.log(typeof logFreqSpectrum, typeof meanTuning, typeof localTuning);
  console.log(logFreqSpectrum, meanTuning, localTuning);
  
  // Running NNLSchroma algorithm on an input audio signal vector
  // check https://essentia.upf.edu/reference/std_NNLSChroma.html
  // NNLSChroma(logSpectrogram: any, meanTuning: any, localTuning: any, chromaNormalization: string='none', frameSize: number=1025, sampleRate: number=44100, spectralShape: number=0.7, spectralWhitening: number=1, tuningMode: string='global', useNNLS: boolean=true)
  let chroma = essentia.NNLSChroma( logFreqSpectrum, // input
    meanTuning,
    localTuning, 
    'none',  //chromaNormalization
    1024, //frameSize 
    8000, //sampleRate 
    0.7, //spectralShape
    1, //spectralWhitening
    'global', //tuningMode
    true).chromagram; //useNNLS

  debugger;

  let chromagram = Array(chroma.size());
  for (var i = 0; i < chroma.size(); i++) {
      console.log(essentia.vectorToArray(chroma.get(i)));
      chromagram[i] = essentia.vectorToArray(chroma.get(i));
  }
  console.log(typeof chromagram);
  // plot the feature
  plotChroma.create(
    chromagram, // input feature array
    "NNLS Chroma", // plot title
    audioData.length, // length of audio in samples
    8000 // audio sample rate
  );
  isComputed = true;

}

$(document).ready(function() {
  
  // create EssentaPlot instance
  plotChroma = new EssentiaPlot.PlotHeatmap(
    Plotly, // Plotly.js global 
    plotContainerId, // HTML container id
    "chroma", // type of plot
    EssentiaPlot.LayoutChromaPlot // layout settings
  );

  // Now let's load the essentia wasm back-end, if so create UI elements for computing features
  EssentiaModule().then(async function(WasmModule) {

    essentia = new Essentia(WasmModule);

    // essentia version log to html div
    $("#logDiv").html(
      "<h5> essentia-" + essentia.version + " wasm backend loaded ... </h5>"
    );

    $("#logDiv").append(
      '<button id="btn" class="ui white inverted button">Compute HPCP Chroma </button>'
    );

    var button = document.getElementById("btn");

    // add onclick event handler to comoute button
    button.addEventListener("click", () => onClickFeatureExtractor(), false);
  });
});

Thank you!!

[PRamoneda]

Moreover, log spectrum always throw this warning. Even with default parameters.

[0;32m[   INFO   ] LogSpectrum: input spectrum size does not match '_frameSize' parameter. Reconfiguring the algorithm.

Thank you so much!!!!

[albincorreya]

Yes, NNLSChroma, accepts frames of log spectrum as input. Also, you may need to apply windowing to each frame before computing the log spectrum. The documentation suggests the following:

This code is ported from NNLS Chroma [1, 2]. To achieve similar results follow this processing chain: frame slicing with sample rate = 44100, frame size = 16384, hop size = 2048 -> Windowing with Hann and no normalization -> Spectrum -> LogSpectrum.

References:
[1] Mauch, M., & Dixon, S. (2010, August). Approximate Note Transcription for the Improved Identification of Difficult Chords. In ISMIR (pp. 135-140).
[2] Chordino and NNLS Chroma, http://www.isophonics.net/nnls-chroma

For example, try this out

const frameSize = 16384;
const hopSize = 2048;

let frames = essentia.FrameGenerator(audioData, 
                                    frameSize, 
                                    hopSize)
let logSpectFrames = new essentia.module.VectorVectorFloat();

for (var i=0; i<frames.size(); i++) {
    // default hanning window (you can change it according to your need)
    let windowing = essentia.Windowing(frame.get(i), false, 1024, 'hann');
    let spect = essentia.Spectrum(windowing.frame, frameSize); // frameSize
    let logSpectrum =  essentia.LogSpectrum(spect.spectrum,
                                           3, // bins per semitone
                                           frameSize
                                           0, // rollon
                                           8000);// sample rate
   
    logSpectFrames.push_back(logSpectrum.logFreqSpectrum);

    meanTuning = logSpectrum.meanTuning;
    localTuning = logSpectrum.localTuning;
}

let nnlsChroma = essentia. NNLSChroma(logSpectFrames, meanTuning, localTuning);

delete windowing:
delete spect;
delete logSpectrum;

Regarding memory allocation, you may need to manually delete any JS objects created from Essentia algorithms as Emscripten documentation suggests. Check here for more details.

Another tip, it might be good for the web app to run your audio feature extraction process inside Web Workers to achieve better performance.

[PRamoneda]

Thank you so much!!!

But it is not working :(. NNLSchroma doesnt compute anything. Results of NNLS are 0 too.

Here a web editor with everything. https://jsfiddle.net/PRamoneda/zc1bnhxk/2/

HTML file, JS file and console output screenshot

script.js

function handleFiles(event) {
  var files = event.target.files;
  $("#src").attr("src", URL.createObjectURL(files[0]));
  document.getElementById("audio").load();
}

document.getElementById("upload").addEventListener("change", handleFiles, false);


let essentia

/* "https://freesound.org/data/previews/328/328857_230356-lq.mp3"; */

let audioData;
// fallback for cross-browser Web Audio API BaseAudioContext
const AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext();
let plotChroma;
let plotContainerId = "plotDiv";

let isComputed = false;

function downsample(buffer, old_sr, new_sr) {
    if (new_sr == old_sr) {
        return buffer;
    }
    if (new_sr > old_sr) {
        throw "downsampling rate show be smaller than original sample rate";
    }
    var sampleRateRatio = old_sr / new_sr;
    var newLength = Math.round(buffer.length / sampleRateRatio);
    var result = new Float32Array(newLength);
    var offsetResult = 0;
    var offsetBuffer = 0;
    while (offsetResult < result.length) {
        var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
        var accum = 0, count = 0;
        for (var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
            accum += buffer[i];
            count++;
        }
        result[offsetResult] = accum / count;
        offsetResult++;
        offsetBuffer = nextOffsetBuffer;
    }
    return result;
}


// callback function which compute the frame-wise HPCP chroma of input audioURL on a button.onclick event
async function onClickFeatureExtractor() {
  let audioURL = document.getElementById("audio").currentSrc;
  console.log(audioURL);

  // load audio file from an url
  let audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
 
  if (isComputed) { plotChroma.destroy(); };
  
  const frameSize = 16384;
  const hopSize = 2048;
  const sampleRate = 8000;

  console.log("audio antes downsampling", audioData);
  audioData = downsample(audioData, 44100, sampleRate); 
  console.log("audio despues downsampling", audioData);
  let frames = essentia.FrameGenerator(audioData, 
                                      frameSize, 
                                      hopSize)
  let logSpectFrames = new essentia.module.VectorVectorFloat();

  for (var i=0; i<frames.size(); i++) {
      // default hanning window (you can change it according to your need)
      let windowing = essentia.Windowing(frames.get(i), false, hopSize, 'hann');
      let spect = essentia.Spectrum(windowing.frame, frameSize); // frameSize
      let logSpectrum =  essentia.LogSpectrum(spect.spectrum,
                                             3, // bins per semitone
                                             frameSize,
                                             0, // rollon
                                             sampleRate);// sample rate
     
      logSpectFrames.push_back(logSpectrum.logFreqSpectrum);

      meanTuning = logSpectrum.meanTuning;
      localTuning = logSpectrum.meanTuning;
  }

  let nnlsChroma = essentia.NNLSChroma(logSpectFrames, meanTuning, localTuning).chromagram;

  delete windowing;
  delete spect;
  delete logSpectrum;
  
  for (var i = 0; i < nnlsChroma.size(); i++)
      console.log(essentia.vectorToArray(nnlsChroma.get(i)));

  // plot the feature
  plotChroma.create(
    nnlsChroma, // input feature array
    "NNLS Chroma", // plot title
    audioData.length, // length of audio in samples
    sampleRate // audio sample rate
  );
  isComputed = true;

  delete nnlsChroma;

}

$(document).ready(function() {
  
  // create EssentaPlot instance
  plotChroma = new EssentiaPlot.PlotHeatmap(
    Plotly, // Plotly.js global 
    plotContainerId, // HTML container id
    "chroma", // type of plot
    EssentiaPlot.LayoutChromaPlot // layout settings
  );

  // Now let's load the essentia wasm back-end, if so create UI elements for computing features
  EssentiaModule().then(async function(WasmModule) {

    essentia = new Essentia(WasmModule);

    // essentia version log to html div
    $("#logDiv").html(
      "<h5> essentia-" + essentia.version + " wasm backend loaded ... </h5>"
    );

    $("#logDiv").append(
      '<button id="btn" class="ui white inverted button">Compute HPCP Chroma </button>'
    );

    var button = document.getElementById("btn");

    // add onclick event handler to comoute button
    button.addEventListener("click", () => onClickFeatureExtractor(), false);
  });
});

Example.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>essentia.js examples</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width,initial-scale=1" />
    <link
      rel="stylesheet"
      type="text/css"
      href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css"
    />
  </head>
  <center>
    <body style="background-color:  #000000!important;">
      <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
      <script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
      <script src="https://unpkg.com/essentia.js@0.0.9/dist/essentia-wasm.web.js"></script>
      <script src="https://unpkg.com/essentia.js@0.0.9/dist/essentia.js-core.js"></script>
      <script src="https://unpkg.com/essentia.js@0.0.9/dist/essentia.js-plot.js"></script>
      <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
      <script src="script.js" defer></script>
      <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>

      <div>
        <a>
          <h1> HARMONIC CHANGE DETECTION FUNCTION </h1>
        </a>
      </div>
      <h2 style="color: azure;">
        HPCP chroma example
      </h2>
      
      <div class="ui divider" style="height: 2px; width: 2px;"></div>

      <input type="file" id="upload" />
      <audio id="audio" controls>
        <source src="https://freesound.org/data/previews/328/328857_230356-lq.mp3" id="src" />
      </audio>

      <div id="logDiv" style="color: azure;"><br /></div>
      <div class="ui divider" style="width: 2px; height: 5px;"></div>
      <div id="plotDiv"></div>
      <br />
      <br />
    </body>
  </center>
</html>

Here the console output:

[albincorreya]

Okay, I just saw that this is a known issue with the NNLS chroma algorithm. See issue MTG/essentia#951 and MTG/essentia#948.

So you need to change the parameter useNNLS=False.

Btw, please only post the necessary code snippet in the comments. No need to share all of your web app code in the comments unless it is related to the issue. Sharing a link to a web editor is enough. In that way, it would be easier for others to find the information in these threads :)

Thanks for reporting the issue.

Hope this helps, cheers!

[PRamoneda]

Ok thank you so much especially for all your tips And then nnlschroma doesn’t have a first symbolic transcription approach, does it??

[albincorreya]

According to the comments in the above-mentioned issue threads, the NNLS symbolic transcription approach is not fully tested and guaranteed to work in every use-case.

Let me know if this setting works for you in the web application.

[PRamoneda]

It is working yeah!!

But without nnls symbolic transcription. I am raising to use CQT chroma :S. I am testing CQT, HPCP and NNLS.

Thank you other time!!

[floydback]

Does Essentia WASM builds includes Resample method for now?

[jmarcosfer]

@floydback It is included in the build, but it does not work yet, sorry. You can try writing the resampling function yourself as suggested above, using a reference like this one or the downsample function implemented by the OP. You can also use an OfflineAudioContext to do the resampling for you (see this StackOverflow answer)