Skip to content

Added RNNs

Latest

Choose a tag to compare

@felixbur felixbur released this 16 Oct 13:00
· 39 commits to main since this release
  • add frame sequences as a data format (#471): with [FEATS] level = frame, features come as one row per frame, indexed file, start, end, frame (the frame's start time relative to the segment start); one-vector-per-sample data stays as it is. opensmile gives its low-level descriptors this way (this also fixes level = lld, which failed before), wav2vec2, HuBERT and WavLM their embeddings every 20 ms instead of their mean, and frame features can be imported from CSV ([FEATS] type = ['import'], recognized by a frame column). Models that take sequences get the frames; every other model gets them pooled to one vector per sample with [FEATS] pooling (default ['mean', 'std']), so frame features work with all models, including predict_sample, bundles and explore. Feature scaling (also per speaker) works on frames, [FEATS] balancing gives a clear error. Frame-level labels use the same layout; nkululeko.utils.sequences.align_frame_labels brings labels with another frame rate to the feature frames (training on frame-level labels is not supported yet). Frame features are cached separately, and frame-level runs get _level-frame (and the pooling statistics) in their name. Documentation: docs/source/sequence_data.md
  • add recurrent-network model types [MODEL] type = lstm (classification) and lstm_reg (regression) over frame sequences ([FEATS] level = frame): a stacked LSTM whose outputs are averaged over each utterance's frames, the label stays one value per utterance. Sequences keep their own length (padded batches, packed sequences, masked averaging). Settings go in a new [RNN] section (hidden_size, num_layers, bidirectional for a BLSTM, batch_size), named for the network family so a later GRU type can reuse it; the [RNN] keys that are set go into the automatic run name. Both types support DANN and SAM; predict_sample, SHAP and ONNX export are not supported yet. Example: examples/exp_emodb_lstm.ini, tutorial: docs/source/rnn_classification.md (#256)
  • fix bug: mlp and cnn named the probability columns after the classes present in the test split instead of the network's outputs, so a test split lacking a class got mislabelled columns and lost one (#472)