[GSoC] FaustNet: Enabling ML in Faust

[FloofCat]

Google Summer of Code -- FaustNet: Enabling ML in Faust

The goal of the project is to introduce a foundational library for automatic differentiation within Faust, enabling the application of differentiable programming to digital signal processing tasks. In layman's terms, this library will make it easier for audio developers to use Faust to solve complex audio problems that involve machine learning. The major motivator behind this project was Google's Magenta project and its success in applying advanced techniques like differentiable programming to audio tasks, such as timbre transfer. My work in this project has been built on existing work on automatic differentiation in Faust, which is extended to machine learning in this PR.

Automatic Differentiation (AD) is a technique for computing derivatives of functions defined by computer programs. It leverages the chain rule of calculus to efficiently calculate derivatives of complex functions composed of elementary operations. In machine learning, AD is essential for training models that involve optimization problems, including parameter estimation and other several DDSP-based audio tasks in Faust. Gradient-based optimization methods, such as gradient descent, use the derivatives of a loss function with respect to model parameters to update the parameters iteratively. AD provides an efficient way to compute these derivatives, enabling effective training of ML models. With respect to Faust, AD can enable efficient gradient computation for ML tasks, facilitating the development of differentiable signal processing models.

This library, originally focused on the introduction of DDSP to Faust, is now extended to include more primitives and refine the previous work on parameter estimation. Additionally, it introduces Faust Neural Network Blocks, which allow for the seamless integration of neural network components directly within Faust programs.

This library is expected to serve as a comprehensive package that will not only broaden Faust's applicability in audio tasks but also provide a robust framework for future research and development in machine learning-based DSP.

PR Overview

The core of the library consists of a collection of Faust primitives, incorporating the concepts of automatic differentiation. These cover basic mathematical operations as well as commonly used audio processing functions, such as delays @ and trigonometric functions atan2. The project also aims to explore and refine the implementation of loss functions specifically for frequency-domain applications.

The PR also introduces the concept of Faust Neural Network Blocks (FNNBs), which are pre-written and encapsulated Faust code modules that represent specific neural network layer functionalities. To date, this is limited to the implementation of fully connected layers. FNNBs offer several benefits, including reduced development time, improved code maintainability, and enabling audio developers to focus on designing neural network architecture without needing to handle low-level programming for each layer.

By integrating the automatic differentiation (AD) library, FNNBs, and with further work, users may design and train complex neural network models for audio processing tasks directly within Faust. This approach eliminates the need to switch between Faust and separate machine learning frameworks, while still offering the benefits of automatic differentiation and optimization for efficient model training.

Deliverables

The following list of contributions have been made:

• Implementation of all major Faust's primitive operators, in the library diff.lib and an extension of the previous work on this library. Other operators have not been covered due to the non-differentiable nature of the operator itself.
  • Introduction of learning rate schedulers (as seen in gain_scheduler.dsp).
  • Introduction of various optimizers such as stochastic gradient descent, adam, RMSProp.
  • Introduction of various major loss functions such as MSE (L2-norm), MAE(L1-norm), MSLE, Huber loss and a linear-frequency loss function.
• Exploration and implementation of parameter estimation in the frequency domain, as seen in the examples test-freqloss.dsp, freq_estimator.dsp and freq_estimator_rms.dsp, expanding beyond the previous examples focused on the time domain.
• Implementation and introduction of Faust Neural Network Blocks (FNNBs), exploring the setup of weights, biases, and the core component of neural networks, a neuron, in Faust.
  • Implementation of a fully connected layer, with end-to-end differentiability, involving the concept of autodiff wherever possible.
  • Exploration and implementation of a backpropagation environment that calculates the appropriate gradient layer-wise, in a symbolic manner.
• Comprehensive documentation is provided, detailing examples, the underlying mathematical logic, and code usage, to help Faust developers thoroughly understand the inner workings of the library.

Usage

The primitives defined in this library are highly effective in scenarios requiring automatic differentiation and are demonstrated through various examples of parameter estimation. A simple yet effective example of the same can be seen here:

df = library("diff.lib");
    
process = no.noise <: _,_
    : hgroup("Differentiable gain", df.backprop(truth,learnable,
    d.learnMAE(1<<5, d.optimizer.SGD(1e-1) : d.learning_rate_scheduler(10000, -0.1))))
with {
    truth = _,hslider("gain",.5,0,1,.01) : *;
    
    vars = df.vars((gain))
    with {
        gain = -~_ <: attach(hbargraph("gain",0,1));
    };
    
    d = df.env(vars);
    
    learnable = d.input,vars.var(1) : d.diff(*);
};

This example allows us to estimate the parameter gain via the definition of a truth and learnable variable here. Via backpropagation, MAE loss functions, SGD as an optimizer, we can effectively setup a parameter estimation example. I also included examples to explore parameter estimation in the frequency-domain, but noticed that the loss landscape is so varied that our algorithm fails to learn about frequencies as a parameter sufficiently, apart from a few ranges. Frequency-domain loss functions are crucial in audio processing applications because they allow for direct control over the spectral characteristics of the output signal.

The FFT is a fundamental tool for transforming time-domain signals into the frequency domain, which is necessary for calculating frequency-domain loss. FFT calculation, however, is limited in Faust. After extensive experimentation, I noticed the following:

• Faust's FFT calculations can be computationally demanding, particularly when using a high number of spectral bins. For example, with 32 or 64 bins, the WebIDE may experience significant compilation delays.
• While each frequency typically exhibits a unique distribution of energy across spectral bins,
  it's challenging to design a loss function that directly accounts for these individual distributions. Currently, my implementation simply sums the frequency values within each bin, which fails to capture the specific distribution of frequencies. This can lead to a situation where multiple frequencies with different distributions still result in zero loss, as their summed values might be identical. This is a limitation of my current approach.

Apart from that, I also defined a fully connected layer as an FNNB in an end-to-end differentiable manner, allowing audio developers to seamlessly integrate neural network layers into Faust programs, facilitating the design and training of more complex audio processing models. An example of the setup can be seen below (consisting of 2 fully connected layers, with backpropagation algorithms set up -- involving a single input $x$ to the structure).

For more details on the background, usage, and mathematical foundations, please refer to the documentation (README.md) and additional examples.

Future Work and Exploration

For the future, I intend to keep contributing to this DDSP library. In relation to this project, there are a few tasks I plan to continue working on:

• Investigate more automatic differentiation methods to replace the current symbolic approach in the backpropagation algorithm. The symbolic implementation's adherence to the chain rule contradicts the principles of automatic differentiation. (Consult README.md for more information about the same.)
  • A few places could have nomenclature improved and an improvement of general code quality.
• Creation of more types of FNNBs (such as convolutional and recurrent layers) along with improving the recursion environment after gradient calculations.
• Addition of a new feature that explores the creation of an offline inference file that stores the weights and biases of a trained neural network.
• Explore alternative approaches that consider the distribution of frequencies within each spectral bin. One promising avenue is to investigate loss functions based on KL divergence, which measures the difference between two probability distributions. By using KL divergence, we can potentially capture the finer-grained details of the frequency distributions and improve the accuracy of our parameter estimation.

Although previously covered in my GSoC proposal, a major area of work in the future could be to explore how architecture files would function in the context of offline training. This should include investigating the process of defining, managing, and optimizing these files to ensure efficient model training outside of real-time environments, while maintaining compatibility with the rest of the Faust ecosystem. With further development, the goal is for automatic differentiation and gradient descent to become core features of the Faust language. This PR represents a foundational step towards a future independent Faust library, offering a valuable contribution to the Faust community.

Challenges and Next Steps

Each challenge posed by autodiff during the creation of FNNBs -- which led to several weeks of unsuccessful experimentation and routing errors -- required careful troubleshooting, extensive testing, and often creative routing to overcome, all of which contributed to the project's overall complexity and the time required to develop a robust and extremely generalizable solution.

• Overall, I gained a deeper understanding of how Faust's routing and recursion system works, dedicating significant time experimenting with tiny details in Faust. It was both fascinating and challenging to delve into the same.
• My understanding of the mathematical foundations of machine learning has improved significantly over the past three months. This has been crucial in mapping out how machine learning can be effectively integrated into Faust and determining the most efficient ways to implement and leverage these techniques.
• A huge thank you to Tommy and Stéphane for their support throughout the project and, even more importantly, their patience during my learning and coding phase and their collaborative spirit, making a huge difference in the success of our work together.

[PrateekSingh070]

hi sir, i have one doubt , i am going to select this org for gsoc 2026 , is this org good to choose

[hatchjaw]

hi sir, i have one doubt , i am going to select this org for gsoc 2026 , is this org good to choose

Hi. It would be a good idea to try to find out more on the Faust discord server.