You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implement the Multiscale Fourier Features (MFF) neural network architecture proposed in this paper "On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks" paper by Sifan Wang, Hanwen Wang, Paris Perdikaris. The paper comes with a github repo which should guide implementation.
The architecture is proposed in equations (3.29 - 3.34) in the paper and figure (8.b) represents it pictorially. This architecture should be implemented as a function that takes in the hyperparameters of the network and outputs a DiffEqFluxFastChain that encapsulates the network. This might mean that it should be implemented in DiffEqFlux instead of here, but the issue makes more sense here. The different hyperparameters that seem relevant to me to include as arguments to the function are:
Input and output dimensions of the network
Number of hidden layers following the Fourier basis
Number of Fourier basis blocks and their scale parameters
Whether an input dimension is a time dimension (and thus that dimension should be split off and use its own Fourier basis and then merged using equation 3.33 as in section 4.3 of the paper)
(i.e.)
function MultiScaleFourierFeatureNetwork(hyperparameters...)
return FastChain(...)
end
The exact form of the hyperparameters seems slightly difficult to define in a way that has good ergonomics though, since you would have to define scale parameters for each of the time-based Fourier bases and for each of the space-based Fourier bases. Perhaps there should just be two different functions, one that generates MFF features for either space or time inputs (or treats space-time together) and the space-time MFF function splits the input along space and time dimensions, calls the interior MFF function, and then splices them together using the elementwise multiplication mechanism in the paper.
This is not too difficult of an implementation project, most of the difficulty is in figuring out a good interface and processing that into the correct DiffEqFlux building blocks. I think that architectures like this are likely to lead to big improvements in the algorithm, since the standard feedforward nets are not well suited to this problem due to spectral bias (as discussed in the paper). There is a lot of interesting and useful theory in the paper (mostly using the NTK theory), but understanding the theory in the first half of the paper isn't necessary to implement the method.
The text was updated successfully, but these errors were encountered:
Randomly generating some untrainable Fourier basis functions is far from being a good model. Of course this paper is "cheating" because it sets hyperparameters based on the spectrum of the real solution. Siren performs much better and is more robust than it: https://www.vincentsitzmann.com/siren/
Implement the Multiscale Fourier Features (MFF) neural network architecture proposed in this paper "On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks" paper by Sifan Wang, Hanwen Wang, Paris Perdikaris. The paper comes with a github repo which should guide implementation.
The architecture is proposed in equations (3.29 - 3.34) in the paper and figure (8.b) represents it pictorially. This architecture should be implemented as a function that takes in the hyperparameters of the network and outputs a
DiffEqFlux
FastChain
that encapsulates the network. This might mean that it should be implemented in DiffEqFlux instead of here, but the issue makes more sense here. The different hyperparameters that seem relevant to me to include as arguments to the function are:(i.e.)
The exact form of the hyperparameters seems slightly difficult to define in a way that has good ergonomics though, since you would have to define scale parameters for each of the time-based Fourier bases and for each of the space-based Fourier bases. Perhaps there should just be two different functions, one that generates MFF features for either space or time inputs (or treats space-time together) and the space-time MFF function splits the input along space and time dimensions, calls the interior MFF function, and then splices them together using the elementwise multiplication mechanism in the paper.
This is not too difficult of an implementation project, most of the difficulty is in figuring out a good interface and processing that into the correct DiffEqFlux building blocks. I think that architectures like this are likely to lead to big improvements in the algorithm, since the standard feedforward nets are not well suited to this problem due to spectral bias (as discussed in the paper). There is a lot of interesting and useful theory in the paper (mostly using the NTK theory), but understanding the theory in the first half of the paper isn't necessary to implement the method.
The text was updated successfully, but these errors were encountered: