Fourier Features

[nikuznetsov]

Dear Dr. Lu，

Thanks for providing such a useful library. I found it really helpful. Currently, I have been testing one of my hypotheses (based on this work) and faced a small problem.

What I want to implement

I want to make an encoding function before network input to encode features. For that, I use such things:

I have an input vector. In my case, it is two-dimensional (x, y coordinates).

And I'd like to provide transform such that:

Using random matrix B. So, the final formula looks like that:

Speaking schematically: 2d vector with coordinates --> gamma function --> gamma fuction transform 2d vector into 2m-dimensional vector --> network input.

What I already implemented

I use a simple problem with the Poisson equation:

PDE definition:

def pde(x, y):
    dy_x = tf.gradients(y, x)[0]
    dy_x, dy_y = dy_x[:, 0:1], dy_x[:, 1:]
    dy_xx = tf.gradients(dy_x, x)[0][:, 0:1]
    dy_yy = tf.gradients(dy_y, x)[0][:, 1:]

    return -dy_xx-dy_yy+12*(x[:, 0:1]*x[:, 0:1] + 4*x[:, 1:]*x[:, 1:])

Other important declarations:

def boundary(x, on_boundary):
    return on_boundary

def func(x):
    return 1 + x[:, 0:1]*x[:, 0:1]*x[:, 0:1]*x[:, 0:1] + 4*x[:, 1:]*x[:, 1:]*x[:, 1:]*x[:, 1:]

geom = dde.geometry.Rectangle([0,0],[1,1])
bc = dde.DirichletBC(geom, func, boundary)
data = dde.data.PDE(geom, pde, bc, 1022, 124, solution=func, num_test=2116)

The problem itself

Now, according to my logic, I have to declare network architecture. And to validate my hypothesis the input layer has to have a width of 2m (for example, consider m = 10). Then for network I have such code:

net = dde.maps.FNN([20] + [50] * 4 + [1], "tanh", "Glorot uniform")

After that I read here that Fourier features can be implemented using apply_feature_transform() function. Therefore, I implemented such code:

def feature_transform(x):
    B = tf.random.normal([10,2], 0, 1, tf.float32, seed=42)
    return tf.concat([
                      tf.cos( 2 * math.pi * tf.linalg.matvec(B, x[:, :2]) ),
                      tf.sin( 2 * math.pi * tf.linalg.matvec(B, x[:, :2]) )
                      ], axis=1)

net.apply_feature_transform(feature_transform)
model = dde.Model(data, net)

After that model.compile("adam", lr=0.001, metrics=["l2 relative error"]) works fine and outputs:

Compiling model...
Building feed-forward neural network...
'build' took 0.120090 s

/usr/local/lib/python3.7/dist-packages/deepxde/nn/tensorflow_compat_v1/fnn.py:110: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.
  kernel_constraint=self.kernel_constraint,
/usr/local/lib/python3.7/dist-packages/keras/legacy_tf_layers/core.py:255: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
  return layer.apply(inputs)
'compile' took 1.912832 s

But after that, I run this part of the code:

checkpointer = dde.callbacks.ModelCheckpoint("model/model.ckpt", verbose=1, save_better_only=True)
losshistory, train_state = model.train(epochs=5000,callbacks=[checkpointer])

And face an error that ValueError: Cannot feed value of shape (1270, 2) for Tensor Placeholder_35:0, which has shape (None, 20).

As I can interpret it, the shape of the input vector is incorrect. As expected it is 2d vector, but the network waits for a 20d vector. I expected that this dimensional crisis can be solved by apply_feature_transform() function, but it seems that something is wrong.

Question

Is it even possible by using today's tools of deepxde to implement such an idea? And if yes, may you advise some way of doing it? I have been trying to solve it myself for a couple of days but seem to be completely stuck.

Thanks in advance for your help!

[lululxvi]

net = dde.maps.FNN([2] + [50] * 4 + [1], "tanh", "Glorot uniform")

[nikuznetsov]

net = dde.maps.FNN([2] + [50] * 4 + [1], "tanh", "Glorot uniform")

I tried this approach and it worked, thank you. But I wonder if using such expression is the same as the approach I'd like to implement. The architecture of the network still has two-dimensional input, but my idea was to make the input 2m-dimensional. Or data is passed through the input layer and then the 2d vector is transformed into a 2m-dimensional vector?

[lululxvi]

Yes, it works as you expected.

[nikuznetsov]

Thanks for the support!