Factorization Machines Layer

[edmondja]

Hello,

I don't know if it is the right way to ask for it and if it already exists but last year I had the idea to put Factorization Machines layers into neural networks (aka Neuralization Machines). I did it very easily on Keras.

So I thought it can be cool to add this functionality into next Keras versions.
Unfortunately I don't have time to learn how to do a pull request and how to respect Keras' code syntax (and I am sorry for that), but please find attached my working 2-way Factorization Machines layer.
custom.py.zip

I think it can be also very interesting

• to make it handle sparse input matrices
• to make it handle more than 2-way interactions using Geffy (https://github.com/geffy/tffm) or Mathieu Blondel and co.'s work ("Higher-Order Factorization Machines")
• and to make it work with RNNs (like making it time distributed or putting it directly inside the LSTM) to handle better temporal effects (like to prevent from recommending always the same kind of music...)

I just hope that my post can help to build a kind of ultimate recommender system.

[Shujian2015]

Sounds like a solution: https://github.com/mzaradzki/factorization-machine-for-prediction/blob/master/keras.ipynb

[edmondja]

Yes yes, I discovered embedding layers afterward, they are very powerful and provide a kind of generalization of FM.

But you cannot do "0, 0, 0, 0, 0, 0, 0, 0, 0.34, 0, 0, 0, 0, ... , 0, 0, 0.798, 0, 0, 0, 0, -1.72, 0, 0" as input
with word embeddings. That's why I say "a kind of generalization".

PS: for those who might be interested https://github.com/lystdo/Codes-for-WSDM-CUP-Music-Rec-1st-place-solution. I tried myself and the MLP version works indeed better than the RNN one (or even attention).
It seemed to overfit a lot due to the many updates made on the embedding layer #8259 .

[abhidipbhattacharyya]

Hi,
I am trying to use the custom.py and the FM as a layer in my model. Here is my code-

self.ip1 = Input(shape=(input_shape,))
self.ip2 = Input(shape=(input_shape,))
self.sim = Input(shape=(1,))

    self.sh_dense = Dense(dimention, activation='softmax')#(Dropout(0.5)(Dense(1500, activation='relu')))
    self.sh_dense1_op = self.sh_dense(self.ip1)
    self.sh_dense2_op = self.sh_dense(self.ip2)

    self.merged_layer = merge([self.sh_dense1_op, self.sh_dense2_op], mode = 'concat')
    self.merged_layer = merge([self.merged_layer, self.sim], mode = 'concat')
    #self.merged_layer = Reshape((101,))(self.merged_layer )

    self.prediction = Dense(1, activation='sigmoid')(FactorizationMachinesLayer(2, activation = 'relu')(self.merged_layer))

I am getting this error-
ValueError: Input 0 is incompatible with layer factorization_machines_layer_1: expected ndim=2+, found ndim=2

Could you help me out. My input dimension is 2- the first one is batch size, the next one is the vector size. I tried by reshaping the dimension to 3. But the error is still there with "found ndim=3". I am sorry if it is not a correct place to post this.

[edmondja]

Hi, I had sometimes that problem but Im sorry I don’t know why it appears.
What if you put FactorizationMachinesLayer(3.... instead of 2 in the last line ?

[abhidipbhattacharyya]

Hello,
Thank you for your quick reply. It is not working with FactorizationMachinesLayer(3.... ). The error is still the same. I am trying to understand topology.py of Keras as the error is thrown from line number 474 and then 403. I might be doing something wrong here. Here is my model
model.py.zip

[edmondja]

I think its more likely that I made a mistake in the coding rather than in the maths, due to my skills..