Transfer Learning MFNN

[materialsguy]

Hi Lu,

first of all, thanks for sharing your library.
I'm trying to use a MFNN with two different but similar multi fidelity datasets for a transfer learning problem. I’m quite new two tensor flow, do you have a code snippet available on how to save and reload the MFNN so that I can reuse the pre-trained MFNN to train it with the new datasets?

Thank you very much I appreciate your work.

Best,
Claus

[materialsguy]

Hi,

I just wanted to tell you that the transfer learning worked out, altough it only works with a little workaround.

I'm using Jupyter Notebooks and I tried the following thing:

Step 1:

model=dde.Model(data, net)
model.compile("adam", lr=lr, loss="MAPE", metrics=["MAPE", "APE SD"])
path="model\\"
checker = dde.callbacks.ModelCheckpoint(f"{path}model_{y_name}.ckpt", verbose=1, save_better_only=True, period=1000)
losshistory, train_state = model.train(epochs=epochs, callbacks=[checker])

training worked without a problem. If I directly reload the model at any checkpoint directly by:
Step 2:

losshistory, train_state =model.train(epochs=epochs_2, model_restore_path=f"{path}model_{y_name}.ckpt-{epochs}")

it works completely fine continuing the training.

But if I want to add new data or reload the model later by using:

Step 3:

path="model\\"
model=create_mfnn(6, data)
model.compile("adam", lr=0.0001, loss="MAPE", metrics=["MAPE", "APE SD"])

losshistory, train_state = model.train(epochs=1000, model_restore_path=f"{path}model_{y_name}.ckpt-1000")

if didn't work out and I got the Error message:

NotFoundError: 2 root error(s) found.
  (0) Not found: Key Variable/Adam not found in checkpoint
	 [[node save_2/RestoreV2 (defined at C:\Users\ctrost\AppData\Local\Continuum\anaconda3\envs\tenosor_flow\lib\site-packages\deepxde\model.py:197) ]]
	 [[save_2/RestoreV2/_225]]
  (1) Not found: Key Variable/Adam not found in checkpoint
	 [[node save_2/RestoreV2 (defined at C:\Users\ctrost\AppData\Local\Continuum\anaconda3\envs\tenosor_flow\lib\site-packages\deepxde\model.py:197) ]]
0 successful operations.
0 derived errors ignored.

If I then restart the kernel and run Step 3: again it loads the former step and retraines the model with the new data.

Best,
Claus

[lululxvi]

Good to know you solve it. For transfer learning, you can check the code at https://github.com/lululxvi/deep-learning-for-indentation and lululxvi/deep-learning-for-indentation#2