Make Weight Initialization configurable

[Samyssmile]

Educational

https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78

Task

e.g. our DenseLayer implement the HE-Weight Initialization. In some cases its better to use XAVIER

  private void initialize() {
    double standartDeviation = Math.sqrt(2.0 / (weights.get().getRows() + weights.get().getCols()));

    for (int i = 0; i < weights.get().getRows(); i++) {
      for (int j = 0; j < weights.get().getCols(); j++) {
        weights.get().set(i, j, random.nextGaussian() * standartDeviation);
      }
    }
    for (int i = 0; i < bias.get().getRows(); i++) {
      for (int j = 0; j < bias.get().getCols(); j++) {
        bias.get().set(i, j, 0);
      }
    }
  }

We want to make user decide what kind of initialization he want to use. XAVIER or HE.

• Create an Enum Initialization with HE function
• Make Network configurable

    new NetworkBuilder()
        .withExecutionMode(singleThread)
        .withEpochs(5)
        .withLearningRates(0.001f, 0.001f)
        **.withWeightInitialization(Initialization.HE)**
        .loadModel("mnist_trained.edux")
        .fit(trainLoader, testLoader);

• If no Initialization paramter is given, HE is used by default

[manumafe98]

I would like to try on this one as well!

[manumafe98]

Hi @Samyssmile where can we discuss about this one? I'm having a bit trouble on knowing how to start.
This Initialization is related to the task?

[Samyssmile]

We have our Discord Chat: https://discord.gg/9aD4pDTQ
@manumafe98

[manumafe98]

Joined!