Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DL4J: SimpleRnnParamInitializer does not respect biasInit configuration #6431

Closed
Charele opened this issue Sep 12, 2018 · 2 comments · Fixed by #6470

Comments

@Charele
Copy link

commented Sep 12, 2018

Let's see the init() method of SimpleRnnParamInitializer:

... ...
INDArray rw = WeightInitUtil.initWeights(nOut, nOut, new long[]{nOut, nOut}, rwInit, rwDist, 'f', m.get(RECURRENT_WEIGHT_KEY));
m.put(RECURRENT_WEIGHT_KEY, rw);
} else {
m = getSubsets(paramsView, nIn, nOut, true);
}

    conf.addVariable(WEIGHT_KEY);
    conf.addVariable(RECURRENT_WEIGHT_KEY);
    conf.addVariable(BIAS_KEY);

    return m;
}

It initialize the "W" and "RW", forget the init of parameter "b"(will keep default 0)
It's a lost? or bias 0 value is helpful for RNN?

@AlexDBlack AlexDBlack changed the title Init in SimpleRnnParamInitializer DL4J: SimpleRnnParamInitializer does not respect biasInit configuration Sep 13, 2018

@AlexDBlack

This comment has been minimized.

Copy link
Contributor

commented Sep 13, 2018

Thanks for reporting. Looks like yes, SimpleRNN does dave a biasInit config option but it doesn't use that in the weight initializer.

@AlexDBlack AlexDBlack self-assigned this Sep 20, 2018

AlexDBlack added a commit that referenced this issue Sep 23, 2018
DL4J/ND4J Misc fixes (#6470)
* #6431 SimpleRNN respect biasInit config value

* #6457 FileSplit android compatibility

* #6451 Improve preconditions class

* Fix pad op mapping

* #6471 Check for OutputLayer as first step in MultiLayerNetwork.computeGradientAndScore()

* Fix Reverse op mapping; map HistogramFixedWidth

* Fix HistogramfixedWidth mapping

* Fix DeConv2DTF op import mapping

* Fix unsorted op mapping

* HistogramFixedWidth fix; map DynamicPartitionBp

* Conv3d and OneHot test fixes

* More test fixes/re-enabling

* Restore ReverseV2 op

* Test tweaks
@lock

This comment has been minimized.

Copy link

commented Oct 24, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Oct 24, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.