New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arbiter: Missing configuration options #6062

Closed
AlexDBlack opened this Issue Aug 3, 2018 · 6 comments

Comments

Projects
None yet
2 participants
@AlexDBlack
Copy link
Member

AlexDBlack commented Aug 3, 2018

Thanks to @eraly

  • BatchNormalizationSpace does not support constrainBeta and constrainGamma options
  • ConvolutionLayerSpace, SubsamplingLayerSpace and Deconvolution2DLayerSpace do not support kernel dilation and use the default value of {1,1}
  • Currently EmbeddingLayerSpace does not support hasBias and uses the default value of False
  • Currently OCNNLayerSpace do not support the configureR (uses default value of true), initialRValue (uses default value of 0.1) and windowSize. What is referred to hiddenLayerSize in the OCNNOutputLayer.Builder is referred to as numHidden in OCNNLayerSpace.Builder.
  • Currently SeparableConvolution2DLayerSpace does not support constrainPointWise
  • while it is possible to set up weightConstraints, l1Bias and l2Bias as part of the NeuralNetConfiguration these have to be setup on a per layer/layerSpace basis in MultiLayerSpace
  • MultiLayerSpace.addLayer: "Currently Arbiter does not support the ability to create independent layers. Therefore the last boolean argument should always be set to true." - let's either fix or remove

new DiscreteParameterSpace<>(new Boolean[]{true, false}) Should be a built-in thing - like new BooleanDiscreteParameterSpace() or something (edit: better yet - maybe just BooleanSpace)

@eraly eraly self-assigned this Aug 6, 2018

@eraly

This comment has been minimized.

Copy link
Member

eraly commented Aug 6, 2018

So looking at the dl4j code it looks like the constrainBeta and constrainGamma options don't get used in BatchNormalizationSpace. It looks like it is getting used in model import for keras layers. I put in the skeleton for it anyway and we can fix it in dl4j later?

It gets really messy on instantiating discrete layer constraints parameter space is...Any thoughts to clean that up?

ArrayList<List<LayerConstraint>> constrainListOptions = new ArrayList<List<LayerConstraint>>();
constrainListOptions.add(Arrays.asList(new LayerConstraint[]{new MaxNormConstraint(0.5, 1)}));
constrainListOptions.add(Arrays.asList(new LayerConstraint[]{new MinMaxNormConstraint(0.3, 0.4, 1.0, 1)}));
constrainListOptions.add(Arrays.asList(new LayerConstraint[]{new NonNegativeConstraint()}));
constrainListOptions.add(Arrays.asList(new LayerConstraint[]{new UnitNormConstraint(1)}));
DiscreteParameterSpace<List<LayerConstraint>> constrainParamSpace = new DiscreteParameterSpace<>(constrainListOptions);

(Edited to say constrainBeta and constrainGamma options don't! get used in BatchNormalizationSpace)

@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented Aug 6, 2018

Yeah, constrainBeta/Gamma aren't a big deal, but we might as wel fix it in DL4J later.

As for the constraints parameter space code there - I'd just do this instead, which is probably good enough for now:
https://gist.github.com/AlexDBlack/bdcfce752a812007c3d9e40e0af5ebe0

@eraly

This comment has been minimized.

Copy link
Member

eraly commented Aug 6, 2018

Ah cool. I'll switch to that in the test. Thanks.
I'll open an issue for constrainBeta/Gamma in a bit.

@eraly

This comment has been minimized.

Copy link
Member

eraly commented Aug 6, 2018

So now I know why we can't have separate constructors for parameter space learning rates and learning rate schedules. The constructor that takes learning rate alone and the one that would take learning rate schedule alone would have the same erasure. I guess we are stuck with SgdSpace(null,lrSchedule)

@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented Aug 6, 2018

@AlexDBlack AlexDBlack closed this Aug 6, 2018

@lock

This comment has been minimized.

Copy link

lock bot commented Sep 21, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.