Skip to content
This repository was archived by the owner on Jul 1, 2023. It is now read-only.

Fix Glorot uniform initialization for convolutional layers #22

Merged
merged 3 commits into from
Feb 22, 2019
Merged

Fix Glorot uniform initialization for convolutional layers #22

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e3e03c7
Fix Glorot uniform initialization for convolutional layers
jekbradbury Feb 22, 2019
1d58ae3
Apply suggestions from code review
jekbradbury Feb 22, 2019
a8491c3
Update Sources/DeepLearning/Initializers.swift
jekbradbury Feb 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions Sources/DeepLearning/Initializers.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,22 +129,27 @@ public extension Tensor where Scalar: TensorFlowFloatingPoint,
Scalar.RawSignificand: FixedWidthInteger {
/// Performs Glorot uniform initialization for the specified shape, creating a tensor by
/// randomly sampling scalar values from a uniform distribution between `-limit` and `limit`,
/// where limit is `sqrt(6 / (fanIn + fanOut))`.
/// where limit is `sqrt(6 / (fanIn + fanOut))` and `fanIn`/`fanOut` represent the number of
/// input and output features multiplied by the receptive field if present.
///
/// - Parameters:
/// - shape: The dimensions of the tensor.
/// - generator: Random number generator to use.
///
init<G: RandomNumberGenerator>(glorotUniform shape: TensorShape, generator: inout G) {
let fanIn = shape[shape.count - 2]
let fanOut = shape[shape.count - 1]
let spatialDimCount = shape.count - 2
let receptiveField = shape[0..<spatialDimCount].contiguousSize
let fanIn = shape[shape.count - 2] * receptiveField
let fanOut = shape[shape.count - 1] * receptiveField
let minusOneToOne = 2 * Tensor(randomUniform: shape, generator: &generator) - 1
self = sqrt(Scalar(6) / Scalar(fanIn + fanOut)) * minusOneToOne
}

/// Creates a tensor by performing Glorot uniform initialization for the specified shape,
/// randomly sampling scalar values from a uniform distribution between `-limit` and `limit`,
/// where limit is `sqrt(6 / (fanIn + fanOut))`, using the default random number generator.
/// generated by the default random number generator, where limit is
/// `sqrt(6 / (fanIn + fanOut))` and `fanIn`/`fanOut` represent the number of input and output
/// features multiplied by the receptive field if present.
///
/// - Parameters:
/// - shape: The dimensions of the tensor.
Expand Down