Dropout layer is an identity placeholder in TRAIN phase — no stochastic masking

[michalharakal]

Summary

sk.ainet.lang.nn.layers.Dropout currently returns its input unchanged in both phases. The class doc says so itself:

Current implementation keeps identity semantics on both TRAIN and EVAL phases, but exposes a context-aware forward that can later be extended to apply stochastic masking when ctx.inTraining is true.

Any model that includes Dropout for regularization silently trains without it — there is no error and no warning, so users following standard architectures (transformers, CNNs) get different training dynamics than expected.

(Related: old closed issue #5 introduced the layer; this is about implementing the actual masking.)

Expected

Inverted dropout under a training-phase context: zero each element with probability p, scale survivors by 1 / (1 - p); identity under inference. Gradients flow through the element-wise multiply to surviving elements only.

Reference implementation

A minimal phase-aware version built only on public APIs (constant mask tensor + element-wise multiply, so autograd handles the backward for free):

override fun onForward(input: Tensor<FP32, Float>, ctx: ExecutionContext): Tensor<FP32, Float> {
    if (!ctx.inTraining || p == 0f) return input
    val scale = 1f / (1f - p)
    val maskData = FloatArray(input.volume) { if (random.nextFloat() < p) 0f else scale }
    val mask = ctx.fromFloatArray<FP32, Float>(input.shape, FP32::class, maskData)
    return input * mask
}

A generic in-core version would need dtype-generic RNG/mask creation, plus a seeding story for reproducibility. Happy to send a PR along these lines.

Found while porting an educational GPT training pipeline (attention-weight dropout) to SKaiNET 0.36.0.