Skip to content

feat: Implement dropout regularization and enhance testing#4

Merged
kholdrex merged 1 commit into
mainfrom
feature/comprehensive-dropout-testing
Jun 10, 2025
Merged

feat: Implement dropout regularization and enhance testing#4
kholdrex merged 1 commit into
mainfrom
feature/comprehensive-dropout-testing

Conversation

@kholdrex

Copy link
Copy Markdown
Member

This PR adds complete dropout regularization support to the Rust LSTM library, implementing multiple dropout types specifically designed for recurrent neural networks along with comprehensive configuration options and training integration.

Features Added

🎯 Core Dropout Implementation (src/layers/dropout.rs)

  • Standard Dropout: Element-wise random zeroing with configurable rates
  • Variational Dropout: Consistent mask across time steps for RNN stability
  • Zoneout: LSTM-specific regularization keeping previous hidden/cell states
  • Proper forward/backward pass with mask saving for gradient computation
  • Training/evaluation mode switching with automatic mask management

🧠 LSTM Integration (src/layers/lstm_cell.rs)

  • Input Dropout: Applied to input sequences before gate computation
  • Recurrent Dropout: Applied to hidden states in recurrent connections
  • Output Dropout: Applied to final hidden state outputs
  • Zoneout Integration: Cell and hidden state regularization
  • Builder pattern for easy configuration: .with_input_dropout(0.2, true)
  • Mathematically correct backward pass with saved dropout masks

@kholdrex kholdrex self-assigned this Jun 10, 2025
@kholdrex kholdrex added the enhancement New feature or request label Jun 10, 2025
@kholdrex
kholdrex merged commit b71112c into main Jun 10, 2025
1 check passed
@kholdrex
kholdrex deleted the feature/comprehensive-dropout-testing branch June 10, 2025 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant