Open
Conversation
added 3 commits
May 1, 2025 15:46
This commit adds support for 5-channel multispectral data in the VAE architecture while maintaining compatibility with Stable Diffusion 3's latent space requirements. Key changes: - Add AutoencoderKLMultispectral5Ch implementation with 5 input/output channels - Implement 8x downsampling to match SD3's latent space requirements - Add comprehensive test suite for multispectral VAE functionality - Add training script for 5-channel multispectral data - Update documentation with detailed implementation notes Technical details: - Uses 4 downsampling blocks for 8x downsampling - Maintains 4-channel latent space for SD3 compatibility - Implements group normalization (32 groups) for stable training - Preserves spectral information through careful normalization - Handles 16-bit multispectral data with proper scaling Files changed: - src/diffusers/models/autoencoders/autoencoder_kl_multispectral_5ch.py - tests/models/autoencoders/test_models_autoencoder_kl_multispectral_5ch.py - examples/multispectral/train_multispectral_vae_5ch.py
Contributor
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
Collaborator
|
can you explain what's the use case for this? |
Contributor
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
new modules: dataloader and dataloader test (local, not optimized for GPU)
multispectral data loading and preprocessing
TODO: explicit caching and order validation
adapted main training script for MS data loading