-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bfloat16
cannot utilize some codes
#114
Comments
@AmitMY hey Amit! i put in a quick fix in curious how well FSQ is performing for you otherwise. are you training an autoencoder? |
Hi! Was waiting for some compute to try this, but actually it fails: (network is now BFloat16, input is cast as float)
FSQ is performing amazingly well for me. Basically 100% codebook util, and the autoencoder can predict the input very well. I did have to normalize my data, but once that was done it was smooth sailing. |
@AmitMY besides code utilization, have you tried running it against regular VQ as an ablation / to compare? |
I have only tried regular VQ in the beginning, saw that FSQ was better/more stable for my problem, and then scaled up data/model size - so no, for my current problem I did not fully compare FSQ and VQ |
@AmitMY ah got it, no biggie. just curious |
@AmitMY finally had the chance to train FSQ myself yesterday evening and wow, it works great! so much more stable than VQ |
When using
FSQ
with[8, 5, 5, 5]
levels, and inpytorch-lightning
specifyingbfloat16
training, the codebook utilization scratches 50% from below, while when training withfloat32
it scratches 100%.I don't know if there is any issue with the implementation or just a limitation with the FSQ, in any case I would guess that this library should force float32 for the quantization step.
Example:
The text was updated successfully, but these errors were encountered: