Any suggestions for tuning the training parameters?

[NUAAZXY]

Thank you for providing an open-source dictionary learning repository. I have trained on other models similar to GPT-2. Despite numerous parameter adjustments, the training results still exhibit a large number of dead features as shown below.

[jbloomAus]

Probably lowering your learning rate will help, but dead features are a somewhat ubiquitous issue at the moment with various people working on solutions. If you share your full hyperparameter set and dashboards on wandb I may have more suggestions.

…

On Sat, Apr 6, 2024, 6:00 AM Xiangyu Zhang ***@***.***> wrote: Thank you for providing an open-source dictionary learning repository. I have trained on other models similar to GPT-2. Despite numerous parameter adjustments, the training results still exhibit a large number of dead features as shown below. 11.png (view on web) <https://github.com/jbloomAus/mats_sae_training/assets/104181523/fd674f7a-88db-447e-9dbc-7aed2b63bb80> — Reply to this email directly, view it on GitHub <#67>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQPMYZ3KOBIPCAN4VQG6UCDY356OBAVCNFSM6AAAAABF2ERFQKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDSMBXHAYTQOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[NUAAZXY]

Lowering the learning rate helps, thanks!