Skip to content

Experiment to consider default zloss#941

Merged
dlwh merged 4 commits intomainfrom
experiments/exp934_zloss
Apr 5, 2025
Merged

Experiment to consider default zloss#941
dlwh merged 4 commits intomainfrom
experiments/exp934_zloss

Conversation

@dlwh
Copy link
Copy Markdown
Member

@dlwh dlwh commented Apr 4, 2025

Description

Closes #935 #934

Based on this I think we can make zloss the default. still want to run fp32 attn and make sure mfus aren't impacted

@dlwh dlwh merged commit ef74efc into main Apr 5, 2025
3 checks passed
@dlwh dlwh deleted the experiments/exp934_zloss branch May 18, 2025 17:51
ryan-williams pushed a commit that referenced this pull request Oct 6, 2025
doing a huge refactor cleanup to text.py so trying to stage as much out
as i can.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Experiment: Default z-loss?

1 participant