-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
208 lines (190 loc) · 10.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, shrink-to-fit=no">
<title>MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.5.0/css/bootstrap.min.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Source+Sans+Pro">
<link rel="stylesheet" href="assets/css/Highlight-Clean.css">
<link rel="stylesheet" href="assets/css/styles.css">
<link rel="stylesheet" href="assets/css/Team-Clean.css">
<!-- Favicon link -->
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>🚀</text></svg>">
<body>
<div class="highlight-clean" style="padding-bottom: 10px; padding-top: 10px;">
<h1 class="text-center">🚀 MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining</h1>
<div><p></p></div>
<div class="container" style="max-width: 768px;background:white;">
<div class="row">
<div class="col-md-12", style="padding: 10px;">
<h4 class="text-center" style="margin: 0px;">
<a class="text-center", href = https://jacobfulano.github.io>Jacob Portes**</a>,
<a class="text-center", href = https://twitter.com/alexrtrott>Alex Trott*</a>,
<a class="text-center">Sam Havens</a>,
<a class="text-center">Daniel King</a>,
<a class="text-center", href = https://twitter.com/abhi_venigalla>Abhinav Venigalla</a>,
<a class="text-center">Moin Nadeem</a>,
<a class="text-center", href = https://nikhilsardana.github.io>Nikhil Sardana</a>,
<a class="text-center">Daya Khudia</a>,
<a class="text-center", href = http://www.jfrankle.com>Jonathan Frankle</a>
</h4>
<h5 class="text-center">MosaicML x Databricks</h5>
<h6 class="text-center">**jacob.portes@databricks.com, *equal contribution</h6>
</div>
</div>
</div>
<div class="buttons" style="margin-bottom: 8px;">
<a class="btn btn-primary" role="button" href="https://www.mosaicml.com/blog/mosaicbert">Blogpost</a>
<a class="btn btn-primary" role="button" href="https://openreview.net/forum?id=5zipcfLC2Z">Paper</a>
<a class="btn btn-primary" role="button" href="https://github.com/mosaicml/examples/tree/main/examples/benchmarks/bert">GitHub</a>
<a class="btn btn-primary" role="button" href="https://huggingface.co/collections/mosaicml/mosaicbert-6564f20cfd712e8aa84c29e4">Hugging Face</a>
<a class="btn btn-primary" role="button" href="https://colab.research.google.com/drive/1r0A3QEbu4Nzs2Jl6LaiNoW5EumIVqrGc?usp=sharing">Colab</a>
</div>
<div><p></p></div>
<!-- Figure -->
<div><p></p></div>
<div class="container" style="max-width: 768px;" >
<div class="row">
<div class="col-md-12">
<h2>TLDR; How to Speed up Transformer Pretraining</h2>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h5>MosaicBERT is a custom BERT architecture optimized for fast pretraining. This study motivated many of the architecture choices around MosaicML's <a href="https://www.mosaicml.com/blog/mpt-7b">MPT-7B</a> and <a href="https://www.mosaicml.com/blog/mpt-30b">MPT-30B</a> models. Below are the main architectural modifications used by MosaicBERT for rapid pretraining 👇</h5>
</div>
</div>
<div class="row">
<div class="col-md-12">
<ul>
<li><a href="https://github.com/Dao-AILab/flash-attention">FlashAttention:</a> This is an efficient implementation of attention that has rapidly become the standard</li>
<li><a href=https://github.com/ofirpress/attention_with_linear_biases>🥸 ALiBi (Attention with Linear Biases):</a> ALiBi allows you to train short on shorter sequence lengths and then extrapolate to longer sequence lengths</li>
<li><a href="https://docs.mosaicml.com/projects/composer/en/latest/method_cards/gated_linear_units.html">⛩ Gated Linear Units</a></li>
<li>👉 Unpad your inputs after the attention block! This avoids wasting compute on PAD tokens</li>
<li><a href="https://docs.mosaicml.com/projects/composer/en/stable/method_cards/low_precision_layernorm.html">Low Precision LayerNorm:</a> set LayerNorm operation to bfloat16 precision instead of the typical float32 precision</li>
</ul>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h5> And here are a few more efficiency tips used by MosaicBERT:</h5>
</div>
</div>
<div class="row">
<div class="col-md-12">
<ul>
<li>Change the Masked Language Modeling ratio to 30% (instead of the default 15%)</li>
<li>Remove dropout from the attention module (dropout often slows things down)</li>
<li>Use bfloat16!</li>
<li>Make your vocab size as a multiple of 64 <a href="https://twitter.com/karpathy/status/1621578354024677377">(Andrej Karpathy says so!)</a></li>
</ul>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h5>Every ML practitioner who is pretraining or finetuning transformers should consider using these modifications in their own stack. Note that all modifications (except for MLM) can be applied to decoder architectures such as GPT and MPT.</h5>
</div>
</div>
</div>
</div>
<script>
</script>
<style>
body {
font-family: 'Source Sans Pro', sans-serif;
font-size: 16px;
margin: 0;
padding: 0;
background: white;
}
.container {
padding: 20px;
margin: 0 auto;
box-shadow: 0 2px 5px rgba(0, 0, 0, 0.1);
border-radius: 8px;
background: white;
margin-bottom: 30px;
}
.news-container {
max-height: 200px;
overflow-y: auto;
}
h1, h2, h3, h4, h5, h6 {
color: #333;
}
/* Button Background */
.btn-primary {
background:#13294E;
border-color: white;
text-align: center;
}
.btn-primary:hover {
background:#13294E;
border-color: white;
text-align: center;
}
/* Footer Gradient Background */
.container:last-child {
background: white;
color: white;
}
img {
max-width: 100%;
height: auto;
border-radius: 8px;
}
em {
font-style: italic;
color: black;
}
.api-appstore {
margin-bottom: 20px;
}
</style>
<!-- Abstract -->
<div class="container" style="max-width: 768px;">
<div class="row">
<div class="col-md-12">
<h2>Abstract</h2>
<p style="text-align: justify;">
Although BERT-style encoder models are heavily used in NLP research, many researchers do not pretrain their own BERTs from scratch due to the high cost of training. In the past half-decade since BERT first rose to prominence, many advances have been made with other transformer architectures and training configurations that have yet to be systematically incorporated into BERT.
Here, we introduce MosaicBERT, a BERT-style encoder architecture and training recipe that is empirically optimized for fast pretraining. This efficient architecture incorporates FlashAttention, Attention with Linear Biases (ALiBi), Gated Linear Units (GLU), a module to dynamically remove padded tokens, and low precision LayerNorm into the classic transformer encoder block. The training recipe includes a 30% masking ratio for the Masked Language Modeling (MLM) objective, bfloat16 precision, and vocabulary size optimized for GPU throughput, in addition to best-practices from RoBERTa and other encoder models.
When pretrained from scratch on the C4 dataset, this base model achieves a downstream average GLUE score of 79.6 in 1.13 hours on 8 A100 80 GB GPUs at a cost of roughly $20. We plot extensive accuracy vs. pretraining speed Pareto curves and show that MosaicBERT base and large are consistently Pareto optimal when compared to a competitive BERT base and large. This empirical speed up in pretraining enables researchers and engineers to pretrain custom BERT-style models at low cost instead of finetune on existing generic models. We open source our model weights and code.
<br>
</p>
</div>
</div>
</div>
<div class="container" style="max-width: 768px;background: #f0f7ff">
<div class="row">
<div class="col-md-12">
<h2>Main Result</h2>
</div>
<div class="col-md-12 text-center"><img src="assets/Figure-1-high-res-7-7-543pm.png"
style="width: 100%;margin-bottom: 8px;" alt="main result (Fig 1)">
</div>
<em> (A) Schematic of MosaicBERT architecture (B) Pareto curves of average GLUE (dev)
scores for MosaicBERT-Base and the standard BERT-Base. Error bars indicate 95% confidence
interval over n=5 pretraining seeds. All training was on 8xA100-80GB GPUs. FlashAttention
schematic adapted from Dao et al. [2022], and unpadding schematic adapted from Zeng et al. [2022]</em>
</div>
</div>
<!-- Citation -->
<div class="container" style="max-width: 768px">
<div class="row">
<div class="col-md-12">
<h2>Citation</h2>
<code>
@article{portes2023MosaicBERT,<br>
title={MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining}, <br>
author={Jacob Portes, Alexander R Trott, Sam Havens, Daniel King, Abhinav Venigalla, <br> Moin Nadeem, Nikhil Sardana, Daya Khudia, Jonathan Frankle},<br>
journal={NeuRIPS https://openreview.net/pdf?id=5zipcfLC2Z},<br>
year={2023},<br>
}
</code>
</div>
</div>
</div>
</body>
</html>