Bias in the output layer, True or False or does it matter?

[tmrogo]

In Chapter 6, and also in Appendix E, we replace the GPTModel output layer
'out_head' in order to prepare the model for fine-tuning, changing the
out_features from 50257 to 2 (num_classes), as shown below:
num_classes = 2 model.out_head = torch.nn.Linear(in_features=768, out_features=num_classes)
In running this code, I did a print(model) before, and then after the
change and I noticed that the 'bias' changed from 'False' to 'True'.

before the change:
(out_head): Linear(in_features=768, out_features=50257, bias=False)

after the change:
(out_head): Linear(in_features=768, out_features=2, bias=True)

My question is: does 'bias' matter in the out_head? And if so, should
the code in Chapter 6 and Appendix E have been:
num_classes = 2 model.out_head = torch.nn.Linear(in_features=768, out_features=num_classes, bias=False)
Thanks for any insights

[gistrec]

The short answer: it should make almost no practical difference for this classification fine-tuning setup

torch.nn.Linear defaults to bias=True, so your replacement head adds only num_classes extra trainable parameters — 2 in this case. Compared with the rest of the GPT model, that overhead is negligible.

The original GPT-2-style language modeling head is bias=False because it follows the LM-head architecture where the output projection is commonly tied to the token embedding weights. Once you replace it with a small classification head, that weight-tying constraint no longer applies

So both are reasonable:

model.out_head = torch.nn.Linear(768, num_classes)              # bias=True, fine
model.out_head = torch.nn.Linear(768, num_classes, bias=False)  # also fine

If you want to mirror the book/code style exactly, use bias=False. But for a newly initialized 2-class classification head, leaving the PyTorch default bias=True is perfectly fine, and I would not expect it to meaningfully affect the result

[aasimansari1]

Good question on output layer bias! In practice for LLMs, the output projection usually does use bias (it helps the model learn token frequency offsets). But some architectures like PaLM omit it for parameter efficiency.

For more interview-level depth on transformer architecture decisions: ML Interview Prep has a detailed section on LLM internals. Hope it helps anyone studying this book!