Alternative gpt2 weight download

[rasbt]

In case the official weights from OpenAI ever become unavailable, I wanted to share a few tips for working around the issue.

 

1) Connection checks

It could be a temporary internet connection issue or a problem with your IP address. I recommend double-checking using a different machine (e.g., Google Colab).

For your convenience, use the following code to download the download script and the GPT-2 weights:

# Download the download file

import urllib.request
url = (
    "https://raw.githubusercontent.com/rasbt/"
    "LLMs-from-scratch/main/ch05/"
    "01_main-chapter-code/gpt_download.py"
)
filename = url.split("/")[-1]
urllib.request.urlretrieve(url, filename)

from gpt_download import download_and_load_gpt2

settings, params = download_and_load_gpt2(model_size="124M", models_dir="gpt2")
# settings, params = download_and_load_gpt2(model_size="355M", models_dir="gpt2")
# settings, params = download_and_load_gpt2(model_size="774M", models_dir="gpt2")
# settings, params = download_and_load_gpt2(model_size="1558M", models_dir="gpt2")

 

2) Alternative download

In case the above doesn't work, you can download the weights from this alternative link here.

Click on "Download" to download the respective weights.

Then,

1. Create a folder called gpt2 in the location where you are planning to execute the code for loading the weights
2. Move the downloaded zip file into this gpt2 folder
3. Load the weights using the code shared in section 1 above

 

3) Alternative weight loading

In case you are still having problems, for example, because of a TensorFlow incompatibility (the original weights were stored as TensorFlow tensors. Hence, we need TensorFlow for the weight conversion to PyTorch), consider this alternative way of loading the weights from the Hugging Face Hub.

[jms-dipadua]

thank you for providing this alternative method. the option within the book wasn't working but Option 1 above did the trick :)

[rasbt]

glad it helped!

[robertfo]

I tried to do according to the book (section 5.5). Everything worked fine up to the command:

from gpt_download import download_and_load_gpt2
which gave the error message:
zsh: illegal hardware instruction python

Is there a simple cure? If not, I will try the proposed alternatives.

Versions

Apple M1 Pro
pip 24.0
tensorflow 2.16.2
python 3.12.4

[rasbt]

hm, could you inspect the download_and_load_gpt2.py file and see if it maybe has malformed contents? And maybe try to replace it by downloading a fresh copy from here: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_download.py

[robertfo]

Hi Sebastian, Nice getting in touch again… The ”gpt_download.py” file looks ok. I have also tried with the fresh copy that you suggest. In both cases the python command ”from gpt_download_and_load_gpt2” causes the hardware error. Robert 2 mars 2025 kl. 17:52 skrev Sebastian Raschka ***@***.***>: hm, could you inspect the download_and_load_gpt2.py file and see if it maybe has malformed contents? And maybe try to replace it by downloading a fresh copy from here: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_download.py — Reply to this email directly, view it on GitHub<#273 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB5DKLBMYOOTZAEF3CPYW3T2SMZLJAVCNFSM6AAAAABLERIUGSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZWG4YTCMY>. You are receiving this because you commented.Message ID: ***@***.***>

[rasbt]

Wow that's so weird. I have been using Python for ~14 years now and never seen anything like this.

1. I wonder what happens if you upload the file to Google Colab and run it there. Do you get the same error?
   If not, maybe your version of Python has a compatibility issue?

2. Another thing worth trying: Could you copy and paste the contents of the gpt_download_and_load_gpt2.py directly into your script/notebook so that you can skip the from gpt_download_and_load_gpt2 import... and see if it still happens? Maybe it will help narrowing down where exactly in the code it happens.

[robertfo]

The problem is that the file gpt_download_and_load_gpt2.py never loads due to the occuring error. Btw, are we talking about the same command? Below you mention "from gpt_download_and_load_gpt2 import…” while the book says "from gpt_download import….”? 2 mars 2025 kl. 18:47 skrev Sebastian Raschka ***@***.***>: Wow that's so weird. I have been using Python for ~14 years now and never seen anything like this. 1. I wonder what happens if you upload the file to Google Colab and run it there. Do you get the same error? If not, maybe your version of Python has a compatibility issue? 2. Another thing worth trying: Could you copy and paste the contents of the gpt_download_and_load_gpt2.py directly into your script/notebook so that you can skip the from gpt_download_and_load_gpt2 import... and see if it still happens? Maybe it will help narrowing down where exactly in the code it happens. — Reply to this email directly, view it on GitHub<#273 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB5DKLB26BZMTTC2ULWFAL32SM72PAVCNFSM6AAAAABLERIUGSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZWG4ZTSMQ>. You are receiving this because you commented.Message ID: ***@***.***>

[rasbt]

My apologies, yes, I meant the

from gpt_download import download_and_load_gpt2

command

[robertfo]

I copied and pasted the contents of ”gpt_download.py” into my python script. The error occurs at the line "import tensorflow as tf”. 2 mars 2025 kl. 18:59 skrev Robert Forchheimer ***@***.***>: The problem is that the file gpt_download_and_load_gpt2.py never loads due to the occuring error. Btw, are we talking about the same command? Below you mention "from gpt_download_and_load_gpt2 import…” while the book says "from gpt_download import….”? 2 mars 2025 kl. 18:47 skrev Sebastian Raschka ***@***.***>: Wow that's so weird. I have been using Python for ~14 years now and never seen anything like this. 1. I wonder what happens if you upload the file to Google Colab and run it there. Do you get the same error? If not, maybe your version of Python has a compatibility issue? 2. Another thing worth trying: Could you copy and paste the contents of the gpt_download_and_load_gpt2.py directly into your script/notebook so that you can skip the from gpt_download_and_load_gpt2 import... and see if it still happens? Maybe it will help narrowing down where exactly in the code it happens. — Reply to this email directly, view it on GitHub<#273 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB5DKLB26BZMTTC2ULWFAL32SM72PAVCNFSM6AAAAABLERIUGSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZWG4ZTSMQ>. You are receiving this because you commented.Message ID: ***@***.***>

[rasbt]

Hm, maybe there's been an issue with installing tensorflow on your computer. You could try to reinstall it via

pip uninstall -y tensorflow tensorflow-cpu tensorflow-gpu tensorflow-intel
pip cache purge  # Clear any cached TensorFlow packages
pip install --no-cache-dir tensorflow

If this still doesn't work, I do have alternative codes to load the weights without using TensorFlow. I just preferred this way in the book because these are the original and official weights provided by OpenAI.

The alternatives can be found here: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/02_alternative_weight_loading

[robertfo]

Thanks Sebastian, problem solved! I changed to another (conda) environment and installed tensorflow. That was sufficient. Robert 2 mars 2025 kl. 19:54 skrev Sebastian Raschka ***@***.***>: Hm, maybe there's been an issue with installing tensorflow on your computer. You could try to reinstall it via pip uninstall -y tensorflow tensorflow-cpu tensorflow-gpu tensorflow-intel pip cache purge # Clear any cached TensorFlow packages pip install --no-cache-dir tensorflow If this still doesn't work, I do have alternative codes to load the weights without using TensorFlow. I just preferred this way in the book because these are the original and official weights provided by OpenAI. The alternatives can be found here: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/02_alternative_weight_loading — Reply to this email directly, view it on GitHub<#273 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB5DKLCKIEBDMYHQCJMS74D2SNHWFAVCNFSM6AAAAABLERIUGSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZWG43TKNQ>. You are receiving this because you commented.Message ID: ***@***.***>

[rasbt]

I am glad you got it to work. I am adding a note (#555) about this to the code notebook in case someone else experiences similar issues.

[jms-dipadua]

apologies for a delay. was away from the laptop for a bit there -- i recalled having a little issue with the download but navigating your github repo and getting the files directly helped. i directly downloaded your py file with the following: ``` import urllib.request url = ( "https://raw.githubusercontent.com/rasbt/" "LLMs-from-scratch/main/ch05/" "01_main-chapter-code/gpt_download.py" ) filename = url.split("/")[-1] urllib.request.urlretrieve(url, filename) ``` so that when i called the following it worked without any issues: ``` from gpt_download import download_and_load_gpt2 settings, params = download_and_load_gpt2(model_size="355M", models_dir="gpt2") ```

…

On Sun, Mar 2, 2025 at 1:12 PM Sebastian Raschka ***@***.***> wrote: I am glad you got it to work. I am adding a note (#555 <#555>) about this to the code notebook in case someone else experiences similar issues. — Reply to this email directly, view it on GitHub <#273 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQCEVHIZSDNQN427IZZSUL2SNX3XAVCNFSM6AAAAABLERIUGSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZWHAZTQOA> . You are receiving this because you commented.Message ID: ***@***.*** com>

[rasbt]

Glad you got it to work!