tiktoken is not running in jupyter notebook

[minaf1]

Hello Razbt,
Nice to meet you! I've been enjoying your book so far (LLMs from scratch), but I find the examples hard to follow as some of the tools used do not mention which versions you used. I tried to follow along but packages like tiktoken and pytorch refuse to work, or even get installed. I tried using conda to install environments with both Python 3.9 and 3.10. and both successfully install tiktoken, but fail to import it in the jupyter notebook. The command I ran to attempt installation was pip install tiktoken.

Can you let me know which version of Python / tiktoken / pytorch you were using? Is there any intermediate step I missed?

I am running Windows 11 and an (non-Nividia) GPU.

[rasbt]

Hi there,
thanks for your interest in my book!

as some of the tools used do not mention which versions you used.

I just double-checked the chapter text and code notebook, and it's tiktoken 0.5.1.

Regarding PyTorch, I used 2.0.1

but newer versions also work.

Please let me know if the version for any external package is missing, and I'd be happy to add it.

Python 3.9 and 3.10. and both successfully install tiktoken, but fail to import it in the jupyter notebook.

From my teaching experience, this is usually very rare, but it can unfortunately be a frustrating experience sometimes when it happens. This is usually not a Python or package version issue but related to the operating system and computer and how Python and Jupyter were installed. What can happen is that on some computers the pip version installs it in an environment that is different from that of the notebook.

Sometimes, the easy fix is to restart Jupyter notebook after pip-installing the package.

Alternatively, I uploaded the code notebook here to Google Colab: https://colab.research.google.com/drive/1tnfoUBz9MraKco-UdDiqr97J418xFTDi?usp=sharing

Please let me know if you have any follow-up questions, I'm happy to help.

[minaf1]

I do think you are correct about the Python & Jupyter installation being the issue. I haven't isolated the actual issue yet, but for now installing the package directly on the jupyter notebook (or colab) seems to work.
Thanks Rasbt