There seems to be a problem with tokenizer installation

[rahilshah105]

I have install tokenizer using when I try to call it with:

import tiktoken

tokenizer = tiktoken.get_encoding("cl100k_base")

df = pd.read_csv('processed/scraped.csv', index_col=0)
df.columns = ['title', 'text']

df['n_tokens'] = df.text.apply(lambda x: len(tokenizer.encode(x)))

df.n_tokens.hist()

I get the following problem

ModuleNotFoundError Traceback (most recent call last)
Cell In [1], line 1
----> 1 import tiktoken
3 # Load the c1100k base tokenizer which is designed to work with the ada-002 model
4 tokenizer = tiktoken.get_encoding("c1100k_base")
ModuleNotFoundError: No module named 'tiktoken'

[hauntsaninja]

pip install tiktoken is the way to install tiktoken

[rahilshah105]

Hi. I have already installed tiktoken using pip install tiktoken and got this problem