Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump version of tiktoken #1204

Merged
merged 4 commits into from Apr 24, 2024
Merged

Bump version of tiktoken #1204

merged 4 commits into from Apr 24, 2024

Conversation

minamorl
Copy link
Contributor

Describe the changes you have made:

Bumped version of tiktoken since build process is broken for some reason. This PR fixes broken process.

Reference any relevant issues (e.g. "Fixes #000"):

Not reported yet?

Pre-Submission Checklist (optional but appreciated):

  • I have included relevant documentation updates (stored in /docs)
  • I have read docs/CONTRIBUTING.md
  • I have read docs/ROADMAP.md

OS Tests (optional but appreciated):

  • Tested on Windows
  • Tested on MacOS
  • Tested on Linux

@dheavy
Copy link
Contributor

dheavy commented Apr 13, 2024

Hi @minamorl! Thanks for your PR!

Can you explain the detailed steps you have taken to install this? Specifically, have you installed the project in a virtual environment?
Furthermore, would you mind sharing the error message you've received?

I just installed the project from scratch in a new directory and could not reproduce the issue.

@minamorl
Copy link
Contributor Author

minamorl commented Apr 13, 2024 •
edited

Hi @dheavy . I really like this product.
I'm using pyenv. And installed it directly through pip install.

$pyenv version
3.12.1 (set by PYENV_VERSION environment variable)

So it might be pyenv specific issue. Hope this info would help!

@minamorl
Copy link
Contributor Author

minamorl commented Apr 13, 2024 •
edited

error dumps:

❯ pip install open-interpreter                                                                                                             
Collecting open-interpreter
  Using cached open_interpreter-0.2.0-py3-none-any.whl.metadata (14 kB)
Collecting appdirs<2.0.0,>=1.4.4 (from open-interpreter)
  Using cached appdirs-1.4.4-py2.py3-none-any.whl.metadata (9.0 kB)
Requirement already satisfied: astor<0.9.0,>=0.8.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (0.8.1)
Requirement already satisfied: git-python<2.0.0,>=1.0.3 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (1.0.3)
Requirement already satisfied: html2image<3.0.0.0,>=2.0.4.3 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (2.0.4.3)
Requirement already satisfied: inquirer<4.0.0,>=3.1.3 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (3.2.4)
Requirement already satisfied: ipykernel<7.0.0,>=6.26.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (6.29.4)
Requirement already satisfied: jupyter-client<9.0.0,>=8.6.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (8.6.1)
Requirement already satisfied: litellm<2.0.0,>=1.15.10 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (1.34.22)
Requirement already satisfied: matplotlib<4.0.0,>=3.8.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (3.8.4)
Requirement already satisfied: openai<2.0.0,>=1.6.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (1.13.3)
Requirement already satisfied: posthog<4.0.0,>=3.1.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (3.5.0)
Requirement already satisfied: psutil<6.0.0,>=5.9.6 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (5.9.8)
Requirement already satisfied: pyyaml<7.0.0,>=6.0.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (6.0.1)
Requirement already satisfied: rich<14.0.0,>=13.4.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (13.7.1)
Requirement already satisfied: six<2.0.0,>=1.16.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (1.16.0)
Collecting tiktoken<0.5.0,>=0.4.0 (from open-interpreter)
  Using cached tiktoken-0.4.0.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: tokentrim<0.2.0,>=0.1.13 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (0.1.13)
Requirement already satisfied: toml<0.11.0,>=0.10.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (0.10.2)
Requirement already satisfied: wget<4.0,>=3.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from open-interpreter) (3.2)
Requirement already satisfied: gitpython in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from git-python<2.0.0,>=1.0.3->open-interpreter) (3.1.43)
Requirement already satisfied: requests in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from html2image<3.0.0.0,>=2.0.4.3->open-interpreter) (2.31.0)
Requirement already satisfied: websocket-client<2.0.0,>=1.0.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from html2image<3.0.0.0,>=2.0.4.3->open-interpreter) (1.7.0)
Requirement already satisfied: blessed>=1.19.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from inquirer<4.0.0,>=3.1.3->open-interpreter) (1.20.0)
Requirement already satisfied: editor>=1.6.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from inquirer<4.0.0,>=3.1.3->open-interpreter) (1.6.6)
Requirement already satisfied: readchar>=3.0.6 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from inquirer<4.0.0,>=3.1.3->open-interpreter) (4.0.6)
Requirement already satisfied: comm>=0.1.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.2.2)
Requirement already satisfied: debugpy>=1.6.5 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (1.8.1)
Requirement already satisfied: ipython>=7.23.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (8.23.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (5.7.2)
Requirement already satisfied: matplotlib-inline>=0.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.1.6)
Requirement already satisfied: nest-asyncio in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (1.6.0)
Requirement already satisfied: packaging in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (24.0)
Requirement already satisfied: pyzmq>=24 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (25.1.2)
Requirement already satisfied: tornado>=6.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (6.4)
Requirement already satisfied: traitlets>=5.4.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipykernel<7.0.0,>=6.26.0->open-interpreter) (5.14.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from jupyter-client<9.0.0,>=8.6.0->open-interpreter) (2.9.0.post0)
Requirement already satisfied: aiohttp in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from litellm<2.0.0,>=1.15.10->open-interpreter) (3.9.3)
Requirement already satisfied: click in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from litellm<2.0.0,>=1.15.10->open-interpreter) (8.1.7)
Requirement already satisfied: importlib-metadata>=6.8.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from litellm<2.0.0,>=1.15.10->open-interpreter) (7.1.0)
Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from litellm<2.0.0,>=1.15.10->open-interpreter) (3.1.3)
Requirement already satisfied: python-dotenv>=0.2.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from litellm<2.0.0,>=1.15.10->open-interpreter) (1.0.1)
Requirement already satisfied: tokenizers in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from litellm<2.0.0,>=1.15.10->open-interpreter) (0.15.2)
Requirement already satisfied: contourpy>=1.0.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (1.2.1)
Requirement already satisfied: cycler>=0.10 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (4.51.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (1.4.5)
Requirement already satisfied: numpy>=1.21 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (1.26.4)
Requirement already satisfied: pillow>=8 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (10.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from matplotlib<4.0.0,>=3.8.2->open-interpreter) (3.1.2)
Requirement already satisfied: anyio<5,>=3.5.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (4.3.0)
Requirement already satisfied: distro<2,>=1.7.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (0.27.0)
Requirement already satisfied: pydantic<3,>=1.9.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (2.6.4)
Requirement already satisfied: sniffio in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (1.3.1)
Requirement already satisfied: tqdm>4 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (4.66.2)
Requirement already satisfied: typing-extensions<5,>=4.7 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from openai<2.0.0,>=1.6.1->open-interpreter) (4.10.0)
Requirement already satisfied: monotonic>=1.5 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from posthog<4.0.0,>=3.1.0->open-interpreter) (1.6)
Requirement already satisfied: backoff>=1.10.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from posthog<4.0.0,>=3.1.0->open-interpreter) (2.2.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from rich<14.0.0,>=13.4.2->open-interpreter) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from rich<14.0.0,>=13.4.2->open-interpreter) (2.17.2)
Requirement already satisfied: regex>=2022.1.18 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from tiktoken<0.5.0,>=0.4.0->open-interpreter) (2023.12.25)
Requirement already satisfied: idna>=2.8 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.6.1->open-interpreter) (3.6)
Requirement already satisfied: wcwidth>=0.1.4 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from blessed>=1.19.0->inquirer<4.0.0,>=3.1.3->open-interpreter) (0.2.13)
Requirement already satisfied: runs in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from editor>=1.6.0->inquirer<4.0.0,>=3.1.3->open-interpreter) (1.2.2)
Requirement already satisfied: xmod in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from editor>=1.6.0->inquirer<4.0.0,>=3.1.3->open-interpreter) (1.8.1)
Requirement already satisfied: certifi in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.6.1->open-interpreter) (2024.2.2)
Requirement already satisfied: httpcore==1.* in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.6.1->open-interpreter) (1.0.4)
Requirement already satisfied: h11<0.15,>=0.13 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=1.6.1->open-interpreter) (0.14.0)
Requirement already satisfied: zipp>=0.5 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from importlib-metadata>=6.8.0->litellm<2.0.0,>=1.15.10->open-interpreter) (3.18.1)
Requirement already satisfied: decorator in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.19.1)
Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (3.0.43)
Requirement already satisfied: stack-data in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.6.3)
Requirement already satisfied: pexpect>4.3 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (4.9.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from jinja2<4.0.0,>=3.1.2->litellm<2.0.0,>=1.15.10->open-interpreter) (2.1.5)
Requirement already satisfied: platformdirs>=2.5 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel<7.0.0,>=6.26.0->open-interpreter) (4.2.0)
Requirement already satisfied: mdurl~=0.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=13.4.2->open-interpreter) (0.1.2)
Requirement already satisfied: annotated-types>=0.4.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from pydantic<3,>=1.9.0->openai<2.0.0,>=1.6.1->open-interpreter) (0.6.0)
Requirement already satisfied: pydantic-core==2.16.3 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from pydantic<3,>=1.9.0->openai<2.0.0,>=1.6.1->open-interpreter) (2.16.3)
Requirement already satisfied: setuptools>=41.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from readchar>=3.0.6->inquirer<4.0.0,>=3.1.3->open-interpreter) (69.1.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from requests->html2image<3.0.0.0,>=2.0.4.3->open-interpreter) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from requests->html2image<3.0.0.0,>=2.0.4.3->open-interpreter) (2.2.1)
Requirement already satisfied: aiosignal>=1.1.2 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from aiohttp->litellm<2.0.0,>=1.15.10->open-interpreter) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from aiohttp->litellm<2.0.0,>=1.15.10->open-interpreter) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from aiohttp->litellm<2.0.0,>=1.15.10->open-interpreter) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from aiohttp->litellm<2.0.0,>=1.15.10->open-interpreter) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from aiohttp->litellm<2.0.0,>=1.15.10->open-interpreter) (1.9.4)
Requirement already satisfied: gitdb<5,>=4.0.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from gitpython->git-python<2.0.0,>=1.0.3->open-interpreter) (4.0.11)
Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from tokenizers->litellm<2.0.0,>=1.15.10->open-interpreter) (0.22.2)
Requirement already satisfied: smmap<6,>=3.0.1 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from gitdb<5,>=4.0.1->gitpython->git-python<2.0.0,>=1.0.3->open-interpreter) (5.0.1)
Requirement already satisfied: filelock in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->litellm<2.0.0,>=1.15.10->open-interpreter) (3.13.3)
Requirement already satisfied: fsspec>=2023.5.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->litellm<2.0.0,>=1.15.10->open-interpreter) (2024.3.1)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.8.4)
Requirement already satisfied: ptyprocess>=0.5 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from pexpect>4.3->ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.7.0)
Requirement already satisfied: executing>=1.2.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from stack-data->ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (2.0.1)
Requirement already satisfied: asttokens>=2.1.0 in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from stack-data->ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (2.4.1)
Requirement already satisfied: pure-eval in /home/minamorl/.pyenv/versions/3.12.1/lib/python3.12/site-packages (from stack-data->ipython>=7.23.1->ipykernel<7.0.0,>=6.26.0->open-interpreter) (0.2.2)
Using cached open_interpreter-0.2.0-py3-none-any.whl (110 kB)
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Building wheels for collected packages: tiktoken
  Building wheel for tiktoken (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tiktoken (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [37 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-312
      creating build/lib.linux-x86_64-cpython-312/tiktoken
      copying tiktoken/core.py -> build/lib.linux-x86_64-cpython-312/tiktoken
      copying tiktoken/load.py -> build/lib.linux-x86_64-cpython-312/tiktoken
      copying tiktoken/model.py -> build/lib.linux-x86_64-cpython-312/tiktoken
      copying tiktoken/registry.py -> build/lib.linux-x86_64-cpython-312/tiktoken
      copying tiktoken/__init__.py -> build/lib.linux-x86_64-cpython-312/tiktoken
      creating build/lib.linux-x86_64-cpython-312/tiktoken_ext
      copying tiktoken_ext/openai_public.py -> build/lib.linux-x86_64-cpython-312/tiktoken_ext
      running egg_info
      writing tiktoken.egg-info/PKG-INFO
      writing dependency_links to tiktoken.egg-info/dependency_links.txt
      writing requirements to tiktoken.egg-info/requires.txt
      writing top-level names to tiktoken.egg-info/top_level.txt
      reading manifest file 'tiktoken.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'Makefile'
      adding license file 'LICENSE'
      writing manifest file 'tiktoken.egg-info/SOURCES.txt'
      copying tiktoken/py.typed -> build/lib.linux-x86_64-cpython-312/tiktoken
      running build_ext
      running build_rust
      error: can't find Rust compiler

      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

      To update pip, run:

          pip install --upgrade pip

      and then retry package installation.

      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tiktoken
Failed to build tiktoken
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

pip version is updated. Rust compiler missing error is solved in bumping version.

@dheavy
Copy link
Contributor

dheavy commented Apr 13, 2024 •
edited

@minamorl pyenv is primary a tool for managing several different versions of Python on the same machine but does not provide a virtual environment by itself.

A virtual environment for a Python project is a way to define a "bubble" within which you can install your project dependencies (like tiktoken) without interference with previous installations from other projects.

It's just an educated guess, but would you go check a software called pyenv-virtualenv, use it to create a virtual environment for the project, and try to reinstall the project inside the virtual env using pip?

This way at the very least we can verify if the issue stems from tiktoken or if it's just your previous global installations that do not work well together.

@minamorl
Copy link
Contributor Author

@dheavy Hi, I tried to set up pyenv-virtualenv and pip install open-interpreter inside the context. But got same result.
I think tiktoken package itself is broken or requiring rust compiler wrongly and it was fixed in latest version.

@minamorl
Copy link
Contributor Author

Anyway, if this PR is not affect to other codes, it's good to keep up with latest version. This PR is just bump version.

@minamorl
Copy link
Contributor Author

I've been looking into it, and given that there are a lot of similar issues up, I think it's natural to assume that they've been fixed in the latest version.
openai/tiktoken#215

@dheavy
Copy link
Contributor

dheavy commented Apr 13, 2024 •
edited

Thank you @minamorl!

(A disclaimer, I'm not a maintainer for this repo, just a contributor like yourself.)

Bumping a dependency can always have side effects — it may work for you, but stop working for other people. This is why I'm trying to find the actual reason why you face this issue and I can't reproduce it. Given the link you have provided, this seems related specifically to Python 3.12.

I'm going to try install the bumped version on 3.10 and 3.11 to make sure it doesn't suddenly break installation there, and will report back. This way the maintainer who will review this PR will know whether or not it is free from regressions. Thanks again for finding this @minamorl!

@dheavy
Copy link
Contributor

dheavy commented Apr 13, 2024 •
edited

Maintainers: this has been tested on macOS, both Python 10 and 11 (brand new installs) without any regressions 🎉

@minamorl there's just one little thing missing. Could you please install poetry and run poetry lock --no-update in the project to update the lock file, then add it to your PR?

Without it, the lock file mismatches the dependency file (pyproject.toml), and installation fails.

Screenshot 2024-04-13 at 22 02 36

Other than that, great job spotting this!

@KillianLucas
Copy link
Collaborator

Thank you so much @minamorl for spotting this— several issues have raised tiktoken installation problems, so it's a priority to keep up with any fixes they've made (I wish they'd just make a pure Python version!).

And thank you so much @dheavy for testing and thoughtfully tracking down the source of the issue, ensuring it's free of regressions! Makes it very easy to improve the repo quickly and confidently. ❤️

Let me know if you run into any issues regenerating the poetry lockfile @minamorl, I think I can do it if you run into any trouble. Will then merge. Great work!

@minamorl
Copy link
Contributor Author

@KillianLucas Hi, I've tested with poetry and it works fine.

@minamorl
Copy link
Contributor Author

Just in case double-check pls 🙏

@minamorl
Copy link
Contributor Author

minamorl commented Apr 18, 2024 •
edited

I resolved merge conflict

@KillianLucas KillianLucas merged commit c7fcc7e into OpenInterpreter:main Apr 24, 2024
@KillianLucas
Copy link
Collaborator

Awesome! Thanks @minamorl, merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@minamorl @dheavy @KillianLucas