failed to deployment

[yibie]

run install command pip3 install khoj-assistant in shell

then get warn message:

 × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Error in sitecustomize; set PYTHONVERBOSE for traceback:
      AssertionError:
      Traceback (most recent call last):
        File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
          self.run_setup()
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
          exec(code, locals())
        File "<string>", line 2, in <module>
      ModuleNotFoundError: No module named 'setuptools_rust'
      [end of output]

[debanjum]

Hey @yibie, try run the commands below and see if that fixes your issue?

pip3 install setuptools-rust
pip3 install --upgrade pip
pip3 install --upgrade khoj-assistant

It seems like setuptools-rust package with an upgraded pip is required based on this github comment and this stackoverflow answer

[yibie]

thx to repsond.
but there are still some problem here, even i was deployed setuptools-rust.

here shows some message:

running build_ext running build_rust error: can't find Rust compiler

and

Applications $ python3 -m pip install --upgrade pip

Requirement already satisfied: pip in /opt/homebrew/lib/python3.10/site-packages (22.2.2)

/Applications $ pip3 install setuptools-rust

Requirement already satisfied: setuptools-rust in /opt/homebrew/lib/python3.10/site-packages (1.5.1)

Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/homebrew/lib/python3.10/site-packages (from setuptools-rust) (4.3.0)

Requirement already satisfied: setuptools>=62.4 in /opt/homebrew/lib/python3.10/site-packages (from setuptools-rust) (63.4.3)

Requirement already satisfied: semantic-version<3,>=2.8.2 in /opt/homebrew/lib/python3.10/site-packages (from setuptools-rust) (2.10.0)

[debanjum]

Hey @yibie I suspect the new issue you're facing is related to huggingface/transformers#2831.

Fix

The easiest way to fix it would be to install rust with below commands. Let me know if that doesn't resolve it for you.

brew install rustup
rustup-init
source ~/.cargo/env
pip install khoj-assistant

Details

I've seen the issue before while building the tokenizer package. It's trying to build the package using rust as it didn't find a binary wheel for your OS/architecture.

1. What OS are you on?
2. Can you share more of the logs around the rust compiler missing message, i.e running build_ext running build_rust error: can't find Rust compiler? It'll tell which package is trying to use rust

[yibie]

problem still. i tried twice, and use your commands.

1. macOS 12.5.1, python 3.10.6
2. here are the complete logs. hopes helpful.

/Applications $ pip3 install khoj-assistant
Collecting khoj-assistant
  Using cached khoj_assistant-0.1.8-py3-none-any.whl (433 kB)
Collecting pyyaml==6.0
  Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Collecting fastapi==0.77.1
  Using cached fastapi-0.77.1-py3-none-any.whl (54 kB)
Collecting sentence-transformers==2.1.0
  Using cached sentence_transformers-2.1.0-py3-none-any.whl
Collecting uvicorn==0.17.6
  Using cached uvicorn-0.17.6-py3-none-any.whl (53 kB)
Collecting pydantic==1.9.1
  Using cached pydantic-1.9.1-cp310-cp310-macosx_11_0_arm64.whl (2.4 MB)
Collecting aiofiles==0.8.0
  Using cached aiofiles-0.8.0-py3-none-any.whl (13 kB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp310-none-macosx_11_0_arm64.whl (49.1 MB)
Collecting pytest==7.1.2
  Using cached pytest-7.1.2-py3-none-any.whl (297 kB)
Collecting pillow==9.2.0
  Using cached Pillow-9.2.0-cp310-cp310-macosx_11_0_arm64.whl (2.8 MB)
Collecting transformers==4.21.0
  Using cached transformers-4.21.0-py3-none-any.whl (4.7 MB)
Collecting huggingface-hub==0.8.1
  Using cached huggingface_hub-0.8.1-py3-none-any.whl (101 kB)
Collecting torchvision==0.13.1
  Using cached torchvision-0.13.1-cp310-cp310-macosx_11_0_arm64.whl (1.2 MB)
Collecting numpy==1.22.4
  Using cached numpy-1.22.4-cp310-cp310-macosx_11_0_arm64.whl (12.8 MB)
Collecting jinja2==3.1.2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting openai==0.20.0
  Using cached openai-0.20.0-py3-none-any.whl
Requirement already satisfied: pyqt6==6.3.1 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from khoj-assistant) (6.3.1)
Collecting dateparser==1.1.1
  Using cached dateparser-1.1.1-py2.py3-none-any.whl (288 kB)
Collecting pytz
  Using cached pytz-2022.2.1-py2.py3-none-any.whl (500 kB)
Collecting regex!=2019.02.19,!=2021.8.27,<2022.3.15
  Using cached regex-2022.3.2-cp310-cp310-macosx_11_0_arm64.whl (281 kB)
Collecting tzlocal
  Using cached tzlocal-4.2-py3-none-any.whl (19 kB)
Collecting python-dateutil
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting starlette==0.19.1
  Using cached starlette-0.19.1-py3-none-any.whl (63 kB)
Collecting tqdm
  Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting filelock
  Using cached filelock-3.8.0-py3-none-any.whl (10 kB)
Collecting requests
  Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting packaging>=20.9
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/homebrew/lib/python3.10/site-packages (from huggingface-hub==0.8.1->khoj-assistant) (4.3.0)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_universal2.whl (17 kB)
Collecting pandas-stubs>=1.1.0.11
  Using cached pandas_stubs-1.4.4.220906-py3-none-any.whl (135 kB)
Collecting pandas>=1.2.3
  Using cached pandas-1.4.4-cp310-cp310-macosx_11_0_arm64.whl (10.4 MB)
Collecting openpyxl>=3.0.7
  Using cached openpyxl-3.0.10-py2.py3-none-any.whl (242 kB)
Requirement already satisfied: PyQt6-Qt6>=6.3.0 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from pyqt6==6.3.1->khoj-assistant) (6.3.1)
Requirement already satisfied: PyQt6-sip<14,>=13.4 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from pyqt6==6.3.1->khoj-assistant) (13.4.0)
Collecting py>=1.8.2
  Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting attrs>=19.2.0
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting sentencepiece
  Using cached sentencepiece-0.1.97-cp310-cp310-macosx_11_0_arm64.whl (1.1 MB)
Collecting nltk
  Using cached nltk-3.7-py3-none-any.whl (1.5 MB)
Collecting scipy
  Using cached scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl (29.9 MB)
Collecting scikit-learn
  Using cached scikit_learn-1.1.2-cp310-cp310-macosx_12_0_arm64.whl (7.7 MB)
Collecting tokenizers>=0.10.3
  Using cached tokenizers-0.12.1.tar.gz (220 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting h11>=0.8
  Using cached h11-0.13.0-py3-none-any.whl (58 kB)
Collecting asgiref>=3.4.0
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting anyio<5,>=3.4.0
  Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
Collecting et-xmlfile
  Using cached et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting types-pytz>=2022.1.1
  Using cached types_pytz-2022.2.1.0-py3-none-any.whl (4.5 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Requirement already satisfied: charset-normalizer<3,>=2 in /Users/chenyibin/Library/Python/3.10/lib/python/site-packages (from requests->huggingface-hub==0.8.1->khoj-assistant) (2.1.1)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.6.15.1-py3-none-any.whl (160 kB)
Collecting joblib
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting pytz-deprecation-shim
  Using cached pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)
Collecting sniffio>=1.1
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting tzdata
  Using cached tzdata-2022.2-py2.py3-none-any.whl (336 kB)
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [53 lines of output]
      Error in sitecustomize; set PYTHONVERBOSE for traceback:
      AssertionError:
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-12-arm64-cpython-310
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers
      copying py_src/tokenizers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/models
      copying py_src/tokenizers/models/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/models
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/decoders
      copying py_src/tokenizers/decoders/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/decoders
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/normalizers
      copying py_src/tokenizers/normalizers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/normalizers
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/pre_tokenizers
      copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/pre_tokenizers
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/processors
      copying py_src/tokenizers/processors/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/processors
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/trainers
      copying py_src/tokenizers/trainers/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/trainers
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/implementations
      creating build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
      copying py_src/tokenizers/tools/__init__.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
      copying py_src/tokenizers/tools/visualizer.py -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
      copying py_src/tokenizers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers
      copying py_src/tokenizers/models/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/models
      copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/decoders
      copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/normalizers
      copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/pre_tokenizers
      copying py_src/tokenizers/processors/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/processors
      copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.macosx-12-arm64-cpython-310/tokenizers/trainers
      copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
      running build_ext
      running build_rust
      error: can't find Rust compiler
      
      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
      
      To update pip, run:
      
          pip install --upgrade pip
      
      and then retry package installation.
      
      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

[debanjum]

Yup, that helps! As suspected, the tokenizer package doesn't seem to have a wheel binary for Mac ARM64 architecture it can directly download. To build the tokenizer package locally, it needs rust. And you don't have rust so it fails.

Setting up rust (as suggested in my previous comment) should resolve this issue. Let me know if that doesn't fix it.

@yibie Are you sure rust is installed? What does rustc --version return?

[yibie]

sure, i installed rust.

/Applications % rustc --version 
rustc 1.63.0 (4b91a6ea7 2022-08-08)

[debanjum]

I see. Interesting that the tokenizer package build is still failing with same error:

 copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-12-arm64-cpython-310/tokenizers/tools
 running build_ext
 running build_rust
 error: can't find Rust compiler

Let me try figure out why it's still failing for you 🤔.

In the meantime, it'd be great if you could try install khoj in a new terminal with:
pip install --no-cache-dir --upgrade khoj-assistant
Just to make sure it's not some stale cache issue

[yibie]

i'm not sure, but it looks succeed installed khoj on my mac.

here are the messages, judge by you. thx again.

Building wheels for collected packages: openai, sentence-transformers, tokenizers
  Building wheel for openai (pyproject.toml) ... done
  Created wheel for openai: filename=openai-0.20.0-py3-none-any.whl size=54094 sha256=ca0db0a2efe7bf5d750cf0ccdc6fd5af72101c82fa3628b17fd61d40b4a7ee1b
  Stored in directory: /private/var/folders/ns/sfzfbcd16d19rky4mtm4gpbm0000gn/T/pip-ephem-wheel-cache-mf1lu864/wheels/42/bd/1d/f1d7e85562515ca1b9504a9450c89623329c0e24af7c749dc6
  Building wheel for sentence-transformers (setup.py) ... done
  Created wheel for sentence-transformers: filename=sentence_transformers-2.1.0-py3-none-any.whl size=120986 sha256=e19c50682f8de4f742b48a7297db55fad8ca1a3eea13fdbf817699b19946afdf
  Stored in directory: /private/var/folders/ns/sfzfbcd16d19rky4mtm4gpbm0000gn/T/pip-ephem-wheel-cache-mf1lu864/wheels/7b/ed/fd/16b8222e673f5eaa48ed71a0a2a6b66767b5e31bea0e5d3895
  Building wheel for tokenizers (pyproject.toml) ... done
  Created wheel for tokenizers: filename=tokenizers-0.12.1-cp310-cp310-macosx_12_0_arm64.whl size=3554034 sha256=4745a415cd94ea827038a4fb216f5b4fdfd459c09a19b21231f5cc534090ebcc
  Stored in directory: /private/var/folders/ns/sfzfbcd16d19rky4mtm4gpbm0000gn/T/pip-ephem-wheel-cache-mf1lu864/wheels/bd/22/bc/fa8337ce1ccf384c8fc4c1dbfa9cb1687934c0f24719082d49
Successfully built openai sentence-transformers tokenizers
Installing collected packages: types-pytz, tokenizers, sentencepiece, pytz, iniconfig, urllib3, tzdata, tqdm, torch, tomli, threadpoolctl, sniffio, six, regex, pyyaml, pyparsing, pydantic, py, pluggy, pillow, pandas-stubs, numpy, MarkupSafe, joblib, idna, h11, filelock, et-xmlfile, click, certifi, attrs, asgiref, aiofiles, uvicorn, scipy, requests, pytz-deprecation-shim, python-dateutil, packaging, openpyxl, nltk, jinja2, anyio, tzlocal, torchvision, starlette, scikit-learn, pytest, pandas, huggingface-hub, transformers, openai, fastapi, dateparser, sentence-transformers, khoj-assistant
Successfully installed MarkupSafe-2.1.1 aiofiles-0.8.0 anyio-3.6.1 asgiref-3.5.2 attrs-22.1.0 certifi-2022.6.15.1 click-8.1.3 dateparser-1.1.1 et-xmlfile-1.1.0 fastapi-0.77.1 filelock-3.8.0 h11-0.13.0 huggingface-hub-0.8.1 idna-3.3 iniconfig-1.1.1 jinja2-3.1.2 joblib-1.1.0 khoj-assistant-0.1.8 nltk-3.7 numpy-1.22.4 openai-0.20.0 openpyxl-3.0.10 packaging-21.3 pandas-1.4.4 pandas-stubs-1.4.4.220906 pillow-9.2.0 pluggy-1.0.0 py-1.11.0 pydantic-1.9.1 pyparsing-3.0.9 pytest-7.1.2 python-dateutil-2.8.2 pytz-2022.2.1 pytz-deprecation-shim-0.1.0.post0 pyyaml-6.0 regex-2022.3.2 requests-2.28.1 scikit-learn-1.1.2 scipy-1.9.1 sentence-transformers-2.1.0 sentencepiece-0.1.97 six-1.16.0 sniffio-1.3.0 starlette-0.19.1 threadpoolctl-3.1.0 tokenizers-0.12.1 tomli-2.0.1 torch-1.12.1 torchvision-0.13.1 tqdm-4.64.1 transformers-4.21.0 types-pytz-2022.2.1.0 tzdata-2022.2 tzlocal-4.2 urllib3-1.26.12 uvicorn-0.17.6
WARNING: There was an error checking the latest version of pip.

[debanjum]

Yup, looks like the tokenizer package installed successfully now! It must have been some stale cache issue that didn't see that rust was installed. If you run khoj you should see the configuration GUI pop up

[yibie]

i run khoj , and see the configuration GUI pop up. done. thx a lot.

and a advice, could khoj search multiple org files?

[debanjum]

That's great! Yes, khoj supports multiple org files.

• Just click the "Add" button multiple times on the Configure GUI to add specific org files. Example shown below:
  
• The GUI (currently) doesn't provide a way to add all org files under a directory. But you can do that by manually editing the input-filter field in the ~/.khoj/khoj.yml file. Example shown below: