ICGPT

---

Try it out on the IC !

---

The full application consists of 2 GitHub repositories:

1. icgpt (This repo)
2. icpp_llm

Setup

Conda

Download MiniConda and then install it:

bash Miniconda3-xxxxx.sh

Create a conda environment with NodeJS & Python 3.11:

conda create --name icgpt nodejs python=3.11
conda activate icgpt

git

git clone git@github.com:icppWorld/icgpt.git
cd icgpt

pre-commit

Create this pre-commit script, file .git/hooks/pre-commit

#!/bin/bash

# Apply all static auto-formatting & perform the static checks
export PATH="$HOME/miniconda3/envs/icgpt/bin:$PATH"
/usr/bin/make all-static

and make the script executable:

chmod +x .git/hooks/pre-commit

toolchain & dependencies

Install the toolchain:

• The dfx release version is specified in dfx.json

conda activate icgpt
make install-all-ubuntu  # for Ubuntu.
make install-all-mac     # for Mac.
                         # see Makefile to replicate for other systems

# ~/bin must be on path
source ~/.profile

# Verify all tools are available
dfx --version

# verify all other items are working
conda activate icgpt
make all-static-check

Development

The backend LLM canisters

ICGPT includes LLM backend canisters of icpp_lmm:

• Clone icpp_lmm as a sibling to this repo
• Follow instructions of llama2_c to :
  • Build the wasm
  • Get the model checkpoints

The following files are used by the ICGPT deployment steps:

../icpp_llm/llama2_c/src/llama2.did
../icpp_llm/llama2_c/build/llama2.wasm
../icpp_llm/llama2_c/scripts/upload.py

#
# For each of the backend canisters you're including
#
../icpp_llm/llama2_c/stories260K/stories260K.bin
../icpp_llm/llama2_c/stories260K/tok512.bin

../icpp_llm/llama2_c/tokenizers/tok4096.bin
../icpp_llm/llama2_c/models/stories15Mtok4096.bin

../icpp_llm/llama2_c/tokenizers/tokenizer.bin
../icpp_llm/llama2_c/models/stories42M.bin
../icpp_llm/llama2_c/models/stories110M.bin

Deploy to local network

Once the files of the backend LLMs are in place, as described in the previous step, you can deploy everything with:

# Start the local network
dfx start --clean

# In another terminal

# Deploy all wasms listed in dfx.json
dfx deploy

# Upload the LLM models to the backend canisters
make upload-260K-local
make upload-15M-local
make upload-42M-local
make upload-110M-local
# Or alternatively
make upload-all-local

# Note: you can stop the local network with
dfx stop

After the deployment steps described above, the full application is now deployed to the local network, including the front-end canister, the LLM back-end canisters, and the internet_identity canister:

You can now open the front-end in the browser at the URL printed by the deploy script:

When you login, just create a new II, and once login completed, you will see the start screen shown at the top of this README. Now you can play with it and have some fun !

Front-end Development

The front-end is a react application with a webpack based build pipeline. Webpack builds with sourcemaps, you can also use the following front-end development workflow:

• Deploy the full application to the local network, as described in previous step

• Do not open the front-end deployed to the local network, but instead run the front-end with the npm development server:

  # from root directory
  
  conda activate icgpt
  
  # start the npm development server, with hot reloading
  npm run start
  
  # to rebuild from scratch
  npm run build

• Open the browser at the URL printed & open the browser devtools for debugging

• Make changes to the front-end code in your favorite editor, and when you save it, everything will auto-rebuild and auto-reload

Styling with Dracula UI

All front-end color styling is done using the open source Dracula UI:

• github
• user guide

Deployment to IC

Step 0: When deploying for the first time:

• Delete canister_ids.json, because when you forked or cloned the github repo icgpt, it contained the canisters used by our deployment at https://icgpt.icpp.world/

Step 1: Build the backend wasm files

• Clone icpp_llm and follow the instructions in llama2_c to build the wasm for each backend canister.

Step 2: Deploy the backend canisters

• Note that dfx.json points to the wasm files build during Step 1

  # Deploy
  dfx deploy --ic llama2_260K -m reinstall
  dfx deploy --ic llama2_15M -m reinstall
  dfx deploy --ic llama2_42M -m reinstall
  dfx deploy --ic llama2_110M -m reinstall
  
  # Upload the LLM models to the backend canisters
  make upload-260K-ic
  make upload-15M-ic
  make upload-42M-ic
  make upload-110M-ic
  # Or, alternatively
  make upload-all-ic
  
  #--------------------------------------------------------------------------
  # IMPORTANT: ic-py might throw a timeout => patch it here:
  # Ubuntu:
  # /home/<user>/miniconda3/envs/<your-env>/lib/python3.11/site-packages/httpx/_config.py
  # Mac:
  # /Users/<user>/miniconda3/envs/<your-env>/lib/python3.11/site-packages/httpx/_config.py
  # DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0)
  DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=99999999.0)
  # And perhaps here:
  # Ubuntu:
  # /home/<user>/miniconda3/envs/<your-env>/lib/python3.11/site-packages/httpcore/_backends/sync.py #L28-L29
  # Mac:
  # /Users/<user>/miniconda3/envs/<your-env>/lib/python3.11/site-packages/httpcore/_backends/sync.py #L28-L29
  #
  class SyncStream(NetworkStream):
      def __init__(self, sock: socket.socket) -> None:
          self._sock = sock
  
      def read(self, max_bytes: int, timeout: typing.Optional[float] = None) -> bytes:
          exc_map: ExceptionMapping = {socket.timeout: ReadTimeout, OSError: ReadError}
          with map_exceptions(exc_map):
              # PATCH AB
              timeout = 999999999
              # ENDPATCH
              self._sock.settimeout(timeout)
              return self._sock.recv(max_bytes)
  # ------------------------------------------------------------------------
  

Step 3: deploy the frontend

• Now that the backend is in place, the frontend can be deployed

  # from root directory
  conda activate icgpt
  
  dfx identity use <identity-of-controller>
  
  # This deploys just the frontend!
  dfx deploy --ic canister_frontend

Verify

scripts/ready.sh --network [local/ic]

Check cycle balance

scripts/balance.sh --network [local/ic]

Top up cycles

# Edit the value of TOPPED_OFF_BALANCE_T in the script.
scripts/top-off.sh --network [local/ic]

Appendix A - NOTES

process.env.CANISTERID

The generated declarations and in our own front-end code the canister Ids are defined with process.env.CANISTER_ID_<NAME>.

The way that these environment variables are created is:

• The command dfx deploy maintains a section in the file .env where it stores the canister id for every deployed canister.
• The commands npm build/run use webpack.config.js, where the webpack.EnvironmentPlugin is used to define the values.

Internet Identity

icgpt is using internet identity for authentication.

When deploying locally, the internet_identity canister will be installed automatically during the make dfx-deploy-local or dfx deploy --network local command. It uses the instructions provided in dfx.json.

When deploying to IC, it will NOT be deployed.

For details, see this forum post.