feat: support CJK languages

Author: johannschopplich

README.md

@@ -1,6 +1,15 @@
 # tokenwise
 
-GPT token count and context size utilities when approximations are good enough.
+GPT token count and context size utilities when approximations are good enough. The following table shows the accuracy of the token count approximation for different input texts:
+
+| Description | Actual GPT Token Count | Estimated Token Count | Error Range (%) |
+| ----------- | ---------------------- | --------------------- | --------------- |
+| Short English text | 10 | 11 | 10.00% |
+| German text with umlauts | 56 | 49 | 12.50% |
+| Metamorphosis by Franz Kafka (English) | 31891 | 36557 | 14.63% |
+| Die Verwandlung by Franz Kafka (German) | 40620 | 37873 | 6.76% |
+| 道德經 by Laozi (Chinese) | 14386 | 12239 | 14.92% |
+| TypeScript ES5 Type Declarations (~ 4000 loc) | 47890 | 54829 | 14.49% |
 
 For advanced use cases, please use a full tokenizer like [`gpt-tokenizer`](https://github.com/niieani/gpt-tokenizer). This library is intended to be used for quick estimations and to avoid the overhead of a full tokenizer, e.g. when you want to limit your bundle size.
 

package.json

@@ -43,6 +43,7 @@
   ],
   "scripts": {
     "build": "unbuild",
+    "docs:generate": "tsx scripts/generateTable.ts",
     "dev": "unbuild --stub",
     "lint": "eslint .",
     "lint:fix": "eslint . --fix",
@@ -51,11 +52,13 @@
     "test:types": "tsc --noEmit"
   },
   "devDependencies": {
-    "@antfu/eslint-config": "^2.1.0",
+    "@antfu/eslint-config": "^2.1.1",
     "bumpp": "^9.2.0",
     "eslint": "^8.54.0",
+    "gpt-tokenizer": "^2.1.2",
+    "tsx": "^4.6.0",
     "typescript": "^5.3.2",
     "unbuild": "^2.0.0",
-    "vitest": "^0.34.6"
+    "vitest": "^1.0.0-beta.6"
   }
 }