feat: tokenizers, visualizer (#39)

* x (#35)

Co-authored-by: DjDeveloperr <hey@djdev.me>

* feat(WIP): data api

* feat: fix logging (#36)

* feat: Learning Rate Schedulers (#37)

* feat: learning rate schedulers

* fix decay schedulers

* fix: scheduler default to none

* feat: initial tokenizers crate

* feat(tokenizers): save tokenizers

* feat(tokenizers): save tokenizers to json files

* feat: gpu init

* feat: xor jupyter notebook

* feat: gpu init

* chore: spelling

* feat(WIP): visualizer

* feat: visualizer

* chore: cleanup

---------

Co-authored-by: DjDeveloperr <hey@djdev.me>
Co-authored-by: Pranev (NeTT) <nett@nett.moe>

.github/workflows/build.yml

@@ -31,8 +31,9 @@ jobs:
           set -xeuo pipefail
           rustc --version
           cargo --version
-          cargo build --release
-          deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts --out src/backend_wasm/lib
+          cargo build --release -p netsaur
+          deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts -p netsaur --out src/backends/wasm/lib
+          deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts -p netsaur-tokenizers --out tokenizers/lib
       - name: Release
         uses: softprops/action-gh-release@master
         env:
@@ -44,4 +45,5 @@ jobs:
             target/release/libnetsaur.so
             target/release/libnetsaur.dylib
             target/release/netsaur.dll
-            src/backend_wasm/lib/netsaur_bg.wasm
+            src/backends/wasm/lib/netsaur_bg.wasm
+            tokenizers/lib/netsaur_tokenizers_bg.wasm

.gitignore

@@ -14,3 +14,5 @@ target/
 
 *.test.bin
 *.test.st
+_test/run.ts
+_test/Student_Performance.csv

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing
 
-## Building `backend_cpu`
+## Building `backends/cpu`
 
 Unoptimized:
 
@@ -14,16 +14,16 @@ Optimized:
 cargo build --release
 ```
 
-## Building `backend_wasm`
+## Building `backends/wasm`
 
 Unoptimized:
 
 ```sh
-deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts --out src/backend_wasm/lib --debug
+deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts --out src/backends/wasm/lib --debug
 ```
 
 Optimized:
 
 ```sh
-deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts --out src/backend_wasm/lib
+deno run -A https://deno.land/x/wasmbuild@0.11.0/main.ts --out src/backends/wasm/lib
 ```