feat(server): use native tokenizer impl

[Brooooooklyn]

Benchmark

yarn workspace @affine/server-native bench

┌─────────┬────────────┬─────────┬────────────────────┬──────────┬─────────┐
│ (index) │ Task Name  │ ops/sec │ Average Time (ns)  │ Margin   │ Samples │
├─────────┼────────────┼─────────┼────────────────────┼──────────┼─────────┤
│ 0       │ 'tiktoken' │ '5'     │ 176932518.76000002 │ '±4.71%' │ 100     │
│ 1       │ 'native'   │ '16'    │ 61041597.51000003  │ '±0.60%' │ 100     │
└─────────┴────────────┴─────────┴────────────────────┴──────────┴─────────┘

[graphite-app]

Your org has enabled the Graphite merge queue for merging into canary

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[Brooooooklyn]

• feat(server): use native tokenizer impl #6960 👈
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Brooooooklyn and the rest of your teammates on Graphite

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit c7ddd67. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 6 targets

• nx test:coverage @affine/monorepo
• nx build @affine/native -- --target x86_64-pc-windows-msvc --use-napi-cross
• nx build @affine/native -- --target x86_64-apple-darwin --use-napi-cross
• nx build @affine/native -- --target x86_64-unknown-linux-gnu --use-napi-cross
• nx build @affine/native -- --target aarch64-apple-darwin --use-napi-cross
• nx build @affine/server-native -- --target x86_64-unknown-linux-gnu --use-napi-cross

---

Sent with 💌 from NxCloud.

[codecov]

Codecov Report

Attention: Patch coverage is 55.00000% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 66.58%. Comparing base (4614003) to head (c7ddd67).

Files	Patch %	Lines
...ckages/backend/server/src/plugins/copilot/types.ts	50.00%	4 Missing ⚠️
packages/backend/server/src/native.ts	60.00%	2 Missing ⚠️
...kages/backend/server/src/plugins/copilot/prompt.ts	50.00%	2 Missing ⚠️
...ages/backend/server/src/plugins/copilot/session.ts	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           canary    #6960      +/-   ##
==========================================
- Coverage   66.65%   66.58%   -0.07%     
==========================================
  Files         582      582              
  Lines       27392    27384       -8     
  Branches     2573     2572       -1     
==========================================
- Hits        18257    18233      -24     
- Misses       8865     8881      +16     
  Partials      270      270              

Flag	Coverage Δ
server-test	77.33% <55.00%> (-0.09%)	⬇️
unittest	39.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[graphite-app]

Merge activity

• May 16, 3:33 AM EDT: Brooooooklyn added this pull request to the Graphite merge queue.
• May 16, 4:10 AM EDT: Brooooooklyn merged this pull request with the Graphite merge queue.