Skip to content

v1.1.4

Choose a tag to compare

View all tags
@github-actions github-actions released this 13 Nov 13:57
· 11 commits to main since this release
v1.1.4
5f33d47

[1.1.4] - 2025-11-13

Added

  • Interactive Token Comparison Tool - /docs/tokenizer.html
    • Real GPT tokenization using gpt-tokenizer library
    • Compare token counts across 6 formats: Pretty JSON, JSON, YAML, TOON, ASON, CSV
    • Visual token highlighting - each token displayed with unique color
    • 16 pastel color palette for token visualization
    • Preset datasets: small-simple, small-nested, medium-users, large-complex (Stripe)
    • Custom JSON input support with live analysis
    • Dynamic baseline comparison (compare any format against any other)
    • All permutations comparison table showing all datasets
    • Compact Vercel-style UI matching docs design
    • Local TOON library integration (/docs/js/toon.js + wrapper)
    • Interactive hover effects on tokens (scale + shadow)
    • Responsive grid layout for format cards
    • Key findings section explaining token efficiency

Changed

  • CI/CD - Automated Publishing with Provenance

    • Updated /.github/workflows/npm-publish.yml with npm provenance support
    • Added --provenance flag to npm publish command
    • Added permissions.id-token: write for cryptographic signing
    • Package now displays verified checkmark ✓ on npm
    • Automated GitHub release creation with changelogithub
    • Supports both manual releases and git tag triggers (v*)
    • Node.js 20.x (updated from 18.x)
    • Full transparency log and build attestation

  • Documentation Navigation

    • Added "Token Comparison" link to main docs navigation
    • Blue highlight for new tokenizer tool link
    • Consistent navigation across all doc pages

Fixed

  • Tokenizer Library Loading

    • Replaced tiktoken (ES module issues) with gpt-tokenizer
    • Added local TOON library at /docs/js/toon.js
    • Created /docs/js/toon-wrapper.js to expose TOON globally
    • Removed broken CDN imports
    • All libraries now load correctly in browser
    • Console logs verify library availability

  • CSV Conversion

    • Improved CSV generator to handle all unique keys
    • Added automatic quoting for values containing commas
    • Better detection of nested objects (shows "N/A (nested objects)")
    • Handles null/undefined values gracefully

  • TOON Conversion

    • Enhanced manual TOON fallback implementation
    • Proper handling of non-uniform arrays
    • String quoting for values with commas/spaces
    • Clear error messages: "N/A (non-uniform)", "N/A (nested objects)"
Assets 2
Loading