Skip to content

v3.0.0: ISO code changes and faster detection

Latest

Choose a tag to compare

@peterc peterc released this 20 Oct 12:12

Breaking change: nine ISO output codes

Update comparisons against Result#iso, instance/class language_iso, and CLI output before upgrading. These outputs now use CLDR macrolanguage aliases for nine languages:

Language Previous code 3.0 code
Swahili swh sw
Central Aymara ayr ay
Central Kanuri knc kr
Cusco Quechua quz qu
Halh Mongolian khk mn
Koongo kng kg
Logudorese Sardinian src sc
Northern Uzbek uzn uz
Tosk Albanian als sq

Northern Uzbek and Uzbek now both return uz; use result.language to distinguish them. Language-name symbols, scores, rankings, options, and return types are unchanged from 2.1. The individual codes quy, qug, koi, and min are preserved.

Faster detection, fewer allocations

  • Shared trigram indexes speed up repeated detection over the full language set. The indexing change made the 60-byte English benchmark 15.28× faster (2.036 → 0.133 ms per warmed call).
  • Rolling trigram windows, combined script/letter counting, and an ASCII fast path reduce scanning and allocations. These changes further reduced that short-text benchmark to 0.089 ms and made the repeated 1 MB English benchmark 2.72× faster (490 → 180 ms).
  • Configured detectors cache candidate selections; winner-only calls avoid sorting the full ranking.

Measurements are seven-run medians on Ruby 4.0.2, Apple M5. Indexes are built lazily and cost additional memory; the first unrestricted call and restricted selections keep the original scorer. The 1 MB repeated-ASCII workload fell from 5.75 million allocated objects to 197; inputs with more distinct trigrams or Unicode allocate more, and string memory still grows with input size.

See the indexing benchmarks and scanning/allocation benchmarks for reproducible commands, raw results, and tradeoffs.

Compatibility and availability

Requires Ruby 3.0 or newer, with no runtime dependencies. All existing test assertions are retained; added equivalence tests check exact scores, rankings, preprocessing, and tie handling. The suite has 67 tests and 5,611 assertions and passes locally on Ruby 3.0.7, 3.3.11, 4.0.2, JRuby 9.4.14.0, and TruffleRuby 34.0.1.

GitHub Actions passed on Ruby 3.0, Ruby 4.0, JRuby, and TruffleRuby.

Available on RubyGems: gem install whatlanguage -v 3.0.0.

Full changelog · Changes since v2.1.0