derive

Derive Data Files

• Unicode Standard
• Unicode Technical Standard #46: IDNA
  • unicode-logic.js/derive_idna_rules() — spec
  • idna.js/ens_idna_rules()
• Unicode Technical Standard #51: Emoji
• Unicode Standard Annex #15: Normalization Forms
• Unicode Standard Annex #24: Script Property
• Unicode Standard Annex #29: Text Segmentation
• Unicode Standard Annex #31: Identifier and Pattern Syntax
  • script-kinds.json — Excluded, Recommended, Limited Use
• Unicode Technical Standard #39: Security Mechanisms
  • unicode-logic.js/augmented_script_set() — spec
  • unicode-logic.js/get_resolved_script_set() — spec
• RFC-3492: Punycode
• RFC-5891: IDNA: Protocol
• RFC-5892: The Unicode Code Points and IDNA
• WHATWG URL: IDNA
• Unicode data files
  • Download Latest: node download.js
  • To download older version: node download.js 12.1.0
  • Already included: Unicode 11-15.1
• CLDR data files
  • Download Latest: node parse-cldr.js
  • To download older version: node parse-cldr.js 42
  • Already included: CLDR 42-44
  • ⚠️ Versioned separately from Unicode!

Instructions

1. edit unicode-version.js — specify which versions to use
2. edit Rules Files
3. node make.js — creates /output/ with data files

Rules

• chars-valid.js
• chars-ignored.js
• chars-mapped.js
• chars-disallow.js
• chars-fenced.js — characters that occur in the middle and can't touch
• chars-escape.js — characters that should be escaped
• emoji.js — various emoji configurations
• cm.js — combining mark sequence whitelist
• scripts.js — various script configurations
• confuse.js — confusables groups
• group-order.js — how groups should be sorted for matching efficiency (auto-generated)

Helpers

• node names.js 61..7A 200D — print Unicode names for hex codepoints
• node names.js script Latn — print Unicode names for Latin
• node names.js prop White_Space — print Unicode names with property White_Space
• node names.js find abc — find characters by name

Upgrade Notes

15.0.0 → 15.1.0

node unicode-diff.js

• CLDR
  • short-names.json Unchanged
  • regions.json New "CQ"
• UCD:
  • New Ideographic Description Characters
  • New CJK Ideograph Extension I Block
• UAX-31:
  • Unchanged Scripts
  • Unchanged Recommended, Excluded, Limited Use Scripts
  • New Zyyy Script Extensions
• UTS-39:
  • New Confusable Bidi Logic (doesn't apply since mixed-Bidi not allowed)
  • confusables.txt Unchanged
• UTS-46:
  • Change 1E9E (ẞ) LATIN CAPITAL LETTER SHARP S → 0xDF (ß) LATIN SMALL LETTER SHARP S
• Prior Validation: node ../test/validate.js prior-tests/1.9.4.json
  • Only fails on emoji that are now valid.