Skip to content

Releases: hnpf/demarkify

Release list

v1.1.0

v1.1.0 Latest
Latest

Choose a tag to compare

View all tags
@hnpf hnpf released this 14 Sep 06:03
v1.1.0
acb2f85

demarkify v1.1.0: expanded watermark detection

a fast, non-dependent tool/library to detect, decode, and remove hidden ai watermarks, steganography, invisible characters, homoglyphs, and even byte anomalies from any text.

what's new in v1.1.0

  • hangul filler detection: U+3164, U+FFA0, U+115F, U+1160 commonly abused invisible hangul characters used for text steganography and prompt tracking
  • line & paragraph separator support: U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR), U+0085 (NEL/NEXT LINE) now detected and normalized
  • braille pattern blank (U+2800): invisible space-lookalike added to anomalous whitespace table
  • expanded deprecated format controls: U+206A–U+206F (deprecated invisible formatting chars), U+070F, U+08E2, U+2065, shorthand format controls (U+1BCA0–U+1BCA3), and musical invisible controls (U+1D173–U+1D17A)
  • procedural unicode injection detector: last-resort \p{Cf} (Format category) scanner catches any unlisted Unicode Format control characters that would otherwise slip through
  • private use area (PUA) detection: U+E000–U+F8FF, U+F0000–U+FFFD, U+100000–U+10FFFD, flags hidden glyphs or payloads embedded in PUA codepoints
  • extended tag block: isUnicodeTag now covers the full U+E0000–U+E007F range (was missing U+E0002–U+E001F)
  • extended variation selectors: isVariationSelector extended to U+180B–U+180F (was U+180D)
  • complete math alphanumeric coverage: fraktur, bold fraktur, double-struck, sans-serif bold/italic, monospace, and all digit variants now fully normalized
  • hangul filler binary stego decoder: decodeZeroWidthBinary now recognizes U+3164/U+FFA0 pairs as a steganographic bit encoding scheme
  • 35 tests passing! (up from 31!!)

installation

npm install -g demarkify
# or
pnpm add -g demarkify
Assets 8
Loading

v1.0.0: init release

v1.0.0: init release

Choose a tag to compare

View all tags
@hnpf hnpf released this 13 Sep 23:06
v1.0.0
aa2bbba

demarkify v1.0.0: init release

a fast, non-dependent tool/library to detect, decode, and remove hidden ai watermarks, steganography, invisible characters, homoglyphs, and even byte anomalies from any text.

features

  • zero runtime deps: pure native nodejs / TS ES module.
  • watermark and steganography detection:
    • zerowidth and invisible joiners (ZWSP, ZWNJ, ZWJ, BOM, Word joiner, soft hyphen, etc.)
    • unicode tag chars (U+E0020 - U+E007E) tracking payloads with automatic decoding
    • zero width and trailing whitespace binary steganography decoder
    • directional overrides and isolate markers (LTR / RTL)
    • mixed-script homoglyphs detection (cyrillic, greek, math, and fullwidth lookalikes)
    • tampered whitespace normalization (NBSP, thin spaces, em/en spaces)
  • scratch text econstruction: char-by-char full rebuild with canonical unicode normalization (NFC/NFKC).
  • flexible cli: direct string input, stdin piping, single/multi-file processing, and recursive directory traversal.
  • CI/CD ready: --check flag with strict exit codes.

installation

npm install -g demarkify
# or
pnpm add -g demarkify
Loading