lexer

TOML-Tools/lexer

A Toml Lexer implemented in JavaScript using the Chevrotain Parsing Toolkit.

This package could be used as part of a Toml Parser, a Toml syntax highlighter or any other use case where a Toml Token Vector would be useful.

Installation

yarn add @toml-tools/lexer
# or if you prefer npm
npm install @toml-tools/lexer

APIs

This package's APIs are exported as a TypeScript definition file.

Usage

const { tokenize } = require("@toml-tools/lexer");

const input = `# This is a TOML document.
title = "TOML Example"
`;
// Tokenize returns a Chevrotain IlexingResult
// - https://sap.github.io/chevrotain/documentation/4_2_0/interfaces/ilexingresult.html
const lexingResult = tokenize(input);
console.log(lexingResult.errors.length); // 0
console.log(lexingResult.tokens.length); // 6
const tokens = lexingResult.tokens;
// Each Token is a Chevrotain Itoken
//  - https://sap.github.io/chevrotain/documentation/4_2_0/interfaces/itoken.html
console.log(tokens[0].image); // "# This is a TOML document."
console.log(tokens[0].startLine); // 1
console.log(tokens[0].endOffset); // 26
// ...