Hasura is a basic lexer implemented in Go. Designed for tokenizing source code.
Hasura provides functionality to break down source code into tokens, identifying syntax elements such as parentheses, integers, and identifiers.
go get github.com/ezrantn/hasura
package main
import (
"fmt"
"github.com/ezrantn/hasura"
)
func main() {
// Create a new LexingContent
lc := hasura.NewLexingContent("example_source_file")
// Get the tokens from the source
tokens := lc.Lex()
// Print the tokens
for _, token := range tokens {
fmt.Println(token.Value)
}
}
Hasura recognizes the following types of tokens:
- Syntax Tokens: Such as '(' and ')'
- Integer Tokens: Numeric values, e.g '123'
- Identifier Tokens: Alphanumeric strings, e.g 'foo', 'define''
Hasura provides two main types that are relevant to tokenization:
-
LexingContent
:- This represents the content to be lexed (tokenized).
- It contains the source content and filename.
- The main method associated with this type is lex(), which tokenizes the source content and returns a slice of tokens.
-
Token
:- This represents a token generated by the lexer.
- It has several fields:
Value
: The actual string value of the token.Kind
: The type of token (syntax, integer, or identifier).Location
: The position of the token in the source content.LC
: The LexingContent instance associated with the token.
- The main method associated with this type is debug(), which provides debugging information about the token's location in the source content.
In summary, the LexingContent type is responsible for managing the source content and initiating the tokenization process, while the Token type represents individual tokens generated during tokenization, providing information about their type, value, and location in the source content.