Skip to content

Lexer Library

Choose a tag to compare

View all tags
@stephen-hqxu stephen-hqxu released this 23 Jan 21:42
v0.16.0
a32636c
This commit was signed with the committer’s verified signature. The key has expired and has been revoked.
stephen-hqxu Stephen Haoqian Xu
GPG key ID: F7A92E1474FAB7BF
Expired Revoked
Verified
Learn about vigilant mode.

Release 0.16.0

STPLexer

We scrap out the lexer from STPTextureDefinitionLanguage and make it a more general-purpose lexer that allows defining custom tokens and matching functions. This makes writing parsers in the project in the future easier. The lexer uses a state-based approach that allows grouping token matching functions in a number of states.

The following new class are added to SuperAlgorithm+Host. A test code STPTestLexer is also added to the test program.

  • STPBasicStringAdaptor, is intended as an replacement to STPINIBasicString.
  • STPLexer

STPTextureDefinitionLanguage

  • Rewrite TDL parser to make use of STPLexer.
  • TDL parser is now stateless.
  • Change the TDL specification a little bit. It is no longer required to write semicolon at the end of the texture directive, and texture variable can contain underscore.

STPINIParser

  • Remove STPINIBasicString, it should be replaced by STPBasicStringAdaptor.
  • Remove implementation for writing to an INI string because it is so flawed.
  • Merge STPINIReader and STPINIWriter into STPINIParser.
  • Merge STPINIStorage and STPINISection into STPINIData.
  • INI parser now preserves order of each item declared in the parsed input.

STPRegularLanguage

The regex library in C++ standard library is terrible, it is a good idea to implement our own.
Basically it is a regex parser, but it uses template as matching terms rather than regex string.
DFA/NFA are constructed at compile-time with no dynamic memory allocation in runtime.

The regular language parser STPRegularLanguage, resides in the algorithm+ package. Test is implemented as STPTestRegLang.cpp.

General fixes and improvement

  • Add file filters for SuperAlgorithm+ and SuperDemo+ built targets.
  • Do some clean up to STPTextureDatabase.
  • Reduce the printed horizontal separator width in the exception error message.
  • Simplify error message for STPParserError.
  • Make sure all use of functions from header cctype have the input argument cast to unsigned char to avoid UB.
  • Make global constexpr variable to use external linkage rather than internal (change from static to inline) for:
    • STPAlgebraCommon
  • Simplify naming for nested namespace for:
    • STPHashCombine
    • STPSmartDeviceMemory
    • STPSmartDeviceObject
  • Remove an unused variable in STPWorldPipeline.
Assets 2
Loading