Skip to content
No description or website provided.
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin Reset literal encoding after flushing a string. Apr 18, 2018
ext/lexer Bump parser to 41cb059ad73212e33bf427c5e3078ce242e6c025. Jan 16, 2019
lib
test Bump parser to 2.6.0.0 Jan 16, 2019
vendor Bump parser to 349a018b998119015ca3b21bbaa94172f2391938. Jan 16, 2019
.gitignore Properly backported literal states. Apr 12, 2018
.gitmodules Moved parser under vendor/parser Apr 16, 2018
.travis.yml Also bump versions in travis matrix. Jan 16, 2019
Gemfile Moved parser under vendor/parser Apr 16, 2018
Gemfile.lock
README.md Bump parser to 2.6.0.0 Jan 16, 2019
Rakefile Define CLexer as Parser::CLexer, added Parser::Ruby25WithCLexer class. Apr 19, 2018
appveyor.yml Install bundler on AppVeyor. Jul 30, 2018
c_lexer.gemspec Also bump versions in travis matrix. Jan 16, 2019

README.md

CLexer

Build Status Build status

A C port of the whitequark/parser's lexer.

Initially based on whitequark/parser#248 (thanks @alexdowad).

Can be use on Linux/Mac/Windows, requires MRI.

Installation

$ gem install c_lexer

Usage

c_lexer doesn't change any parser classes.

It provides 3 classes:

  • Parser::CLexer (C version of the lexer)
  • Parser::Ruby25WithCLexer (a subclass of parser/ruby25 that uses Parser::CLexer for lexing)
  • Parser::Ruby26WithCLexer (a subclass of parser/ruby26 that uses Parser::CLexer for lexing)

If you want CLexer to be a default lexer you can use the following patch:

require 'parser'
require 'c_lexer'

module Parser
  dedenter = Lexer::Dedenter

  remove_const(:Lexer)
  Lexer = CLexer
  Lexer::Dedenter = dedenter

  remove_const(:Ruby26)
  Ruby26 = Ruby26WithCLexer
end

Versioning

c_lexer follows versioning of the parser gem to avoid confusions like "which version of c_lexer should be used with the parser vX.Y.Z". For parser 2.5.1.0 you should use c_lexer 2.5.1.0.X.

c_lexer supports all versions of parser starting from 2.5.1.0

Development

  1. Make sure that you have ragel installed.
  2. Clone the repo and fetch submodules with git submodule update --init.
  3. The parser gem is located under vendor/parser.
  4. rake ruby_parser:generate generates lexer.rb and parsers for the parser gem.
  5. rake c_lexer:generate generates lexer.c.
  6. rake compile compiles lexer.c to lexer.so (or lexer.bundle depending on your platform).
  7. rake test runs parser tests using c_lexer as a default lexer.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/opal/c_lexer.

Before submitting a bug report, please, make sure that parser is not affected by the same issue. If it's a parser bug please report it to the parser repo and we will backport it afterwards.

You can’t perform that action at this time.