README

.travis.yml

@@ -3,6 +3,8 @@ rvm:
   - 2.4.0
   - 2.2.2
   - 2.1.0
+before_install: gem install bundler
+install: bundle install
 script:
   - bundle exec rubocop --format clang
   - bundle exec rspec spec

Gemfile

@@ -6,6 +6,4 @@ source "https://rubygems.org"
 # Specify your gem's dependencies in yoga.gemspec
 gemspec
 
-gem "mixture"
-gem "pry"
-gem "pry-stack_explorer"
+gem "coveralls", require: false

README.md

@@ -1,36 +1,153 @@
 # Yoga
+[![Build Status][build-status]][build-status-link] [![Coverage Status][coverage-status]][coverage-status-link]
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/yoga`. To experiment with that code, run `bin/console` for an interactive prompt.
+A helper for your Ruby parsers.  This adds helper methods to make parsing
+(and scanning!) easier and more structured.  If you're looking for an LALR
+parser generator, that isn't this.  This is designed to help you construct
+Recursive Descent parsers - which are solely LL(k).  If you want an LALR parser
+generator, see [_Antelope_](https://github.com/medcat/antelope) or [Bison](https://www.gnu.org/software/bison/).
 
-TODO: Delete this and the text above, and describe your gem
+Yoga requires [Mixture](https://github.com/medcat/mixture) for parser node
+attributes.  However, the use of the parser nodes included with Yoga are
+completely optional.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'yoga'
+gem "yoga"
 ```
 
 And then execute:
 
     $ bundle
 
-Or install it yourself as:
+## Usage
 
-    $ gem install yoga
+To begin your parser, you will first have to create a scanner.  A scanner
+takes the source text and generates "tokens."  These tokens are abstract
+representations of the source text of the document.  For example, for the
+text `class A do`, you could have the tokens `:class`, `:CNAME`, and `:do`.
+The actual names of the tokens are completely up to you.  These token names
+are later used in the parser to set up expectations - for example, for the
+definition of a class, you could expect a `:class`, `:CNAME`, and a `:do`
+token.
 
-## Usage
+Essentially, the scanner breaks up the text into usable, bite-sized pieces
+for the parser to chomp on.  Here's what scanner may look like:
+
+```ruby
+module MyLanguage
+  class Scanner
+    # All of the behavior from Yoga for scanners.  This provides the
+    # `match/2` method, the `call/0` method, the `match_line/1` method,
+    # the `location/1` method, and the `emit/2` method.  The major ones that
+    # are used are the `match/2`, the `call/0`, and the `match_line/1`
+    # methods.
+    include Yoga::Scanner
+
+    # This must be implemented.  This is called for the next token.  This
+    # should only return a Token, or true.
+    def scan
+      # Match with a string value escapes the string, then turns it into a
+      # regular expression.
+      match("[") || match("]") ||
+      # Match with a symbol escapes the symbol, and turns it into a regular
+      # expression, suffixing it with `symbol_negative_assertion`.  This is
+      # to prevent issues with identifiers and keywords.
+      match(:class) || match(:func) ||
+      # With a regular expression, it's matched exactly.  However, a token
+      # name is highly recommended.
+      match(/[a-z][a-zA-Z0-9_]*[!?=]?/, :IDENT)
+    end
+  end
+end
+```
+
+And that's it!  You now have a fully functioning scanner.  In order to use it,
+all you have to do is this:
+
+```ruby
+source = "class alpha [func a []]"
+MyLanguage::Scanner.new(source).call # => #<Enumerable ...>
+```
+
+Note that `Scanner#call` returns an enumerable.  `#call` is aliased as `#each`.
+What this means is that tokens aren't generated until they're requested by the
+parser - each token is generated from the source incrementally.  If you want
+to retrieve all of the tokens immediately, you have to first convert it into
+a string, or perform some other operation on the enumerable (since it isn't
+lazy):
+
+```ruby
+MyLanguage::Scanner.new(source).call.to_a # => [...]
+```
+
+The scanner also automatically adds location information to all of the tokens.
+This is handled automatically by `match/2` and `emit/2` - the only issue being
+that all regular expressions **must not** include a newline.  Newlines should
+be matched with `match_line/1`; if lines must be emitted as a token, you can
+pass the kind of token to emit to `match_line/1` using the `kind:` keyword.
+
+You may notice that all of the tokens have `<anon>` set as the location's file.
+This is the default location, which is provided to the initializer:
+
+```ruby
+MyLanguage::Scanner.new(source, "foo").call.first.location.to_s # => "foo:1.1-6"
+```
+
+Parsers are a little bit more complicated.  Before we can pull up the parser,
+let's define a grammar and some node classes.
+
+```
+; This is the grammar.
+<root> = *<statement>
+<statement> = <expression> ';'
+<expression> = <expression> <op> <expression>
+<expression> /= <int> ; here, <int> is defined by the scanner.
+<op> = '+' / '-' / '*' / '/' / '^' / '%' / '='
+```
+
+```ruby
+module MyLanguage
+  class Parser
+    class Root < Yoga::Node
+      # An attribute on the node.  This is required for Yoga nodes since the
+      # update syntax requires them.  The type for the attribute is optional.
+      attribute :statements, type: [Yoga::Node]
+    end
+
+    class Expression < Yoga::Node
+    end
+
+    class Operation < Expression
+      attribute :operator, type: ::Symbol
+      attribute :left, type: Expression
+      attribute :right, type: Expression
+    end
+
+    class Literal < Expression
+      attribute :value, type: ::Integer
+    end
+  end
+end
+```
 
-TODO: Write usage instructions here
+With those out of the way, let's take a look at the
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/yoga. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at
+<https://github.com/medcat/yoga>. This project is intended to be a safe,
+welcoming space for collaboration, and contributors are expected to adhere to
+the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
+[build-status]: https://travis-ci.org/medcat/yoga.svg?branch=master
+[coverage-status]: https://coveralls.io/repos/github/medcat/yoga/badge.svg?branch=master
+[build-status-link]: https://travis-ci.org/medcat/yoga
+[coverage-status-link]: https://coveralls.io/github/medcat/yoga?branch=master

lib/yoga/errors.rb

@@ -33,6 +33,22 @@ class LocationError < Error
     attr_reader :location
   end
 
+  # An error that occurred with scanning.
+  #
+  # @api private
+  class ScanError < LocationError; end
+
+
+  # An unexpected character was encountered while scanning.
+  #
+  # @api private
+  class UnexpectedCharacterError < LocationError
+    # (see Error#generate_message)
+    private def generate_message
+      "An unexpected character was encountered at #{@location}"
+    end
+  end
+
   # An error that occurred with parsing.
   #
   # @api private

lib/yoga/scanner.rb

@@ -6,12 +6,19 @@ module Yoga
   # It is built to lazily scan whenever it is required, instead
   # of all at once.  This integrates nicely with the parser.
   module Scanner
+    # The file of the scanner.  This can be overwritten to provide a descriptor
+    # for the file.
+    #
+    # @return [::String]
+    attr_reader :file
+
     # Initializes the scanner with the given source.  Once the
     # source is set, it shouldn't be changed.
     #
     # @param source [::String] The source.
-    def initialize(source)
+    def initialize(source, file)
       @source = source
+      @file = file
       @line = 1
       @last_line_at = 0
     end
@@ -32,10 +39,10 @@ def call
 
       until @scanner.eos?
         value = scan
-        yield value if value.is_a?(Token)
+        yield value unless value == true || !value
       end
 
-      yield Token.eof(location)
+      yield eof_token
       self
     end
 
@@ -53,7 +60,7 @@ def scan
       fail NotImplementedError, "Please implement #{self.class}#scan"
     end
 
-  private
+  protected
 
     # Returns a location at the given location.  If a size is given, it reduces
     # the column number by the size and returns the size from that.
@@ -115,12 +122,12 @@ def match(matcher, kind = :"#{matcher}")
     # such as line counting and caching, to be performed.
     #
     # @return [Boolean] If the line was matched.
-    def match_line(kind = false)
-      match(LINE_MATCHER, kind).tap do |t|
-        break unless t
-        @line += 1
-        @last_line_at = @scanner.charpos
-      end
+    def match_line(kind: false, required: false)
+      result = @scanner.scan(LINE_MATCHER)
+      (required ? return : fail UnexpectedCharacterError) unless result
+      @line += 1
+      @last_line_at = @scanner.charpos
+      (kind && emit(kind)) || true
     end
 
     # Returns the number of lines that have been covered so far in the scanner.
@@ -145,12 +152,11 @@ def symbol_negative_assertion
       "(?![a-zA-Z])"
     end
 
-    # The file of the scanner.  This can be overwritten to provide a descriptor
-    # for the file.
+    # Returns a token that denotes that the scanner is done scanning.
     #
-    # @return [::String]
-    def file
-      @file ||= "<anon>"
+    # @return [Yoga::Token]
+    def eof_token
+      emit(:EOF, "")
     end
   end
 end

spec/spec_helper.rb

@@ -2,6 +2,12 @@
 # frozen_string_literal: true
 
 require "simplecov"
+require "coveralls"
+
+SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
+  SimpleCov::Formatter::HTMLFormatter,
+  Coveralls::SimpleCov::Formatter
+]
 SimpleCov.start { add_filter "/spec/" }
 require "yoga"