Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

494 lines (307 sloc) 15.642 kB

CodeRay Version History

This files lists all changes in the CodeRay library since the 0.8.4 release.

Changes in 1.0

Direct Streaming

CodeRay 1.0 introduces Direct Streaming as a faster and simpler alternative to Tokens. It means that all Scanners, Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with this version.

The benefit of this change is more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9), a simpler API, and less code.

Changes related to the new tokens handling include:

  • CHANGED: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a special case; Tokens just encodes the given tokens into an Array for later use.
  • CHANGED: The token actions (text_token, begin_group etc.) are now public methods of Encoder and Tokens.
  • REWRITE of all Scanners, Encoders, Filters, and Tokens.
  • RENAMED :open and :close actions to :begin_group and :end_group.
  • RENAMED open_token and close_token methods to begin_group and end_group.
  • CHANGED The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible with earlier versions.
  • REMOVED TokenStream and the Streamable API and all related features like NotStreamableError are now obsolete and have been removed.

General changes

  • IMPROVED documentation in general; additions, corrections and cleanups
  • FIXED some image links in the documentation
  • IMPROVED Ruby 1.9 support (. not in $LOAD_PATH)


  • REMOVED method stream?.
  • NEW methods encode_with, count, begin_group, end_group, begin_line, and end_line.

RENAMED: Tokens::AbbreviationForKind

Renamed from ClassOfKind; the term “token class” is no longer used in CodeRay. Instead, tokens have kinds.
See #122.

  • REMOVED token kinds :attribute_name_fat, :attribute_value_fat, :operator_fat, :tag_fat, :xml_text, :open, and :close.
  • ADDED token kinds :filename and :namespace.
  • CHANGED: Don’t raise error for unknown token kinds unless in $CODERAY_DEBUG mode.


  • NEW alias :remove_comments


  • NEW option tokens.
  • CHANGED: Now it simply delegates to the output.
  • REMOVED include_text_token? and include_block_token? methods.


The HTML encoder was cleaned up and simplified.

  • CHANGED the default style to :alpha.
  • NEW: HTML 5 and CSS 3 compatible, IE incompatible.
    See #215.
  • ADDED support for :line_number_anchors.
    See #208.
  • CHANGED: Use double click to toggle line numbers in table mode (as single
    click jumps to an anchor.)
  • REMOVED support for :line_numbers => :list.
  • FIXED splitting of lines for :line_numbers => :inline, so that the line
    numbers don’t get colored, too.


  • CHANGED: compile and finish methods are now protected.


  • RENAMED from Encoders::Term, added :term alias.
  • CLEANUP: Use #setup‘s super, don’t use :procedure token class.
  • CHANGED: #token’s second parameter is no longer optional.
  • REMOVED colors for obsolete token kinds.
  • FIXED handling of line tokens.

RENAMED: Encoders::TokenKindFilter

Renamed from TokenClassFilter.

  • NEW: Handles token groups.
    See #223.
  • RENAMED include_block_token? to include_group?.


  • CHANGED: Tokens actions are counted separately.


  • NEW method call for allowing code like CodeRay::Duo[:python => :yaml].(code) in Ruby 1.9.


  • REMOVED helper method String#to_unix.
  • REMOVED method streamable?.


  • NEW: Rudimentary support for the attr, counter, and counters functions.
    See #224.
  • NEW: Rudimentary support for CSS 3 colors.
  • CHANGED: Attribute selectors are highlighted as :attribute_name instead of :string.


  • NEW: Support for line tokens (begin_line and end_line represented by [ and ].)
  • FIXED: Don’t send :error and nil tokens for buggy input any more.
  • FIXED: Closes unclosed tokens at the end of scan_tokens.
  • IMPROVED: Highlight unknown tokens as :error.
  • CHANGED: Raises an error when trying to end an invalid token group.


  • FIXED: Closes open string groups.


  • NEW: Highlighting of code based on file names.
    See ticket #52.
This is a very original feature. It enables multi-language highlighting for diff files, which is especially helpful for CodeRay development itself. The updated version of the scanner test suite generated .debug.diff.html files using this. Note: This is still buggy for multi-line tokens.
  • NEW: Highlight the file name in the change headers as :filename.
  • CHANGED: Highlight unknown lines as :comment instead of :head.


  • FIXED: Closes open string groups.


  • IMPROVED: Added NaN and Infinity to list of predefined constants.


  • NEW: Package names are highlighted as :namespace.
    See #210.


  • IMPROVED: Just returns the string without scanning (faster).
This is much faster than scanning until /\z/ in Ruby 1.8.


  • CHANGED: Docstrings are highlighted as :comment.
    See #190.

NEW: Scanners::Raydebug

Copied from Scanners::Debug, highlights the token dump instead of importing it. It also reacts to the .raydebug file name suffix now.


  • ADDED more predefined keywords (see
  • IMPROVED support for singleton method definitions.
    See #147.
  • FIXED: Don’t highlight methods with a capital letter as constants
    (eg. GL.PushMatrix).
  • NEW: Highlight buggy floats (like .5) as :error.
  • CLEANUP of documentation, names of constants and variables, state handling.
  • NEW: Complicated rule for recognition of foo= style method names.
  • NEW: Handles :keep_state option (a bit; experimental).
Actually, Ruby checks if there is [~>=], but not => following the name.


  • CHANGED: Does use :operator instead of :operator_fat now.


  • IMPROVED: Extended list of keywords and functions (thanks to Joshua Galvez).
See #221.
  • FIXED: Closes open string groups.

NEW: Styles::Alpha

A style that uses transparent HSLA colors as defined in CSS 3.
See #199.


  • REMOVED FileType#shebang is a protected method now.
  • NEW: Recognizes .gemspec, .rjs, .rpdf extensions and Capfile as Ruby.
Thanks to the authors of the TextMate Ruby bundle!


  • IMPROVED: register_for sets the plugin_id; it can now be a Symbol.

Internal API changes

  • FIXED Encoders::HTML#token’s second parameter is no longer optional.

Changes in 0.9.3

  • FIXED: Documentation of Tokens.

coderay executable

  • NEW: automatic TTY detection (uses Term encoder)
  • NEW: optional 3rd parameter for the filename
  • FIXED: Converted to UNIX format.
  • FIXED: Warn about generated files.
  • FIXED: Ensure line break after the output (especially for LoC counter).


  • FIXED: Don’t keep state of XML scanner between calls for E4X literals.

Scanners::Java, Scanners::JSON

  • FIXED: Close unfinished strings with the correct token kind.

Changes in 0.9.2

  • NEW Basic tests and a Rakefile are now included in the Gem. [Flameeyes]
    A doc task is also included.
  • FIXED Use $CODERAY_DEBUG for debugging instead of $DEBUG. [Trans]
  • REMOVED Term::Ansicolor was bundled under lib/, but not used. [Flameeyes]
  • WORKAROUND for Ruby bug


  • FIXED strings are closed correctly
  • FIXED several token kinds had no associated color
  • NEW alias terminal
NOTE: This encoder will be renamed to Encoders::Terminal in the next release.


  • FIXED Don’t close tokens that are not open. Send :error token instead.


  • FIXED token kind of closing brackets is :operator instead of nil


  • FIXED allow \ operator (namespace separator)


  • FIXED doesn’t send debug tokens when $DEBUG is true [Trans]

Changes in 0.9.1

Token classes

  • NEW token classes :complex, :decorator, :imaginary
    (all for Python)
  • REMOVED token class :procedure
    – use :function or :method instead.


  • NEW method #scanner
Stores the scanner.
  • REMOVED methods .write_token, .read_token, .escape, .unescape
They were only used by the Tokens encoder, which was removed also.


  • REMOVED Don’t require the stringio library.
  • NEW public methods #open_token, #close_token, #begin_line, #end_line
    These methods are called automatically, like #text_token.
  • NEW proteced method #append_encoded_token_to_output


  • REMOVED – use Tokens#dump and Tokens.load.


  • NEW
    A Filter encoder has another Tokens instance as output.


  • NEW
It takes 2 options, :exclude and :include, that specify which token classes to include or exclude for the output. They can be a single token class, an Array of classes, or the value :all.


  • NEW
Removes tokens of the :comment class.


  • NEW
Counts the lines of code according to the KINDS_NOT_LOC token class list defined by the scanner. It uses the new TokenClassFilter. Alias: :loc, as in tokens.loc.


  • NEW
Outputs tokens in a simple JSON format.


  • NEW (beta, by Rob Aldred)
Outputs code highlighted for a color terminal.


  • NEW option :title (default value is CodeRay output)
Setting this changes the title of the HTML page.
  • NEW option :highlight_lines (default: nil)
Highlights the given set of line numbers.

- REMOVED option :level

It didn’t do anything. CodeRay always outputs XHTML.


  • Uses Encoder interface with super and #text_token.


It didn’t work at all.


  • NEW Mapped :h to :c, :cplusplus and :'c++' to :cpp,
    :ecma, :ecmascript, :ecma_script to :java_script,
    :pascal to :delphi, and :plain to :plaintext.


  • NEW constant KINDS_NOT_LOC
A list of all token classes not considered in LOC count. Added appropriate values for scanners.
  • NEW method #lang returns the scanner’s lang, which is its plugin_id.
  • FIXED automatic, safe UTF-8 detection [Ruby 1.9]
  • FIXED column takes care of multibyte encodings [Ruby 1.9]
  • FIXED is dumpable (Tokens store their scanner in an @scanner variable)


  • NEW (C++)


  • NEW (beta)


  • NEW


  • NEW (based on Stefan Walk’s work)


  • NEW (based on code by Josh Goebel)


  • IMPROVED added a list of :directive tokens that were :reserved before
  • IMPROVED detection of labels
  • IMPROVED allow 1L and 1LL style literals


  • IMPROVED element selectors are highlighted as :type instead of :keyword


  • IMPROVED Don’t cache tokens in CaseIgnoringWordList.


  • IMPROVED assert is highlighted as a :keyword now
  • IMPROVED const and goto are highlighted as :reserved
  • IMPROVED false, true, and null are highlighted as :pre_constant
  • IMPROVED threadsafe is no longer a :directive
  • IMPROVED String is highlighted as a :pre_type
  • IMPROVED built-in classes ending with Error or Exception are
    highlighted as a :exception instead of :pre_type


  • NEW a list of PREDEFINED_CONSTANTS to be highlighted as :pre_constant
  • NEW XML literals are recognized and highlighted
  • NEW function name highlighting
  • IMPROVED .1 is highlighted a number
  • FIXED strings close with the correct kind when terminated unexpectedly


  • IMPROVED constants (true, false, nil) are highlighted as :value


  • IMPROVED Patterns::KEYWORDS_EXPECTING_VALUE for more accurate
    value_expected detection
  • IMPROVED handling of \ as a string delimiter
  • IMPROVED handling of unicode strings; automatic switching to unicode
  • IMPROVED highlighting of self.method definitions
  • REMOVED Patterns::FANCY_START_SAVE (obsolete)
  • FIXED encoding issues [Ruby 1.9]
  • FIXED a problem in early Ruby 1.8.6 patch versions with Regexp.escape


  • IMPROVED indentation detection


  • changed a few colors (exceptions, inline strings, predefined types)


  • NEW method #title
Set and get the plugin’s title. Titles can be arbitrary strings.
  • NEW method #helper loads helpers from different plugins
Use this syntax: helper 'other_plugin/helper_name'


  • NEW FileType[] takes Pathname instances
  • NEW regonizes .cc, .cpp, .cp, .cxx, .c++, .C, .hh, .hpp, .h++, .cu extensions (C++)
Thanks to Sander Cox and the TextMate C bundle.
  • NEW regonizes .pas, .dpr extensions (Delphi)
  • NEW regonizes .gvy, .groovy extensions (Groovy)
  • NEW regonizes .php, .php3, .php4, .php5 extensions (PHP)
  • NEW regonizes .py, .py3, .pyw extensions (Python)
  • NEW regonizes .rxml extension (Ruby)
  • NEW regonizes .sql extension (SQL)
  • File types list was sorted alphabetically.


The default value is no longer ignored.


  • FIXED for RedCloth versions 4.2.0+ (#119)


  • warnings about character classes [Ruby 1.9]
  • encoding issues [Ruby 1.9]
  • documentation, code
Jump to Line
Something went wrong with that request. Please try again.