Changes.textile

CodeRay Version History

This files lists all changes in the CodeRay library since the 0.9.8 release.

Changes in 1.1.1

• SQL scanner: Allow $ signs in SQL identifiers [#164, thanks to jasir and Ben Basson]
• SQL scanner: Fix open strings [#163, thanks to Adam]
• Ruby scanner: Accept number literal suffixes r and i (Ruby 2.1)
• Ruby scanner: Accept quoted hash keys like { "a": boss } (Ruby 2.2)
• Ruby scanner: Accept save navigation operator &. (Ruby 2.3)
• Ruby scanner: Accept squiggly heredoc <<~ (Ruby 2.3)
• Diff scanner: Prevent running out of regexp stack.
• HTML encoder: You can keep tabs intact now by setting tab_width: false.

Changes in 1.1

New scanners:

• Go [#28, thanks to Eric Guo and Nathan Youngman]
• Lua [#21, #22, thanks to Quintus]
• Sass [#93]
• Taskpaper [#39, thanks to shimomura]

More new stuff:

• .xaml file type [#121, thanks to Kozman Bálint]
• recognize Guardfile, Vagrantfile, and Appraisals as Ruby files [#121, thanks to Kozman Bálint]
• new token kind :id for CSS/Sass [#27]
• new token kind :done for Taskpaper [#39]
• new token kind :map for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
• new token kind :unknown for Debug scanner
• new DebugLint encoder that checks for empty tokens and correct nesting

Improvements:

• CSS scanner uses :id and :tag now [#27]
• Diff scanner: Highlight inline changes in multi-line changes [#99]
• JavaScript scanner: Highlight multi-line comments in diff correctly
• JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
• HTML scanner displays style tags and attributes now [#145]
• Ruby scanner: Accept %i(…) and %I(…) symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
• Ruby scanner: Accept keywords as Ruby hash keys [#126]
• performance improvements to several scanners and encoders, especially Terminal and HTML
• added :keep_state functionality to more scanners so they work nicely with diff now [#116]
• refactoring and cleanup to achieve better “Code Climate” ratings (but I don’t really care)
• updated and cleaned up the documentation,
• documented list of TokenKinds
• Alpha style: tweaked colors for .binary, .local-variable, and .predefined-type
• rake generate supports Git now instead of Subversion

Removed:

• Tokens#dump, Tokens.load, Tokens::Undumping, and zlib dependency
• double-click toggle handler from HTML table output
• rake_helpers, sample directories and several other ancient garbage

Fixes:

• fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
• fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
• fixed :docstring token type style
• fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
• fixed HTML encoder when output is a StringIO (eg. when using -HTML as a command line parameter)
• TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
• display line numbers in HTML :table mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
• override Bootstrap’s pre { word-break: break-all } styling for line numbers [#102, thanks to lightswitch05]
• HTML encoder will not warn about unclosed token groups at the end of the stream
• fixed problem with coderay/version.rb being loaded twice

Internals:

• The Debug scanner maps unknown token kinds to :unknown (to avoid creating Symbols based on possibly unsafe input).
• The Raydebug scanner highlights unknown token kinds as :plain.
• The Debug encoder refactored; use DebugLint if you want strict checking now..
• The Debug encoder will not warn about errors in the token stream.
• Plugin does not warn about fallback when default is defined.
• PluginHost now works with Strings instead of Symbols internally (to avoid using #to_sym).

Changes in 1.0.9

• Fix Ruby scanner: Ruby 1.9 hash syntax { key: value } is highlighted correctly. [GH #106, thanks to Seth Vargo]
• Fix HTML scanner: Accept DTDs. [GH #83]
• Fix PHP scanner: Accept Unicode. [GH #40, thanks to Lance Li]

Changes in 1.0.8

• add :string/:char, remove :regexp/:function color from Terminal encoder [GH #29, thanks to Kyrylo Silin]
• allow - in line number anchor prefix for HTML encoder [GH #32, thanks to shurizzle]
• Fix HTML scanner: Don’t crash if HTML in a diff contains a JavaScript tag.

Changes in 1.0.7

• Changed license from LGPL to MIT. [GH-25, thanks to jessehu]
• Fix issue with plugin files not being loaded. [GH-20, thanks to Will Read]
• Fix HTML scanner bug: Don’t choke on boolean attributes. [GH-26, thanks to jugglinmike]

Changes in 1.0.6

• New option :break_lines for the HTML encoder (splits tokens at line breaks). [GH-15, thanks to Etienne Massip]
• Improved speed of :line_numbers => :inline option for the HTML encoder.
• Fixed wrong HTML file type. (was :page) [GH-16, thanks to Doug Hammond]
• The CSS Scanner now highlights tokens like url(...) as :function instead of :string. [GH-13, thanks to Joel Holdbrooks]

Changes in 1.0.5

Fixes:

• autoload calls do not depend on coderay/lib being in the load path (GitHub issue #6; thanks to tvon, banister, envygeeks, and ConradIrwin)
• avoid dark blue as terminal color (GitHub issue #9; thanks to shevegen)

Changes in 1.0.4

Fixes in the CSS scanner:

• understands the unit “s” (seconds)
• ignores unexpected curly braces
• code inside of diffs is highlighted correctly

Changes in 1.0.3

New:

• .tmproj files are recognized as XML.

Fixes:

• Removed files are highlighted inside diffs generated by git.

Changes in 1.0.2

Fixes:

• .erb files are recognized as ERB.

Changes in 1.0.1

New:

• YAML scanner allows “-” and “/” in key names

Changes:

• HTML page output has no white border anymore (alpha style)

Fixes:

• fixed warning in the output of “coderay stylesheet”
• fixed additional scrollbar in code when last line contains an eyecatcher
• minor fixes in the tests (issue github-#4)

Changes in 1.0

CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.

The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.

General changes

• NEW: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
• NEW: Extended support and usage of HTML5 and CSS 3 features.
• NEW: Direct Streaming
• NEW scanners: Clojure and HAML
• CHANGED: Token classes (used as CSS classes) are readable names; breaks you stylesheet!
• IMPROVED documentation
• IMPROVED speed: faster startup (using autoload), scanning, and encoding
• IMPROVED Ruby 1.9 encodings support
• IMPROVED Tests: There are more of them now!

Direct Streaming

CodeRay 1.0 introduces Direct Streaming as a faster and simpler alternative to Tokens. It means that all Scanners,
Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
this version.

The main benefits of this change are:

• more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
• the ability to stream output into a pipe on the command line
• a simpler API
• less code

Changes related to the new tokens handling include:

• CHANGED: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
  Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
  special case; Tokens just encodes the given tokens into an Array for later use.
• CHANGED: The token actions (text_token, begin_group etc.) are now public methods of Encoder and Tokens.
• REWRITE of all Scanners, Encoders, Filters, and Tokens.
• RENAMED :open and :close actions to :begin_group and :end_group.
• RENAMED open_token and close_token methods to begin_group and end_group.
• NEW method #tokens allows to add several tokens to the stream. Tokens and Encoders::Encoder define this
  method.
• CHANGED The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible
  with earlier versions.
• REMOVED TokenStream and the Streamable API and all related features like NotStreamableError are now obsolete
  and have been removed.

Command Line

The coderay executable was rewritten and has a few new features:

• NEW Ability to stream into a pipe; try coderay file | more -r
• NEW help
• IMPROVED: more consistent parameter handling
• REMOVED coderay_stylesheet executable; use coderay stylesheet [name].

Tokens

• NEW methods count, begin_group, end_group, begin_line, and end_line.
• REMOVED methods #stream?, #each_text_token.
• REMOVED methods #optimize, #fix, #split_into_lines along with their bang! variants.
• REMOVED #text and #text_size methods. Use the Text encoder instead.
• REMOVED special implementation of #each taking a filter parameter. Use TokenKindFilter instead.

RENAMED: TokenKinds

Renamed from Tokens::ClassOfKind (was also Tokens::AbbreviationForKind for a while).
The term “token class” is no longer used in CodeRay. Instead, tokens have kinds.
See #122.

• CHANGED all token CSS classes to readable names.
• ADDED token kinds :filename, :namespace, and :eyecatcher.
• RENAMED :pre_constant and :pre_type to :predefined_constant and predefined_type.
• RENAMED :oct and :bin to :octal and binary.
• REMOVED token kinds :attribute_name_fat, :attribute_value_fat, :operator_fat, interpreted,
  :tag_fat, tag_special, :xml_text, :nesting_delimiter, :open, and :close.
• CHANGED: Don’t raise error for unknown token kinds unless in $CODERAY_DEBUG mode.
• CHANGED the value for a token kind that is not highlighted from
  :NO_HIGHLIGHT to false.

Duo

• NEW method call for allowing code like CodeRay::Duo[:python => :yaml].(code) in Ruby 1.9.

Encoders::CommentFilter

• NEW alias :remove_comments

Encoders::Filter

• NEW option tokens.
• CHANGED: Now it simply delegates to the output.
• REMOVED include_text_token? and include_block_token? methods.

Encoders::HTML

The HTML encoder was cleaned up and simplified.

• NEW: HTML5 and CSS 3 compatible.
  See #215.
• ADDED support for :line_number_anchors.
  See #208.
• CHANGED the default style to :alpha.
• CHANGED: Use double click to toggle line numbers in table mode (as single
  click jumps to an anchor.)
• REMOVED support for :line_numbers => :list.
• FIXED splitting of lines for :line_numbers => :inline, so that the line
  numbers don’t get colored, too.
• RENAMED Output#numerize to #number, which is an actual English word.

Encoders::LinesOfCode

• CHANGED: compile and finish methods are now protected.

Renamed: Encoders::Terminal (was Encoders::Term)

• RENAMED from Encoders::Term, added :term alias.
• CLEANUP: Use #setup‘s super, don’t use :procedure token class.
• CHANGED: #token’s second parameter is no longer optional.
• REMOVED colors for obsolete token kinds.
• FIXED handling of line tokens.

Encoders::Text

• FIXED default behavior of stripping the trailing newline.

RENAMED: Encoders::TokenKindFilter (was Encoders::TokenClassFilter)

• NEW: Handles token groups.
  See #223.
• RENAMED include_block_token? to include_group?.

Encoders::Statistic

• CHANGED: Tokens actions are counted separately.

Scanners::Scanner

• NEW methods #file_extension and #encoding.
• NEW: The #tokenize method also takes an Array of Strings as source. The
  code is highlighted as one and split into parts of the input lengths
  after that using Tokens#split_into_parts.
• NEW method #binary_string
• REMOVED helper method String#to_unix.
• REMOVED method #streamable?.
• REMOVED #marshal_load and #marshal_dump.
• RENAMED class method normify to normalize; it also deals with encoding now.
• CHANGED: #column starts counting with 1 instead of 0

NEW: Scanners::Clojure

Thanks to Licenser, CodeRay now supports the Clojure language.

Scanners::CSS

• NEW: Rudimentary support for the attr, counter, and counters functions.
  See #224.
• NEW: Rudimentary support for CSS 3 colors.
• CHANGED: Attribute selectors are highlighted as :attribute_name instead of :string.
• CHANGED: Comments are scanned as one token instead of three.

Scanners::Debug

• NEW: Support for line tokens (begin_line and end_line represented by [ and ].)
• FIXED: Don’t send :error and nil tokens for buggy input any more.
• FIXED: Closes unclosed tokens at the end of scan_tokens.
• IMPROVED: Highlight unknown tokens as :error.
• CHANGED: Raises an error when trying to end an invalid token group.

Scanners::Delphi

• FIXED: Closes open string groups.

Scanners::Diff

• NEW: Highlighting of code based on file names.
  See ticket #52.

Use the :highlight_code option to turn this feature off. It’s enabled by default. This is a very original feature. It enables multi-language highlighting for diff files, which is especially helpful for CodeRay development itself. The updated version of the scanner test suite generated .debug.diff.html files using this. Note: This is still experimental. Tokens spanning more than one line may get highlighted incorrectly. CodeRay tries to keep scanner states between the lines and changes, but the quality of the results depend on the scanner.

• NEW: Inline change highlighting, as suggested by Eric Thomas.
  See ticket #227 for details.

Use the :inline_diff option to turn this feature off. It’s enabled by default. For single-line changes (that is, a single deleted line followed by a single inserted line), this feature surrounds the changed parts with an :eyecatcher group which appears in a more saturated background color. The implementation is quite complex, and highly experimental. The problem with multi-layer tokenizing is that the tokens have to be split into parts. If the inline change starts, say, in the middle of a string, then additional :end_group and :begin_group tokens must be inserted to keep the group nesting intact. The extended Scanner#tokenize method and the new Tokens#split_into_parts method take care of this.

• NEW: Highlight the file name in the change headers as :filename.
• CHANGED: Highlight unknown lines as :comment instead of :head.
• IMPROVED: Background colors for Diff output have been optimized.

RENAMED: Scanners::ERB (was Scanners::RHTML)

NEW: Scanners::HAML

It uses the new :state options of the HTML and Ruby scanners.

Some rare cases are not considered (like #{...} snippets inside of :javascript blocks),
but it highlights pretty well.

Scanners::HTML

• FIXED: Closes open string groups.

Scanners::JavaScript

• IMPROVED: Added NaN and Infinity to list of predefined constants.
• IMPROVED recognition of RegExp literals with leading spaces.

Scanners::Java

• NEW: Package names are highlighted as :namespace.
  See #210.

REMOVED: Scanners::NitroXHTML

Nitro is dead.

RENAMED: Scanners::Text (was Scanners::Plaintext)

• IMPROVED: Just returns the string without scanning (faster).

This is much faster than scanning until /\z/ in Ruby 1.8.

Scanners::Python

• CHANGED: Docstrings are highlighted as :comment.
  See #190.

NEW: Scanners::Raydebug

Copied from Scanners::Debug, highlights the token dump instead of importing it. It also reacts to the .raydebug file
name suffix now.

Scanners::Ruby

• ADDED more predefined keywords (see http://murfy.de/ruby-constants).
• IMPROVED support for singleton method definitions.
  See #147.
• FIXED: Don’t highlight methods with a capital letter as constants
  (eg. GL.PushMatrix).
• NEW: Highlight buggy floats (like .5) as :error.
• CLEANUP of documentation, names of constants and variables, state handling.

Moved StringState class from patterns.rb into a separate file.

• NEW: Complicated rule for recognition of foo= style method names.
• NEW: Handles :keep_state option (a bit; experimental).

Actually, Ruby checks if there is [~>=], but not => following the name.

• REMOVED EncodingError

REMOVED Scanners::Scheme

• It is too buggy, and nobody was using it. To be added again when it’s fixed.
  See #59.

Scanners::SQL

• IMPROVED: Extended list of keywords and functions (thanks to
  Joshua Galvez, Etienne Massip, and others).

See #221.

• FIXED: Closes open string groups.
• FIXED: Words after . are always recognized as :ident.

Scanners::YAML

• FIXED: Allow spaces before colon in mappings.

See #231.

NEW: Styles::Alpha

A style that uses transparent HSLA colors as defined in CSS 3. See #199.

It also uses the CSS 3 property user-select: none to keep the user from selecting the line numbers. This is especially
nice for :inline line numbers. See #226.

WordList

Stripped down to 19 LOC.

• RENAMED CaseIgnoringWordList to WordList::CaseIgnoring.
• REMOVED caching option because it creates memory leaks.
• REMOVED block option.

FileType

• NEW: Recognizes .gemspec, .rjs, .rpdf extensions, Gemfile, and Capfile as Ruby.

Thanks to the authors of the TextMate Ruby bundle!

• REMOVED FileType#shebang is a protected method now.

Plugin

• IMPROVED: register_for sets the plugin_id; it can now be a Symbol.
• ADDED PluginHost#const_missing method: Plugins are loaded automatically.
  Using Scanners::JavaScript in your code loads scanners/java_script.rb.
• ADDED #all_plugins method to simplify getting
  information about all available plugins (suggested by bnhymn).
• CHANGED the default plugin key from nil to :default.

GZip

• MOVED into CodeRay namespace.
• MOVED file from gzip_simple.rb to gzip.rb.
• REMOVED String extensions.

More API changes

• FIXED Encoders::HTML#token’s second parameter is no longer optional.
• CHANGED Encoders::HTML::Output’s API.
• REMOVED lots of unused methods.

The helper classes were cleaned up; see above for details.

• CHANGED Plugin API was simplified and stripped of all unnecessary features.
• CHANGED Moved GZip and FileType libraries into CodeRay; cleaned them up.