Permalink
Browse files

Major rewrite of encoders to support IO output; fixed some minor scan…

…ner bugs; cleanups; dropped NitroXHTML scanner; improved tests
  • Loading branch information...
1 parent fdd17b6 commit 75bc5455af8c3c3381066aac3d5fff42264cac6f @korny korny committed Aug 19, 2011
Showing with 713 additions and 671 deletions.
  1. +329 −0 Changes-1.0.textile
  2. +0 −296 Changes.textile
  3. +1 −0 Gemfile
  4. +31 −9 bin/coderay
  5. +17 −33 etc/CodeRay.tmproj
  6. +6 −6 etc/language_report.textile
  7. +2 −1 lib/coderay.rb
  8. +11 −1 lib/coderay/encoder.rb
  9. +0 −1 lib/coderay/encoders/_map.rb
  10. +9 −3 lib/coderay/encoders/count.rb
  11. +1 −1 lib/coderay/encoders/debug.rb
  12. +12 −6 lib/coderay/encoders/filter.rb
  13. +11 −1 lib/coderay/encoders/html.rb
  14. +1 −1 lib/coderay/encoders/html/css.rb
  15. +0 −1 lib/coderay/encoders/html/output.rb
  16. +20 −7 lib/coderay/encoders/json.rb
  17. +2 −1 lib/coderay/encoders/lines_of_code.rb
  18. +5 −6 lib/coderay/encoders/statistic.rb
  19. +8 −5 lib/coderay/encoders/text.rb
  20. +1 −0 lib/coderay/encoders/token_kind_filter.rb
  21. +5 −3 lib/coderay/encoders/xml.rb
  22. +13 −8 lib/coderay/encoders/yaml.rb
  23. +2 −2 lib/coderay/helpers/file_type.rb
  24. +7 −5 lib/coderay/helpers/plugin.rb
  25. +30 −18 lib/coderay/scanner.rb
  26. +2 −2 lib/coderay/scanners/_map.rb
  27. +1 −1 lib/coderay/scanners/clojure.rb
  28. +36 −27 lib/coderay/scanners/css.rb
  29. +3 −3 lib/coderay/scanners/{rhtml.rb → erb.rb}
  30. +1 −1 lib/coderay/scanners/groovy.rb
  31. +1 −1 lib/coderay/scanners/java_script.rb
  32. +0 −136 lib/coderay/scanners/nitro_xhtml.rb
  33. +2 −2 lib/coderay/scanners/php.rb
  34. +11 −6 lib/coderay/scanners/ruby.rb
  35. +1 −3 lib/coderay/tokens.rb
  36. +1 −1 rake_tasks/diff.rake
  37. +3 −0 test/executable/source_with_comments.rb
  38. +52 −24 test/executable/suite.rb
  39. +26 −15 test/functional/basic.rb
  40. +2 −0 test/functional/examples.rb
  41. +12 −6 test/functional/for_redcloth.rb
  42. +2 −1 test/functional/suite.rb
  43. +0 −1 test/lib/README
  44. +15 −0 test/lib/assert_warning.rb
  45. +2 −2 test/unit/file_type.rb
  46. +7 −15 test/unit/plugin.rb
  47. +1 −1 test/unit/plugins/example.rb
  48. +1 −1 test/unit/plugins/user_defined/user_plugin.rb
  49. +0 −5 test/unit/plugins_with_default/default.rb
  50. +5 −0 test/unit/plugins_with_default/default_plugin.rb
  51. +2 −2 test/unit/plugins_with_default/example_without_register_for.rb
View
@@ -0,0 +1,329 @@
+h1=. CodeRay Version History
+
+p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
+
+{{toc}}
+
+h2. Changes in 1.0
+
+CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.
+
+The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.
+
+h3. General changes
+
+* *NEW*: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
+* *NEW*: Extended support and usage of HTML5 and CSS 3 features.
+* *NEW*: Direct Streaming
+* *IMPROVED* documentation
+* *IMPROVED* speed: faster startup (using @autoload@), scanning, and encoding
+* *IMPROVED* Ruby 1.9 encodings support
+* *IMPROVED* Tests: There are more of them now!
+
+h3. Direct Streaming
+
+CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners,
+Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
+this version.
+
+The main benefits of this change are:
+
+* more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
+* the ability to stream output into a pipe on the command line
+* a simpler API
+* less code
+
+Changes related to the new tokens handling include:
+* *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
+ Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
+ special case; Tokens just encodes the given tokens into an Array for later use.
+* *CHANGED*: The token actions (@text_token@, @begin_group@ etc.) are now public methods of @Encoder@ and @Tokens@.
+* *REWRITE* of all Scanners, Encoders, Filters, and Tokens.
+* *RENAMED* @:open@ and @:close@ actions to @:begin_group@ and @:end_group@.
+* *RENAMED* @open_token@ and @close_token@ methods to @begin_group@ and @end_group@.
+* *NEW* method @#tokens@ allows to add several tokens to the stream. @Tokens@ and @Encoders::Encoder@ define this
+ method.
+* *CHANGED* The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible
+ with earlier versions.
+* *REMOVED* @TokenStream@ and the @Streamable@ API and all related features like @NotStreamableError@ are now obsolete
+ and have been removed.
+
+h3. Command Line
+
+The @coderay@ executable was rewritten and has a few new features:
+
+* *NEW* Ability to stream into a pipe; try @coderay file | more -r@
+* *NEW* help
+* *IMPROVED*: more consistent parameter handling
+* *REMOVED* @coderay_stylesheet@ executable; use @coderay stylesheet [name]@.
+
+h3. @Tokens@
+
+* *NEW* methods @encode_with@, @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.
+* *REMOVED* methods @#stream?@, @#each_text_token@.
+* *REMOVED* @#text@ and @#text_size@ methods. Use the @Text@ encoder instead.
+* *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.
+
+h3. *RENAMED*: @TokenKinds@
+
+Renamed from @Tokens::ClassOfKind@ (was also @Tokens::AbbreviationForKind@ for a while).
+The term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.
+See "#122":http://redmine.rubychan.de/issues/122.
+
+* *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.
+* *RENAMED* @:pre_constant@ and @:pre_type@ to @:predefined_constant@ and @predefined_type@.
+* *REMOVED* token kinds @:attribute_name_fat@, @:attribute_value_fat@, @:operator_fat@,
+ @:tag_fat@, @:xml_text@, @:nesting_delimiter@, @:open@, and @:close@.
+* *CHANGED*: Don't raise error for unknown token kinds unless in @$CODERAY_DEBUG@ mode.
+* *CHANGED* the value for a token kind that is not highlighted from
+ @:NO_HIGHLIGHT@ to @false@.
+
+h3. @Duo@
+
+* *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.
+
+h3. @Encoders::CommentFilter@
+
+* *NEW* alias @:remove_comments@
+
+h3. @Encoders::Filter@
+
+* *NEW* option @tokens@.
+* *CHANGED*: Now it simply delegates to the output.
+* *REMOVED* @include_text_token?@ and @include_block_token?@ methods.
+
+h3. @Encoders::HTML@
+
+The HTML encoder was cleaned up and simplified.
+
+* *NEW*: HTML5 and CSS 3 compatible.
+ See "#215":http://redmine.rubychan.de/issues/215.
+* *ADDED* support for @:line_number_anchors@.
+ See "#208":http://redmine.rubychan.de/issues/208.
+* *CHANGED* the default style to @:alpha@.
+* *CHANGED*: Use double click to toggle line numbers in table mode (as single
+ click jumps to an anchor.)
+* *REMOVED* support for @:line_numbers => :list@.
+* *FIXED* splitting of lines for @:line_numbers => :inline@, so that the line
+ numbers don't get colored, too.
+* *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.
+
+h3. @Encoders::LinesOfCode@
+
+* *CHANGED*: @compile@ and @finish@ methods are now protected.
+
+h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)
+
+* *RENAMED* from @Encoders::Term@, added @:term@ alias.
+* *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.
+* *CHANGED*: @#token@'s second parameter is no longer optional.
+* *REMOVED* colors for obsolete token kinds.
+* *FIXED* handling of line tokens.
+
+h3. @Encoders::Text@
+
+* *FIXED* default behavior of stripping the trailing newline.
+
+h3. *RENAMED*: @Encoders::TokenKindFilter@ (was @Encoders::TokenClassFilter@)
+
+* *NEW*: Handles token groups.
+ See "#223":http://redmine.rubychan.de/issues/223.
+* *RENAMED* @include_block_token?@ to @include_group?@.
+
+h3. @Encoders::Statistic@
+
+* *CHANGED*: Tokens actions are counted separately.
+
+h3. @Scanners::Scanner@
+
+* *NEW* methods @#file_extension@ and @#encoding@.
+* *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The
+ code is highlighted as one and split into parts of the input lengths
+ after that using @Tokens#split_into_parts@.
+* *NEW* method @#binary_string@
+* *REMOVED* helper method @String#to_unix@.
+* *REMOVED* method @#streamable?@.
+* *REMOVED* @#marshal_load@ and @#marshal_dump@.
+* *RENAMED* class method @normify@ to @normalize@; it also deals with encoding now.
+* *CHANGED*: @#column@ starts counting with 1 instead of 0
+
+h3. *NEW*: @Scanners::Clojure@
+
+Thanks to Licenser, CodeRay now supports the Clojure language.
+
+h3. @Scanners::CSS@
+
+* *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.
+ See "#224":http://redmine.rubychan.de/issues/224.
+* *NEW*: Rudimentary support for CSS 3 colors.
+* *CHANGED*: Attribute selectors are highlighted as @:attribute_name@ instead of @:string@.
+* *CHANGED*: Comments are scanned as one token instead of three.
+
+h3. @Scanners::Debug@
+
+* *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)
+* *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.
+* *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.
+* *IMPROVED*: Highlight unknown tokens as @:error@.
+* *CHANGED*: Raises an error when trying to end an invalid token group.
+
+h3. @Scanners::Delphi@
+
+* *FIXED*: Closes open string groups.
+
+h3. @Scanners::Diff@
+
+* *NEW*: Highlighting of code based on file names.
+ See ticket "#52":http://redmine.rubychan.de/issues/52.
+
+ Use the @:highlight_code@ option to turn this feature off. It's enabled
+ by default.
+
+ This is a very original feature. It enables multi-language highlighting for
+ diff files, which is especially helpful for CodeRay development itself. The
+ updated version of the scanner test suite generated .debug.diff.html files
+ using this.
+
+ Note: This is still experimental. Tokens spanning more than one line
+ may get highlighted incorrectly. CodeRay tries to keep scanner states
+ between the lines and changes, but the quality of the results depend on
+ the scanner.
+* *NEW*: Inline change highlighting, as suggested by Eric Thomas.
+ See ticket "#227":http://redmine.rubychan.de/issues/227 for details.
+
+ Use the @:inline_diff@ option to turn this feature off. It's enabled by
+ default.
+
+ For single-line changes (that is, a single deleted line followed by a single
+ inserted line), this feature surrounds the changed parts with an
+ @:eyecatcher@ group which appears in a more saturated background color.
+ The implementation is quite complex, and highly experimental. The problem
+ with multi-layer tokenizing is that the tokens have to be split into parts.
+ If the inline change starts, say, in the middle of a string, then additional
+ @:end_group@ and @:begin_group@ tokens must be inserted to keep the group
+ nesting intact. The extended @Scanner#tokenize@ method and the new
+ @Tokens#split_into_parts@ method take care of this.
+* *NEW*: Highlight the file name in the change headers as @:filename@.
+* *CHANGED*: Highlight unknown lines as @:comment@ instead of @:head@.
+* *IMPROVED*: Background colors for Diff output have been optimized.
+
+h3. *RENAMED*: @Scanners::ERB@ (was @Scanners::RHTML@)
+
+
+h3. @Scanners::HTML@
+
+* *FIXED*: Closes open string groups.
+
+h3. @Scanners::JavaScript@
+
+* *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.
+* *IMPROVED* recognition of RegExp literals with leading spaces.
+
+h3. @Scanners::Java@
+
+* *NEW*: Package names are highlighted as @:namespace@.
+ See "#210":http://redmine.rubychan.de/issues/210.
+
+h3. *RENAMED*: @Scanners::Text@ (was @Scanners::Plaintext@)
+
+* *IMPROVED*: Just returns the string without scanning (faster).
+
+ This is much faster than scanning until @/\z/@ in Ruby 1.8.
+
+h3. @Scanners::Python@
+
+* *CHANGED*: Docstrings are highlighted as @:comment@.
+ See "#190":http://redmine.rubychan.de/issues/190.
+
+h3. *NEW*: @Scanners::Raydebug@
+
+Copied from @Scanners::Debug@, highlights the token dump instead of importing it. It also reacts to the @.raydebug@ file
+name suffix now.
+
+h3. @Scanners::Ruby@
+
+* *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).
+* *IMPROVED* support for singleton method definitions.
+ See "#147":http://redmine.rubychan.de/issues/147.
+* *FIXED*: Don't highlight methods with a capital letter as constants
+ (eg. GL.PushMatrix).
+* *NEW*: Highlight buggy floats (like .5) as @:error@.
+* *CLEANUP* of documentation, names of constants and variables, state handling.
+
+ Moved @StringState@ class from @patterns.rb@ into a separate file.
+* *NEW*: Complicated rule for recognition of @foo=@ style method names.
+* *NEW*: Handles @:keep_state@ option (a bit; experimental).
+
+ Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.
+
+* *REMOVED* @EncodingError@
+
+h3. *REMOVED* @Scanners::Scheme@
+
+* It is too buggy, and nobody was using it. To be added again when it's fixed.
+ See "#59":http://redmine.rubychan.de/issues/59.
+
+h3. @Scanners::SQL@
+
+* *IMPROVED*: Extended list of keywords and functions (thanks to
+ Joshua Galvez, Etienne Massip, and others).
+
+ See "#221":http://redmine.rubychan.de/issues/221.
+* *FIXED*: Closes open string groups.
+* *FIXED*: Words after @.@ are always recognized as @:ident@.
+
+h3. @Scanners::YAML@
+
+* *FIXED*: Allow spaces before colon in mappings.
+
+ See "#231":http://redmine.rubychan.de/issues/231.
+
+h3. *NEW*: @Styles::Alpha@
+
+A style that uses transparent HSLA colors as defined in CSS 3. See "#199":http://redmine.rubychan.de/issues/199.
+
+It also uses the CSS 3 property @user-select: none@ to keep the user from selecting the line numbers. This is especially
+nice for @:inline@ line numbers. See "#226":http://redmine.rubychan.de/issues/226.
+
+h3. @WordList@
+
+Stripped down to 19 LOC.
+
+* *REMOVED* caching option because it creates memory leaks.
+* *REMOVED* block option.
+
+h3. @FileType@
+
+* *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions, @Gemfile@, and @Capfile@ as Ruby.
+
+ Thanks to the authors of the TextMate Ruby bundle!
+* *REMOVED* @FileType#shebang@ is a protected method now.
+
+h3. @Plugin@
+
+* *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.
+* *ADDED* @PluginHost#const_missing@ method: Plugins are loaded automatically.
+ Using @Scanners::JavaScript@ in your code loads @scanners/java_script.rb@.
+* *ADDED* @#all_plugins@ method to simplify getting
+ information about all available plugins (suggested by bnhymn).
+* *CHANGED* the default plugin key from @nil@ to @:default@.
+
+h3. @GZip@
+
+* *MOVED* into @CodeRay@ namespace.
+* *MOVED* file from @gzip_simple.rb@ to @gzip.rb@.
+* *REMOVED* @String@ extensions.
+
+h3. More API changes
+
+* *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.
+* *CHANGED* @Encoders::HTML::Output@'s API.
+* *REMOVED* lots of unused methods.
+
+The helper classes were cleaned up; see above for details.
+
+* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
+* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
+
+
+
Oops, something went wrong.

0 comments on commit 75bc545

Please sign in to comment.