Use Ripper for lexical analyser #512

aycabta · 2017-09-06T04:14:03Z

I replaced legacy lexical analyser from IRB with Ripper what is standard library of Ruby. It's important to support new syntax.

I explain what this code can parse Ruby code with checking diff for all Ruby documents and all Rails documents.

Preparing

Checkout each branches

mkdir test-for-rdoc
cd test-for-rdoc
git clone -b master --single-branch git@github.com:ruby/rdoc.git original-rdoc
cd original-rdoc
git checkout 0c5eb20 # This is HEAD of master as of now
bundle
cd ..
git clone -b use-ripper --single-branch git@github.com:aycabta/rdoc.git ripper-rdoc
cd ripper-rdoc
git checkout 23f12c1 # This is HEAD of use-ripper branch as of now
bundle
cd ..
git clone -b trunk --single-branch git@github.com:ruby/ruby.git
cd ruby
git checkout d68a6b3
cd ..
git clone -b master --single-branch git@github.com:rails/rails.git
cd rails
git checkout 665ac7c
cd ..

Generate CRuby documents

ruby -Ioriginal-rdoc/lib original-rdoc/exe/rdoc -o ruby-doc-original ruby
ruby -Iripper-rdoc/lib ripper-rdoc/exe/rdoc -https://github.com/zzak/sdoc/pull/112o ruby-doc-ripper ruby

Generate Rails documents

cd rials
# Tha patch below for removing monkey patch (https://github.com/zzak/sdoc/pull/112) and,
# using suitable variable (https://github.com/zzak/sdoc/pull/113) with
# suitable behavior of RDoc (https://github.com/ruby/rdoc/pull/502)
curl -fsSL https://gist.githubusercontent.com/aycabta/1d4eac210d3ede1b91a520804499a457/raw/ca90a8ce8fe3089a1d45629a01550012b2be46e1/sdoc.diff | patch -p1 
bundle update
RUBYOPT=-I../original-rdoc/lib bundle exec rake rdoc
mv doc/rdoc ../rails-doc-original
RUBYOPT=-I../ripper-rdoc/lib bundle exec rake rdoc
mv doc/rdoc ../rails-doc-ripper
cd ..

Check diff

Ruby

diff -ru --exclude "*.js" --exclude "*.gz" --exclude "*.css" --exclude "*.png" --exclude "*.gif" --exclude "*.ttf" rails-doc-original rails-doc-ripper | less

Rails

diff -ru --exclude "*.js" --exclude "*.gz" --exclude "*.css" --exclude "*.png" --exclude "*.gif" --exclude "*.ttf" ruby-doc-original ruby-doc-ripper | less

Explanation of remaining differences

These are unnecessary to fix.

Ruby

`Exception2MessageMapper.html`

  def bind(cl)
    self.module_eval "#{<<-"begin;"}\n#{<<-"end;"}", __FILE__, __LINE__+1
    begin;
      def Raise(err = nil, *rest)
        Exception2MessageMapper.Raise(self.class, err, *rest)
      end
      alias Fail Raise

      class << self
        undef included
      end
      def self.included(mod)
        mod.extend Exception2MessageMapper
      end
    end;
  end

I heard about this from @nobu, this evaluation code meta programming with complex heredoc is a trick for syntax highlighting of editor. This is strange and complex case, so I think that conforming output of tihs perfectly is unnecessary.

A method is listed at table_of_contents.html too.

`alias`

MonitorMixin.html
ruby/doc/syntax/miscellaneous_rdoc.html

For example,

alias :old_method :method

This is difficult to lexical analyse by IRB's one because it's not separated by comma. The alias syntax is special case of Ruby. In Ripper version, it becomes perfect.

`ruby/doc/contributing_rdoc.html`

git remote add my_fork git@github.com:my_username/ruby.git

Syntax highlighting for shell command as Ruby. RDoc has only Ruby syntax highlighting feature and it becomes normal quotation without syntex highlighting as Ruby when lexical analysing and parsing as Ruby fails with exception.

`Regexp.html`

def o.to_regexp() /foo/ end

A method definition one liner with regexp literal. It's so difficult to support this case, and if supports it, RDoc code becomes so complex. In Ripper version, it becomes perfect.

(GitHub syntax highlighting fails too...the correct syntax highlighted regexp literal is like below)

a =~ /foo/

Some commented out methods what are treated as document

I sent issues to ruby-core:

But in Ripper version, these skipped.

Rails

There is one difference.

--- rails-doc-original/classes/AbstractController/Callbacks/ClassMethods.html   2017-09-06 02:14:55.777896261 +0900
+++ rails-doc-ripper/classes/AbstractController/Callbacks/ClassMethods.html     2017-09-06 02:11:53.167905708 +0900
@@ -268,12 +268,14 @@
 <p>Note that <code>:only</code> has priority over <code>:if</code> in case
 they are used together.</p>
 
-<pre><code>only: :index, if: -&gt; { true } # the :if option will be ignored.</code></pre>
+<pre><code>only: :index, if: -&gt; { true } # the :if option will be ignored.
+</code></pre>

In original RDoc, the end of source code quotation is closed without newline. In Ripper RDoc, it's with newline. The IRB lexical analyser separates token to comment and the end of newline, but Ripper has the end of newline in a comment token. This is difficult to conform perfectly.

There are many cases of this in Rails diff, but these are the same as HTML rendering.

Excursus

I sent a patch for Ripper to ruby-core about lexical analysis state because this branch needed it, and I took some advices about lexical analysis state layer by @mrkn and I wrote RDoc::RipperStateLex for it. If the patch what is for Ripper is taken, I'll replace RDoc::RipperStateLex with the new Ripper after all old Ripper has gone to EOL.

def a, b, # comment c end This is valid definition of method with a comment inside args. This commit fixes behavior for it.

Remove String#match? and heredoc with ">>~".

aycabta · 2017-09-06T04:32:47Z

I warry about Rails using forked SDoc and it has wrong behavior (those are fixed original repository of SDoc, 1 and 2).

@zzak: I want to join SDoc collaborators and RubyGems owners for fix some trouble in the future, and it's just an aside, let's talk about some documentations in the future.

@amatsuda and/or other Rails committers: This SDoc problem is so severe for replacement lexical analyser with Ripper because RDoc crashes with this Pull Request certainly until now, especially about this monkey patch. I think that Rails should use original SDoc as new version.

drbrain · 2017-09-06T21:26:28Z

My years-long dream come true

drbrain · 2017-09-06T21:47:14Z

lib/rdoc/parser/ruby.rb

+        }  && $~
+        line
+      end.join("\n")
+    end


Too bad expand_tab doesn't support @options.tab_width

I filed an issue about it.

amatsuda · 2017-09-06T22:04:59Z

No worries about the sdoc thing. We can switch to any version so far as it produces correct output.

aycabta · 2017-09-07T09:20:45Z

Everyone, please test this branch for documentation to your software and check diff between original and this one.

aycabta · 2017-09-07T15:56:49Z

I wrote doctest.sh for checking diff between original RDoc and Ripper version RDoc of specified GitHub repository.

$ git clone git@gist.github.com:24a122ba798337cc3b9a7bfb32fc28f4.git doctest
$ cd doctest
$ ./doctest.sh
Usage: ./doctest.sh username/repository
$ ./doctest.sh ruby/ruby
# run diff command with less command

robin850 · 2017-09-08T16:02:41Z

Awesome work @aycabta ! 👏 ❤️

Rails is currently pointing to a fork of SDoc because the API site relies on unreleased changed and we are waiting for 1.0.0 final to be cut before updating the Gemfile but no worries, this is temporary.

By the way, the generation time is reduced by ~10s in the Rails repository with this patch ; great ! As you mentioned earlier, the outputs aren't exactly the same but apart from different code highlighting, everything seems fine regarding Rails.

aycabta · 2017-09-08T19:11:00Z

Rails is currently pointing to a fork of SDoc because the API site relies on unreleased changed and we are waiting for 1.0.0 final to be cut before updating the Gemfile but no worries, this is temporary.

OK, I understand what is going on.

everything seems fine regarding Rails.

Thank you, I think that this Pull Request is practically settled.

hsbt · 2017-09-11T06:13:28Z

Some commented out methods what are treated as document

I merged these patches into ruby core.

hsbt · 2017-09-11T06:35:07Z

I confirmed that differences of rdoc documents generated old-rdoc and ripper version. I didn't find another difference without Explanation of remaining differences.

I approved to merge this pull request and release 6.0.0.beta2 before RubyKaigi 2017.

@aycabta How do you think about this release?

aycabta · 2017-09-11T13:52:37Z

@hsbt You sure can, that's why I wrote all code.

aycabta · 2017-09-12T02:09:40Z

💕 🌸 💞

* This version changed lexer used Ripper from lexer based IRB. see details: ruby/rdoc#512 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59845 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

ghost · 2017-10-11T09:09:50Z

🌸

aycabta added 30 commits September 5, 2017 19:43

Use Ripper instead of IRB's lex

9fe38ca

Fix collect_first_comment for Ripper

ae8e8ce

Fix get_class_or_module for Ripper

d4d684c

Fix parse_statements for Ripper

c39f7f0

Fix read_directive for Ripper

1e61ab1

Remove unnecessary method calling

9f59a91

Fix parse_module for Ripper

5dcc452

Fix test_get_class_or_module

9338eec

Raise RDoc::Error when invalid class or module definition

7cd8575

Comment out some old code

b5443f9

Fix test_parse_method for Ripper

e914d68

Fix for test_parse_alias

9eab386

Fix for rdoc command running

32eec2a

Fix for test_parse_method_nil

5021a1b

Fix for test_parse_multi_ghost_methods

abc30fd

Fix for test_parse_method_constant

352f5ed

Fix for test_parse_method_toplevel

829fdba

Fix test_parse_statements_def_percent_string_pound for Ripper

1a2b601

Fix for test_parse_method_gvar_insane

d94a38d

Fix for test_parse_method_parameters_comment

e2d63ff

Fix for test_collect_first_comment

83b5b78

Fix for test_parse_method_toplevel_class

f6111fd

Fix for test_parse_meta_method_name

d10dcc5

Fix for test_parse_meta_method_string_name

b9d53b2

Fix test_parse_meta_method for Ripper

a104381

Fix for test_parse_meta_method_block

55a18bb

Fix for test_parse_meta_method_unknown

6fe9b92

Fix for test_parse_class_single

e37e73e

Fix for test_parse_class_single_root

9d677e8

Fix for test_parse_constant.*

b7ec322

aycabta added 6 commits September 5, 2017 19:57

Remove unnecessary variable

a2b511c

Fix test_parse_statements_def_percent_string_pound

2725175

Fix test_parse_redefinable_methods

80b7445

Fix handling method parmeter

f3807d7

def a, b, # comment c end This is valid definition of method with a comment inside args. This commit fixes behavior for it.

Add test_parse_method_parameters_with_paren_comment_continue

6ff1766

Support earlier than Ruby 2.4

1946c08

Remove String#match? and heredoc with ">>~".

drbrain reviewed Sep 6, 2017

View reviewed changes

hsbt self-requested a review September 7, 2017 01:44

This was referenced Sep 10, 2017

Ripper fails to lexical analyse around escaped newline on JRuby jruby/jruby#4787

Closed

Use "bundle exec" in .travis.yml for JRuby test #515

Merged

hsbt approved these changes Sep 12, 2017

View reviewed changes

hsbt merged commit e96d747 into ruby:master Sep 12, 2017

aycabta deleted the use-ripper branch September 12, 2017 02:03

aycabta mentioned this pull request Sep 19, 2017

Ruby 2.5.0-preview1 Released ruby/www.ruby-lang.org#1641

Merged

2 tasks

headius mentioned this pull request Dec 1, 2017

Ruby 2.5 Support jruby/jruby#4876

Closed

75 tasks

aycabta mentioned this pull request Dec 20, 2017

Bullet points of 6.0 changes for CRuby's ChangeLog #504

Closed

robin850 mentioned this pull request Jan 25, 2018

Fix #tokens_to_s method for token streams #592

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Ripper for lexical analyser #512

Use Ripper for lexical analyser #512

aycabta commented Sep 6, 2017 •

edited

aycabta commented Sep 6, 2017 •

edited

drbrain commented Sep 6, 2017

drbrain Sep 6, 2017

aycabta Sep 7, 2017

amatsuda commented Sep 6, 2017

aycabta commented Sep 7, 2017

aycabta commented Sep 7, 2017

robin850 commented Sep 8, 2017

aycabta commented Sep 8, 2017

hsbt commented Sep 11, 2017

hsbt commented Sep 11, 2017

aycabta commented Sep 11, 2017

aycabta commented Sep 12, 2017

ghost commented Oct 11, 2017

Use Ripper for lexical analyser #512

Use Ripper for lexical analyser #512

Conversation

aycabta commented Sep 6, 2017 • edited

Preparing

Checkout each branches

Generate CRuby documents

Generate Rails documents

Check diff

Ruby

Rails

Explanation of remaining differences

Ruby

Exception2MessageMapper.html

alias

ruby/doc/contributing_rdoc.html

Regexp.html

Some commented out methods what are treated as document

Rails

Excursus

aycabta commented Sep 6, 2017 • edited

drbrain commented Sep 6, 2017

drbrain Sep 6, 2017

Choose a reason for hiding this comment

aycabta Sep 7, 2017

Choose a reason for hiding this comment

amatsuda commented Sep 6, 2017

aycabta commented Sep 7, 2017

aycabta commented Sep 7, 2017

robin850 commented Sep 8, 2017

aycabta commented Sep 8, 2017

hsbt commented Sep 11, 2017

hsbt commented Sep 11, 2017

aycabta commented Sep 11, 2017

aycabta commented Sep 12, 2017

ghost commented Oct 11, 2017

aycabta commented Sep 6, 2017 •

edited

`Exception2MessageMapper.html`

`alias`

`ruby/doc/contributing_rdoc.html`

`Regexp.html`

aycabta commented Sep 6, 2017 •

edited