Fatal error scanning invalid Ruby syntax

[declanvk]

Hello,
I'm running into an issue while using the pry debugger (version 0.10.4) that in turns uses the coderay gem (version 1.1.1) for syntax highlighting. Inputting a: / 1 leads to the pry debugger exiting with the following error:

[1] pry(main)> a: / 1
~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/coderay-1.1.1/lib/coderay/scanners/ruby.rb:434:in `scan_tokens':  (CodeRay::Scanners::Scanner::ScanError)

***ERROR in scanner.rb:315:in `raise_inspect': undefined method `+' for nil:NilClass (after 10 tokens)

tokens:
"a"
:key
":"
:operator
" "
:space
:begin_group
:regexp
"/"
:delimiter

current line: 1  column: 7  pos: 6
matched: nil  state: "Error in CodeRay::Scanners::Ruby#scan_tokens, initial state was: :initial"
bol?: false,  eos?: true


surrounding code:
nil  ~~  ""


***ERROR***

	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/coderay-1.1.1/lib/coderay/scanners/scanner.rb:172:in `tokenize'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/coderay-1.1.1/lib/coderay/tokens_proxy.rb:39:in `tokens'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/indent.rb:273:in `tokenize'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/indent.rb:150:in `block in indent'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/indent.rb:143:in `each'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/indent.rb:143:in `indent'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:105:in `read'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:68:in `block in repl'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:67:in `loop'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:67:in `repl'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:38:in `block in start'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/input_lock.rb:61:in `__with_ownership'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/input_lock.rb:79:in `with_ownership'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:38:in `start'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/repl.rb:15:in `start'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-0.10.4/lib/pry/pry_class.rb:169:in `start'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/pry-byebug-3.4.2/lib/pry-byebug/pry_ext.rb:11:in `start_with_pry_byebug'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/rails-552151f36571/railties/lib/rails/commands/console.rb:110:in `start'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/rails-552151f36571/railties/lib/rails/commands/console.rb:9:in `start'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/rails-552151f36571/railties/lib/rails/commands/commands_tasks.rb:68:in `console'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/rails-552151f36571/railties/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/rails-552151f36571/railties/lib/rails/commands.rb:17:in `<top (required)>'
	from script/rails:6:in `require'
	from script/rails:6:in `<main>'

Checking to make sure that it wasn't an issue with pry I reproduced the issue with the following script:

require 'coderay'
CodeRay.scan('a: / 1').term

which produced a similar, if truncated, error:

CodeRay::Scanners::Scanner::ScanError: 

***ERROR in scanner.rb:315:in `raise_inspect': undefined method `+' for nil:NilClass (after  tokens)

tokens:


current line: 1  column: 7  pos: 6
matched: nil  state: "Error in CodeRay::Scanners::Ruby#scan_tokens, initial state was: :initial"
bol?: false,  eos?: true


surrounding code:
nil  ~~  ""


***ERROR***

from ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/coderay-1.1.1/lib/coderay/scanners/ruby.rb:434:in `scan_tokens'

I can't say that I know very much about this software, so I did not submit a PR with this issue.

Thank you for your time,
Declan Kelly

P.S. maybe some sort of fuzzing would be a cool addition to your testing code

[korny]

Hi Declan!

Thanks for the bug report. I wasn't able to reproduce it with 1.1.1:

git co v1.1.1
irb -Ilib -rcoderay
2.4.1 :001 > CodeRay.scan('a: / 1').term
ArgumentError: wrong number of arguments (given 1, expected 2..3)
	from /Users/murphy/ruby/coderay/lib/coderay.rb:168:in `scan'
	from (irb):1
	from /Users/murphy/.rvm/rubies/ruby-2.4.1/bin/irb:11:in `<main>'
2.4.1 :002 > CodeRay.scan('a: / 1', :ruby).term
 => "\e[35ma\e[0m: \e[35m\e[1;35m/\e[0m\e[35m 1\e[0m" 

Is it possible that this isn't the exact code you were using?

[korny]

P.S. maybe some sort of fuzzing would be a cool addition to your testing code

You mean something like this? ;-) It's already part of the extensive test suite. Scanners are pretty much bullet-proof, at least against typical strange code (your example would be a Ruby syntax error).

[declanvk]

I apologize for the delay. I think this might have been an issue with a previous version, or maybe something to do with the pry debugger on top. I'm no longer in a position to debug the error I submitted, but thank you for this response.

P.S. Thats a pretty cool test you have there.