The lexer hits the maximum recursion depth given a file with too many comment lines

[Kodiologist]

 $ yes ';' | head -n 500 >/tmp/foo.hy
 $ hy /tmp/foo.hy
 Traceback (most recent call last):
   File "/home/hippo/Desktop/hyenv/bin/hy", line 11, in <module>
     load_entry_point('hy', 'console_scripts', 'hy')()
   File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 344, in hy_main
     sys.exit(cmdline_handler("hy", sys.argv))
   File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 332, in cmdline_handler
     return run_file(options.args[0])
   File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 211, in run_file
     pretty_error(import_file_to_module, "__main__", filename)
   File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 184, in pretty_error
     return func(*args, **kw)
   File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 95, in import_file_to_module
     _ast = import_file_to_ast(fpath, module_name)
   File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 53, in import_file_to_ast
     return hy_compile(import_file_to_hst(fpath), module_name)
   File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 41, in import_file_to_hst
     return import_buffer_to_hst(f.read())
   File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 34, in import_buffer_to_hst
     return tokenize(buf + "\n")
   File "/home/hippo/Desktop/hyenv/hy/hy/lex/__init__.py", line 17, in tokenize
     return parser.parse(lexer.lex(buf))
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/parser.py", line 32, in parse
     lookahead = next(tokenizer)
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 56, in __next__
     return self.next()
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 41, in next
     return self.next()
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 41, in next
     return self.next()
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 41, in next
     return self.next()
   [Previous line repeated 976 more times]
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 38, in next
     match = rule.matches(self.s, self.idx)
   File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexergenerator.py", line 33, in matches
     return Match(*m.span(0)) if m is not None else None
 RecursionError: maximum recursion depth exceeded

[refi64]

This feels awfully...familiar? What version of rply are you using?

[Kodiologist]

0.7.4, the latest on PyPI. Can you reproduce this?

[Kodiologist]

This is a bug in rply (alex/rply#52) that is fixed in master but not yet in any release.