Skip to content

Conversation

@Earlopain
Copy link
Collaborator

@Earlopain Earlopain commented Jan 15, 2025

Without a change from parser, this is impossible to correctly handle.

I played around with the following:

diff --git a/lib/prism/translation/parser.rb b/lib/prism/translation/parser.rb
index 6b417be4..92525bd4 100644
--- a/lib/prism/translation/parser.rb
+++ b/lib/prism/translation/parser.rb
@@ -76,9 +76,9 @@ module Prism
 
       # Parses a source buffer and returns the AST, the source code comments,
       # and the tokens emitted by the lexer.
-      def tokenize(source_buffer, recover = false)
+      def tokenize(source_buffer, recover = false, raw_source: source_buffer.source)
         @source_buffer = source_buffer
-        source = source_buffer.source
+        source = raw_source
 
         offset_cache = build_offset_cache(source)
         result =
diff --git a/test/prism/ruby/parser_test.rb b/test/prism/ruby/parser_test.rb
index 0a2ca196..9df3c48e 100644
--- a/test/prism/ruby/parser_test.rb
+++ b/test/prism/ruby/parser_test.rb
@@ -145,7 +145,8 @@ module Prism
 
     def assert_equal_parses(fixture, compare_asts: true, compare_tokens: true, compare_comments: true)
       buffer = Parser::Source::Buffer.new(fixture.path, 1)
-      buffer.source = fixture.read
+      raw_source = fixture.read
+      buffer.source = raw_source
 
       parser = Parser::Ruby33.new
       parser.diagnostics.consumer = ->(*) {}
@@ -159,7 +160,7 @@ module Prism
         end
 
       actual_ast, actual_comments, actual_tokens =
-        ignore_warnings { Prism::Translation::Parser33.new.tokenize(buffer) }
+        ignore_warnings { Prism::Translation::Parser33.new.tokenize(buffer, raw_source: raw_source) }
 
       if expected_ast == actual_ast
         if !compare_asts

But I think that would mean to keep track of every encountered \r\n and keep some global offset to make everything correct. That seems like quite the hassle for barely any gain.

Without a change from `parser`, this is impossible to correctly handle.
@Earlopain Earlopain force-pushed the parser-translator-known-failures branch from b7a341f to ca1d44e Compare January 15, 2025 09:45
@kddnewton kddnewton merged commit de7bb68 into ruby:main Jan 15, 2025
57 checks passed
@Earlopain Earlopain deleted the parser-translator-known-failures branch January 16, 2025 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants