Parser: correctly lex the prefix of keyword in macro

[makenowjust]

Fixed #4656

I'll explain why this fix is related to #4656. Following code is minified #4656 example:

{% begin %}
  {{ :foo }}s
{% end %}

foo({
  foo: 1,
  foo_bar: 42,
})

This example produces formatting error because of IndexError which is raised from align_infos. In other words, #4656's reason is the formatter try to align incorrect line (If this incorrect line has enough columns to align, it produces incorrect result like #4656 gif, otherwise if it doesn't have, it raises IndexError).

Then, why the formatter try to align incorrect line, is the lexer is buggy. The lexer will report incorrect line number when a newline follows the part of some keywords inside macro (for example s is the part of struct). This bug is explained in detail on the source code comments, see them. And so, the formatter mistakes line number by this bug.

---

@sdogruyol expected me good thing, this helped me really. Thank you 😊

[sdogruyol]

You're awesome @makenowjust 🎉

[makenowjust]

ping anyone. Many people need this fix.

[RX14]

This just needs another review, seems like a simple fix to me.

[makenowjust]

Sorry 😄

[bmulvihill]

Small typo here: @reader

[makenowjust]

@RX14

I consider another way.

Add such a method:

def lookahead
  old_pos = @reader.pos
  old_line_number, old_column_number = @line_number, @column_number

  yield.tap do |result|
    unless result
      @reader.pos = old_pos
      @line_number, @column_number = old_line_number, old_column_number    
    end
  end
end

And, use it to check keyword, like:

          if !delimiter_state && whitespace && lookahead { char == 'y' && next_char == 'i' && next_char == 'e' && next_char == 'l' && next_char == 'd' && !ident_part_or_end?(peek_next_char) }

Perhaps it makes same result (I don't test it.) However, it is a bit heavier process than this PR way.

[RX14]

@makenowjust I think I like that idea more. It seems like a cleaner and more reusable way to do it.

[sdogruyol]

Can someone please review this 🙏

[RX14]

Are we not waiting for @makenowjust to etst the suggested improvement?

[makenowjust]

@RX14 Fixed! What do you think?

[RX14]

👌