Improve compatibility with MRI's Ripper for JRuby 9.1 #4900
First, this ignores the entire file that relies on the MRI file structure, as it seems unlikely that any of those tests will ever be applicable. Second, it adds the test_sexp.rb file which was absent from the index for some unknown reason. It also introduces a couple of excludes for failing tests in that file. Third, it clears the excludes for a couple of tests that are currently passing, and updates the description for one of the excludes.
The previous implementation called assignable on a bunch of things that were unlikely to be strings, while only supporting string arguments. This changes the logic so that it relies on getting the identifier value from the lexer instead. In addition, this inlines the logic from assignable in all cases where the implementation is known, and splits the cases for identifiers and constants, as they have very little to do with each other. Given that the parser has inlined the definition of MRI's user_varaible and keyword variable, it seems to make sense to also inline the assignable actions. While inlining the assignable implementation, it seamed to make sense to also inline is_id_var (and remove its unused argument).
Instead of ending the heredoc before emitting the final string content part, this instead just emits the final string content part and positions the cursor right before the end marker, such that the next call to the heredoc term should end the heredoc using the early return. The motivation for doing this was that with ripper the final string content of a heredoc wouldn't be dispatched to any token handler, but instead just propagated. The old behavior was clearly wrong for this corner-case, but this approach might have other problems.
As an added bonus, the parsing of magic comment key values no longer stops at the first unrecognized key, and thus writing # -*- warn_past_scope: true; coding: ISO-8859-1 -*- will now apply the encoding even if JRuby doesn't support the magic comment attribute warn_past_scope.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments.