resyntax analyze exits with contract violation in closed-range

[blakelewis]

When I run resyntax analyze on a smallish piece of code that I wrote, it exits with the message

resyntax: analyzing /Users/blakelewis/Exercism/racket/house/house.rkt
closed-range: contract violation;
 lower endpoint must be less than or equal to upper endpoint
      lower: 18
      upper: 17
      cmp: #<comparator:natural<=>>
...
  blaming: <pkgs>/resyntax/private/linemap.rkt

followed by a long list of context (full output attached).

The code is for an exercise on exercism.org, a site that lets you solve fairly simple programming challenges in a wide variety of languages. It produces verses from the nursery rhyme "The House that Jack Built." We are looking at adding resyntax to our submission pipeline so that students will see its suggestions when they submit a solution. It is a very nice tool and I ran it successfully on several other pieces of code, but something about this one seems to have upset it.

Background information: I am running Racket v8.14 on an x86_64 Macbook running MacOS 12.7.6. I installed resyntax today using raco install as shown in the README file. I am using the command resyntax analyze --file house.rkt and the contract violation happens every time. The code passes the tests on exercism.org, so it is at least somewhat sane.

Apparently, I can't attach .rkt files, so I have pasted .txt onto the end of the file name.

house.rkt.txt
anal.txt

[blakelewis]

I had the same contract violation with another file.
triangle.rkt.txt

[jackfirth]

Okay I was able to reproduce this, and it has something to do with the whitespace. That triangle.rkt file seems to have tab characters in it. When I replace them with spaces, Resyntax works fine.

Still no clue why tab characters would screw with things though. Will investigate further.

[blakelewis]

Thanks. I use vim, which indents with tabs followed by spaces unless you tell it not to, and I was suspicious about that. But I tried the experiment of converting the tabs to 8 spaces and still had the same problem. I even tried reindenting the whole thing by hand and made sure that there were only spaces, but it didn't help.

Then I noticed that vim says that the file is in "dos" format, i.e., using "\r\n" as line terminators. I guess that the stubs that I downloaded from the Exercism site must have been in that format. I told vim to change to "unix" format ("\n" only) and then it worked for me. Same for the other file, even with tabs in it.

[blakelewis]

Just to confirm, I have tried several other files with the same result: changing to dos format makes resyntax fail as described above and going back to unix format makes it succeed again.

[jackfirth]

Digging into this more, this documentation sheds some light on matters:

When counting lines, Racket treats linefeed, return, and return-linefeed combinations as a line terminator and as a single position (on all platforms).

So, even though \r\n counts as two characters in the underlying string, the syntax object position considers it a single character. Which is why these two syntax objects:

#lang racket
(require resyntax/private/source)
(define program1
    (string-append "#lang racket/base\r\n"
                   "(define (foo)\r\n"
                   "  (let ([x 42])\r\n"
                   "    (* x 2)))\r\n"))

(define program2
    (string-append "#lang racket/base\n"
                   "(define (foo)\n"
                   "  (let ([x 42])\n"
                   "    (* x 2)))\n"))


(source-read-syntax (string-source program1))
(source-read-syntax (string-source program2))

...both report the exact same source location for the define identifier.

I might be able to fix this by auto-replacing any \r\n character sequences in the file with a single \n character. But that seems pretty destructive to do in resyntax analyze. Alternatively, Resyntax could refuse to analyze files with \r\n character sequences, and display an error message informing the user of what's going on.

[blakelewis]

Can you walk me through how this leads to the contract violation? I see that the Racket port code is counting "\r\n" as one "position" and I assume that this is throwing resyntax off the rails, but I have only taken a quick and superficial look at the code, so I can't connect the dots. I am wondering if there is some way to use the : location information in a way that works for either style of terminator.

[jackfirth]

I'm not sure exactly how it leads to that specific contract error, but I know it violates one of the fundamental assumptions Resyntax relies on: that the syntax-position and syntax-span information on a syntax object is enough to uniquely identify the character range in the original source code that syntax object represents. There's lots of places in Resyntax where I relied on that assumption. I'm sure this breaks things in multiple ways.

I think I can fix the root problem by extending the "linemap" data structure I use internally to track where multi-character newlines are. If I do that, then given a linemap for a file, the above assumption I mentioned will hold. I'll try that fix sometime soon.

In the meantime, you can guarantee this won't be an issue by removing \r\n from any text/files you want to analyze. Is that feasible for your use case?

[blakelewis]

Thanks, I figured that it was like that. We can probably change \r\n to \n in files that we analyze. There is some risk of unintended consequences (like if \r\n was part of a character class in a regular expression), but if we only do it at the ends of lines, I doubt that anything will go wrong.

[jackfirth]

@blakelewis I think I came up with a better fix, so give it a try and let me know if the issue persists.

[blakelewis]

Sorry, dumb question: how do I pick up your change? 'raco pkg update' told me that there was no update. I tried 'raco pkg remove' and then 'raco pkg install', but I still have the old code.