BLS test vectors

[tdammers]

Integrate Rust test vectors with the Haskell BLS code and test suite.

[lehins]

Looks good. Just a few comments

[lehins]

It's quite peculiar.
@tdammers and @iquerejeta do you guys know why are the tests failing on Windows?
In particular, is it a problem with tests or the actual implementation?

[tdammers]

@tdammers and @iquerejeta do you guys know why are the tests failing on Windows?
In particular, is it a problem with tests or the actual implementation?

I think I know.

The error originates in Base16.decode, specifically here; it occurs when the length of the bytestring to be decoded is not even (which it has to be because each byte encodes to two hex digits).

And the reason it is uneven on Windows is because (as per review comment) we now use ByteString IO instead of String IO and lines from Data.ByteString.Char8 instead of the Prelude one on String. My hypothesis is that it goes like this:

• git on the Windows CI machines is configured to automatically convert line endings on checkout (I think that's the default on Windows), so the hex files have \r\n line endings on Windows CI, but plain \n on Linux and OSX.
• We then open the file in binary mode (Data.ByteString.readFile; previously this was System.IO.readFile, which uses text mode), so the \r is retained.
• Data.ByteString.Char8.lines then splits on \n; it does not consider \r to be a newline character, and does not strip it away.
• The bytestring we feed to Data.ByteString.Base16.decode now has a trailing \r attached to it on Windows (but not on the other OSes, where git does not inject \r, because plain \n is the native line ending convention).

The quick-and-dirty solution: just cull the \r characters, e.g. using BS8.filter (/= '\r').

The morally correct, but more elaborate and potentially less performant solution: read the file into a Data.Text (this will use openFile, not openBinaryFile), line-split (still on Text), UTF8-encode each line into a ByteString, and then do the hex decoding.

[lehins]

@tdammers Oh yeah, Windows newline strikes again. I always forget about it. In that case we can go back to using String for reading and splitting on newlines: map BS8.pack . Prelude.lines <$> Prelude.readFile f. Hex decoding still needs to be done on ByteStrings.

line-split (still on Text),

I thought about the same thing, but unfortunately text package also ignores the CR character, from haddock: "lines does not treat '\r' (CR, carriage return) as a newline character."

[tdammers]

@tdammers Oh yeah, Windows newline strikes again. I always forget about it. In that case we can go back to using String for reading and splitting on newlines: map BS8.pack . Prelude.lines <$> Prelude.readFile f. Hex decoding still needs to be done on ByteStrings.

line-split (still on Text),

I thought about the same thing, but unfortunately text package also ignores the CR character, from haddock: "lines does not treat '\r' (CR, carriage return) as a newline character."

Indeed it does not, but Data.Text.IO.readFile converts newlines as per the system locale, which is exactly what we want in this case. Then again, the documentation for that also suggests to use decodeUtf8 . Data.ByteString.readFile instead, for performance reasons, conveniently failing to mention that this changes the behavior to no longer perform any line ending conversions...

Bottom line, just stripping \r like I did in the last commit is probably the sanest solution here.

[lehins]

LGTM

[lehins]

Bottom line, just stripping \r like I did in the last commit is probably the sanest solution here.

Yeah, sounds reasonable.