Data.Text.Array.new: size overflow

[amesgen]

Sorry for this report being very non-minimally reproducible, please don't feel obligated to act on this in any way.

We are using hs-pcre2 to extract some information from the Hackage Hoogle database (500MB, 70k files): relevant function, and see this script for how to execute the app (this is Nix-based and hence fully reproducible, but it should be rather easily adaptable to not rely on Nix).

At some point (it still worked in May), the app started to crash after processing a few thousand files:

extract-hackage-info: Data.Text.Array.new: size overflow
CallStack (from HasCallStack):
  error, called at src/Data/Text/Array.hs:82:20 in text-1.2.5.0-B336hruu8LkEc19BqHCzAw:Data.Text.Array

This happens both with hs-pcre2 2.1.1.1 and 2.2.1 (so text-1.2 and text-2.0), but the error message for the latter goes like this:

extract-hackage-info: Out of memory

I replaced hs-pcre2 by megaparsec in tweag/ormolu#958, which makes the error go away, so we are not blocked on this and I didn't investigate further what exactly is going on with hs-pcre2. But maybe this is still interesting to you or is ringing some bell.

[sjshuck]

Thanks for the report.

I have no intuition about what the issue is. Maybe dig into uses of Text.fromPtr, Text.copy, Text.dropWhile8, Text.useAsPtr, etc.

That's great you got use out of the library for Ormolu, at least for a little while! ❤️

[sjshuck]

https://github.com/haskell/text/blob/2.0.1/src/Data/Text/Array.hs#L75
Just a remark: It's funny that Data.Text.Array.new takes a signed Int for size, and calls it "overflow" when it's negative.