large file write causing errors

[jkthorne]

while running this code

data = Random::PCG32.new(1_u64).random_bytes(1_000_000_000)
File.write("static/data", data.to_s)

I get the error Negative size

Negative size (ArgumentError)
  from /usr/lib/crystal/pointer.cr:0:7 in 'realloc'
  from /usr/lib/crystal/string/builder.cr:124:5 in 'resize_to_capacity'
  from /usr/lib/crystal/string/builder.cr:45:7 in 'write'
  from /usr/lib/crystal/io.cr:487:7 in 'write_utf8'
  from /usr/lib/crystal/string.cr:4139:5 in 'to_s'
  from /usr/lib/crystal/io.cr:192:5 in '<<'
  from /usr/lib/crystal/slice.cr:669:7 in 'to_s'
  from /usr/lib/crystal/object.cr:96:7 in 'to_s'
  from temp.cr:2:27 in '__crystal_main'
  from /usr/lib/crystal/crystal/main.cr:11:3 in '_crystal_main'
  from /usr/lib/crystal/crystal/main.cr:112:5 in 'main_user_code'
  from /usr/lib/crystal/crystal/main.cr:101:7 in 'main'
  from /usr/lib/crystal/crystal/main.cr:135:3 in 'main'
  from __libc_start_main
  from _start

[ysbaddaden]

You overflow the Int32::MAX of sizes in Crystal. This is a duplicate of another issue (not validating sizes) but I can't find it.

[bew]

@ysbaddaden talking about #3209 ?

[ysbaddaden]

Yes. Thought 1 billion bytes is smaller than Int32::MAX, so an overflow is weird.

[RX14]

@ysbaddaden there's a data.to_s in there, which is both almost certainly wrong and also will take it far beyond 1GiB of data.

[jkthorne]

So if this is not at Int::MAX why is it overflowing?

[RX14]

@wontruefree it is > Int::MAX.

your data.to_s is like "Slice[123, 12, 240, 100 ... ]", which is a vastly longer and more useless representation than the raw slice. That's why I said it's "almost certainly wrong" meaning I really don't think you meant to_s.

[jkthorne]

@RX14 I was able to write write just the slice to the file and it worked fine. If a char is a byte how is writing a string more space then a slice of bytes?

[RX14]

@wontruefree because the string isn't binary. to_s on slice literally returns a string like "Slice[123, 12, 240, 100]" just the same as calling inspect.

[jkthorne]

I have been messing with this a little more I will have to read more about it.
Thanks