Strange error while saving large BSON

[fipelle]

I am getting a InexactError: trunc(Int32, 8431957686) error while saving a large BSON. I was wondering if you know what could be causing it and how to get around it. Unfortunately, I am not sure how to replicate the error.

[devmotion]

Here are some examples that reproduce the error resulting from the calls of Int32 in

BSON.jl/src/write.jl

Line 14 in f66a639

bson_primitive(io::IO, x::Vector{UInt8}) = write(io, Int32(length(x)), 0x00, x)

,

BSON.jl/src/write.jl

Line 15 in f66a639

bson_primitive(io::IO, x::String) = write(io, Int32(sizeof(x)+1), x, 0x00)

, and

BSON.jl/src/write.jl

Line 32 in f66a639

write(io, Int32(length(bytes)+4), bytes)

:

julia> using BSON

julia> a = zeros(UInt8, 2147483634);

julia> BSON.@save "testa.bson" a # works

julia> b = zeros(UInt8, 2147483635);

julia> BSON.@save "testb.bson" b # fails due to call of `Int32` in line 32
ERROR: InexactError: trunc(Int32, 2147483648)                                                                       
Stacktrace:
 [1] throw_inexacterror(::Symbol, ::Type{Int32}, ::Int64) at ./boot.jl:557
 [2] checked_trunc_sint at ./boot.jl:579 [inlined]                                                                  
 [3] toInt32 at ./boot.jl:616 [inlined]                                                                             
 [4] Int32 at ./boot.jl:706 [inlined]                                                                               
 [5] bson_doc(::IOStream, ::Dict{Symbol,Any}) at /home/david/.julia/dev/BSON/src/write.jl:32
 ...

julia> c = zeros(UInt8, 2147483648);

julia> BSON.@save "testc.bson" c # fails due to call of `Int32` in line 14
ERROR: InexactError: trunc(Int32, 2147483648)
Stacktrace:
 [1] throw_inexacterror(::Symbol, ::Type{Int32}, ::Int64) at ./boot.jl:557
 [2] checked_trunc_sint at ./boot.jl:579 [inlined]
 [3] toInt32 at ./boot.jl:616 [inlined]
 [4] Int32 at ./boot.jl:706 [inlined]
 [5] bson_primitive at /home/david/.julia/dev/BSON/src/write.jl:14 [inlined]
 ...

julia> d = "0" ^ 2147483647;

julia> BSON.@save "testd.bson" d # fails due to call of `Int32` in line 15
ERROR: InexactError: trunc(Int32, 2147483648)
Stacktrace:
 [1] throw_inexacterror(::Symbol, ::Type{Int32}, ::Int64) at ./boot.jl:557
 [2] checked_trunc_sint at ./boot.jl:579 [inlined]
 [3] toInt32 at ./boot.jl:616 [inlined]
 [4] Int32 at ./boot.jl:706 [inlined]
 [5] bson_primitive at /home/david/.julia/dev/BSON/src/write.jl:15 [inlined]
 ...

The BSON specification requires the total number of bytes in a document, the number of bytes in binary data, and the number of bytes in a string (plus the trailing \x00) being specified with 4 bytes as signed 32bit integer (i.e., as Int32), so it seems the calls of Int32 in lines 14, 15, and 32 can't be avoided and it's an inherent limitation of the BSON file format that a document can contain at most typemax(Int32) = 2147483647 bytes. However, IMO more helpful error messages that actually explain this fact might be useful.