Can't read content > 4GB with Content Receiver on 32bit platform

[GEEKiDoS]

I'm using the client

[GEEKiDoS]

seems like the size_t issue, modified some of variable types to uint64_t works fine

• read_len in read_content_with_length
• payload_max_length on read_content
• std::numeric_limits<size_t> to std::numeric_limits<uint64_t> in detail::read_content call

[falbrechtskirchinger]

Some other read functions already use uint64_t, FYI. Seems like an oversight.

Edit: Looking more closely, I'm unsure how to square storing the body in std::string, which imposes a very real limit on 32-bit systems, with the content receiver API, which is not necessarily subject to the same constraints. IMO, it would make sense for the content receiver API to bypass max payload length. Also, max payload length should be set (and clamped) to std::string::max_size() – not std::numeric_limits<size_t>::max() – as those values differ on all implementations I'm familiar with.
Alternatively, max payload length needs to be changed to uint64_t, and when the destination of the body is a std::string, we clamp it to std::string::max_size(). Maybe that's the least surprising choice.

[yhirose]

@GEEKiDoS @falbrechtskirchinger thanks for the report and investigation!

@GEEKiDoS Since I don't have a 32bit machine, I am not able to reproduce and confirm this issue at this point. Could you make a Unit test case in test/test.cc which can reproduce this issue, and send me a pull request including the test case? Meanwhile, I'll try to find an easy way to build a 32bit executable and run it on 64 bit Ubuntu on my 64 bit CPU machine. Thanks a lot!

[falbrechtskirchinger]

Meanwhile, I'll try to find an easy way to build a 32bit executable and run it on 64 bit Ubuntu on my 64 bit CPU machine. Thanks a lot!

That's as easy as adding -m32. I have some experience with this 32-bit and size_t issue from nlohmann/json. I implemented the 32-bit unit tests in nlohmann/json#3523.

I did start work on a PR but quickly gave up after realizing what other problems this issue reveals:

• Only one compressor can handle >4GB on 32-bit without modification.
• File handling is severely limited without memory-mapped files (1GB on some implementations). Memory-mapped files are capped by the address space, which can be worked around with offsets. (Edit: read_file() is only used by test.cc)
• Some places check std::string::max_size() but not all. None set the proper status code.

It's more realistic to document the limitations of 32-bit systems first and then address some of the low-hanging fruit. Triggering all issues in unit tests may not be possible due to RAM limits.

If I find the time later, I'll work on documentation and some Makefile changes to allow local testing.

[yhirose]

@falbrechtskirchinger @GEEKiDoS Any updates on this issue? If it's too challenging to fix, I am considering officially dropping support for the 32-bit environment. Since I only use this library in 64-bit environments, maintaining support for 32-bit environments may no longer be necessary. This would also reduce the extra maintenance workload. (Please note that the development and maintenance of this library are carried out during my personal time. 😄)

[GEEKiDoS]

@falbrechtskirchinger @GEEKiDoS Any updates on this issue? If it's too challenging to fix, I am considering officially dropping support for the 32-bit environment. Since I only use this library in 64-bit environments, maintaining support for 32-bit environments may no longer be necessary. This would also reduce the extra maintenance workload. (Please note that the development and maintenance of this library are carried out during my personal time. 😄)

tbh there's really not much 32bit environments these days, The only mainstream 32bit platform leftover is Micro-controllers and maybe 32bit processes on Wow64

I think it's safe to drop the support