Repository navigation
Error for binaries larger than 2Gb #3939
Description
For final binaries larger than 2Gb, the following exception message is printed out:
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/archive/writers.py#L264
class CTOC(object):
ENTRYSTRUCT = '!iiiiBB' # (structlen, dpos, dlen, ulen, flag, typcd) followed by name
Activity
htgoebel commentedon Jan 2, 2019
A possible solution would be to change this from signed to unigned values (need to be done in the bootloader, too).
Can you please provide a test-case we can include into the test-suite. Thanks.
lsoica commentedon Jan 7, 2019
The switch from signed to unsigned moves the limit from 2gb to 4gb, right ? Is there a way to go over the 4gb limit ? Basically, could we go with an 8byte specifier ?
As for a test case, do you need a specific format or just a set of steps ?
htgoebel commentedon Jan 7, 2019
Right.
Basically we could be, but this required more changes. Also we should keep in mind 32-bit platforms, which might have trouble using 8-byte specifiers.
Well, this should fit into our test-quite, which uses py-test. The most problematic point is to generate same data which surly ends up to be > 4gb when creating the CTOC.
I suggest several test-cases, which (as I assume) also ease developing tests:
tests/unit/.This test could create a huge file containing some known data at some relevant positions (e.g. start, just before 2GB, just after 2GB, just before 4GB, just after 4GB), pass it using
pyi_args=["--add-data", "xxx:yyy"]and the code should verify the expected data can be read.Shall go into
tests/function/test_basic.py.Kenblair1226 commentedon Sep 17, 2019
Sorry this might be a dumb question, but how to do it? Change '!iiiiBB' to '!IIIIBB' ?
how to to this for bootloader for they are all binary files?
Legorooj commentedon Feb 26, 2020
Pinging
Red-Eyed commentedon Jul 10, 2020
Any news?
I tried to change datastructure but have no luck
Legorooj commentedon Jul 13, 2020
@Red-Eyed I've added this to my todo-list, and I'll get this in the 4.1 release, maybe even the 4.0 release depending on when that actually gets released.
sshuair commentedon Nov 9, 2020
@Legorooj is this fixed now?
Red-Eyed commentedon Nov 9, 2020
@Legorooj sorry for the annoying comments, but this issue is important to me: Currently I have to do my installer with additional tarballs rather than having just a single large executable.
So, If you don't mind, I just want to ask a few questions in order to have a better understanding the status of this issue:
Thanks!
Legorooj commentedon Nov 9, 2020
Ah right. Let me take a lok at this right now - my apologies, I completely forgot about this.
bwoodsend commentedon Nov 9, 2020
I question the usefulness of this. It'd take a good 30 minutes to pack a 4GB application then a good minute or so to unpack again followed by however long it takes for your antivirus to give it a good sniff which would have to happen every time the user opens the application. I already find ~200MB applications to be annoyingly slow to startup. I think you'd be better off turning your one-dir applications into an installable bundle using NSIS or something equivalent. That way your user only has to unpack it once.
16 remaining items
sshuair commentedon Nov 11, 2020
Yes! In my case, the
libtorch_cuda.sois 900+MB. The package will larger than 2 GB if you add thecuda,cudnnandTensorRTlibrary.Red-Eyed commentedon Nov 11, 2020
Currently I have 6 GB of python environment (pytorch, tensorflow, scipy etc) and pack it into 1.6GB tar.xz archive to get under 2GB limit
but I want to use zstandard compression (to speed up decompression) but zstandart compresses to 2.6GB which is above of 2GB limit
Red-Eyed commentedon Nov 11, 2020
Note: I just built bootloader and it works, thank you @rokm !
rokm commentedon Nov 11, 2020
Here's a further branch, in which 64-bit integers are used: https://github.com/rokm/pyinstaller/tree/large-file-support-v2
So now in theory, sky's the limit - you can chuck in all your deep learning frameworks, CUDA libraries, pretrained models, ...
In practice, however, the 5 GB onefile test passes only on linux (tested only 64-bit for now). Windows (even 64-bit) do not seem to support executables larger than 4 GB. On (64-bit) macOS, the
macholibused in a signing preparation step in the assembly pipeline seems to assume that the file size can be represented with an unsigned integer, so 4 GB max as well (and this is consistent with macOS signature searching code in the bootloader, which uses 32-bit unsigned integers).So really huge
onefileexecutables (> 4 GB) work only on linux. But on all three OSes, this limitation can be worked around by using .spec file and addingappend_pkg=Falseas an extra argument toEXE(). This will give you a small executable (e.g.,program) and a single large archive file to go with it (e.g.,program.pkg).sshuair commentedon Nov 18, 2020
@Legorooj any progress?
Legorooj commentedon Nov 18, 2020
I stopped work on this because by the time I woke up the next morning @rokm had already written everything that needed to be😅. Maybe he could submit a PR?
sshuair commentedon Nov 18, 2020
Nice job!
rokm commentedon Nov 18, 2020
@sshuair the current plan is to have the changes from those experimental branches submitted and merged gradually - first the endian-handling cleanup, then the file extraction cleanup, then switch to unsigned 32-bit ints, and finally to 64-bit ones.
In the meantime, if you require this functionality, you can use either of the experimental branches linked above.
Red-Eyed commentedon Nov 18, 2020
@sshuair
If u need >2GB just use branch that @rokm published. I just use it and it works on Ubuntu 20.04 and Windows (but before using it, u need to rebuild the bootloader, it's easy, just read the documentation)
sshuair commentedon Nov 20, 2020
@Red-Eyed the branch that @rokm published seems not work for me. When the file
PKG-00.pkgup to 2.7BG, the error appeared again. My OS system is Ubuntu 16.04.sshuair commentedon Nov 20, 2020
@Red-Eyed Sorry~~It's my fault, I install the package from
masterbranch but not thelarge-file-support-v2. Now it works for me.EricPengShuai commentedon Sep 14, 2022
@rokm so how to solve the error problem for binaries larger than 4G, my OS system is CentOS Linux release 7.6.1810 (Core)
the final error information is as follows.
rokm commentedon Sep 14, 2022
You cannot. If you wanted to generate embedded archive that's larger than 4 GB, you would need to switch the types used by corresponding PyInstaller code (both python and C) to 64-bit integers. While this would work on linux, neither Windows nor macOS support executables larger than 4 GB, so we only extended max size from 2 GB to 4 GB by switching to unsigned 32-bit integers.