Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] for testing : Fix 4Gb limit for large files on Git for Windows #2179

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

[DRAFT] for testing : Fix 4Gb limit for large files on Git for Windows #2179

wants to merge 14 commits into from

Commits on Jun 4, 2019

  1. zlib.c: use size_t for size 

    Signed-off-by: Martin Koegler <martin.koegler@chello.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Helped-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
    @e9925248 @dscho
    e9925248 authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    5b60f72 View commit details
    Browse the repository at this point in the history

  2. Use size_t instead of 'unsigned long' for data in memory 

    Currently the length of data which is stored in memory is stored
in "unsigned long" at many places in the code base.
This is OK when both "unsigned long" and size_t are 32 bits,
(and is OK when both are 64 bits).
On a 64 bit Windows system am "unsigned long" is 32 bit, and
that may be too short to measure the size of objects in memory,
a size_t is the natural choice.

Improve the code base in "small steps", as small as possible.
The smallest step seems to be much bigger than expected.
See this code-snippet from convert.c:
        const char *ret;
        unsigned long sz;
        void *data = read_blob_data_from_index(istate, path, &sz);
        ret = gather_convert_stats_ascii(data, sz);

The corrected version looks like this:
        const char *ret;
        size_t sz;
        void *data = read_blob_data_from_index(istate, path, &sz);
        ret = gather_convert_stats_ascii(data, sz);

However, when the Git code base is compiled with a compiler that
complains that "unsigned long" is different from size_t, we end
up in this huge patch, before the code base cleanly compiles.

And: there is more to be done in the zlib interface.

Reviewed-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
    @tboegi @dscho
    tboegi authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    9c2f8de View commit details
    Browse the repository at this point in the history

  3. Add test prereq for size_t being 64bit wide 

    Signed-off-by: Thomas Braun <thomas.braun@virtuell-zuhause.de>
    @t-b @dscho
    t-b authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    5554ca1 View commit details
    Browse the repository at this point in the history

  4. Add test for large files on windows 

    Original test by Thomas.
Add the extra fsck to get diagnostics after the add.
Verify the pack at earliest opportunity
Slight confusion as to why index-pack vs verify-pack...
It's -v (verbose) not --verify
Specify an output file to index-pack, otherwise it clashes with
the existing index file.
Check that the sha1 matches the existing value;-)

Signed-off-by: Thomas Braun <thomas.braun@virtuell-zuhause.de>
Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @t-b @dscho
    t-b authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    f05d8a6 View commit details
    Browse the repository at this point in the history

  5. t/helper/zlib-compile-flags: read zlib flags 

    zlib provides a convenience function that indicates the sizeOf
the standard types. Windows builds should test if their zlib
supports >4Gb address space for large file support.

Also update the t-large-files-on-windows.sh test script to
record the zlibFlags

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    fc00458 View commit details
    Browse the repository at this point in the history

  6. zlib.c,http_push.c: explicit cast comparisons of potential 32bit long… 

    … to size_t

On Windows, long may only be 32 bits and their use for pointer sized
comparison is potentially implemenation defined.
Make explicit the appropriate type conversion.

Ensure they are up-cast to size_t

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    a4b2bb5 View commit details
    Browse the repository at this point in the history

  7. zlib.c: refactor stream chuncking to fit uLong 32bit counting 

    On Windows uLong and size_t are different, being 32bit and 64bit
respectively. Computations of mixed 32/64 bit types can be
implementation defined leading to potential accuracy loss and error.

Avoid wraparound of z.total_in and z.total_in by always
starting at zero. The chunk size is kept well within 32bit limits.

Ensure the z.total_in and z.total_in are _upcast_ when computing the
overall avail_in and avail_out values

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    fb1a906 View commit details
    Browse the repository at this point in the history

  8. zlib.c, packfile.h,config.mak.uname: deflateBound size coercion 

    The zlib deflateBound() 'size' variable is 32 bits on Windows, but
64 bits on Linux and hence needs careful selection a-priori of
NO_DEFLATE_BOUND on Windows.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    2ea5193 View commit details
    Browse the repository at this point in the history

  9. index-pack.c: split bad object error messages 

    Tell the developer which condition failed.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    4bf4bda View commit details
    Browse the repository at this point in the history

  10. xcrc32(): create extended crc32 for 64 bit compatibility 

    On Windows, uInt/uLong are only 32 bits. Create a compatibility function.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    21d51c0 View commit details
    Browse the repository at this point in the history

  11. xcrc32: use xcrc32 wrapper 

    Zero length initialisations are not converted.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    8891566 View commit details
    Browse the repository at this point in the history

  12. builtin/index-pack.c,csum-file.c: use size_t for memsized variables 

    use size_t for Windows compatibility.
Also use appropriate format for printing.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    eeb79ff View commit details
    Browse the repository at this point in the history

  13. packfile.c: set shift limit to sizeof(size_t), not long 

    The code base now uses size_t for all memsized variables.
Allow shift to reach that bitness level.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    a8e0ee7 View commit details
    Browse the repository at this point in the history

  14. config.[ch]: provide config_size_t function and use it 

    For Windows compatibility.
cache.h: big_file_threshold & pack_size_limit_cfg are potentially size_t,
plus a few others convereted in this pass.

Other potential >4Gb variables are left for others.

Signed-off-by: Philip Oakley <philipoakley@iee.org>
    @dscho
    Philip Oakley authored and dscho committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    a85708c View commit details
    Browse the repository at this point in the history