32-bit file size problem

[MagaTailor]

It seems anything over 2G causes an error:

 ls -l torcs-disk.img 
-rw-r--r-- 1 odroid odroid 2147483648 Nov  7 13:19 torcs-disk.img

exa -l torcs-disk.img 
torcs-disk.img: Value too large for defined data type (os error 75)

Sparse files trigger the error as well.

[ogham]

I need to get my hands on an ARM platform for testing these things! The len() method on Metadata returns u64, so I'm not sure where the problem lies...

[MagaTailor]

I'm getting the same error on 32-bit x86. So it's not arm specific at all.

[MagaTailor]

Judging from the error, the returned uint64 is being crammed into some int32 which is the word size on 32-bit obviously. Could there be some temporary variable somewhere?

BTW, maybe these compiler warnings are related? (coming from x86, rustc 1.5 nightly)

Compiling exa v0.4.0 (file:///home/petevine/exa-master)
src/feature/xattr.rs:180:59: 180:65 warning: found Rust type `usize` in foreign module, while `libc::c_uint` or `libc::c_ulong` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:180             path: *const c_char, list: *mut c_char, size: size_t
                                                                                   ^~~~~~
src/feature/xattr.rs:181:14: 181:21 warning: found Rust type `isize` in foreign module, while `libc::c_int` or `libc::c_long` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:181         ) -> ssize_t;
                                      ^~~~~~~
src/feature/xattr.rs:184:59: 184:65 warning: found Rust type `usize` in foreign module, while `libc::c_uint` or `libc::c_ulong` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:184             path: *const c_char, list: *mut c_char, size: size_t
                                                                                   ^~~~~~
src/feature/xattr.rs:185:14: 185:21 warning: found Rust type `isize` in foreign module, while `libc::c_int` or `libc::c_long` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:185         ) -> ssize_t;
                                      ^~~~~~~
src/feature/xattr.rs:189:39: 189:45 warning: found Rust type `usize` in foreign module, while `libc::c_uint` or `libc::c_ulong` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:189             value: *mut c_void, size: size_t
                                                               ^~~~~~
src/feature/xattr.rs:190:14: 190:21 warning: found Rust type `isize` in foreign module, while `libc::c_int` or `libc::c_long` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:190         ) -> ssize_t;
                                      ^~~~~~~
src/feature/xattr.rs:194:39: 194:45 warning: found Rust type `usize` in foreign module, while `libc::c_uint` or `libc::c_ulong` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:194             value: *mut c_void, size: size_t
                                                               ^~~~~~
src/feature/xattr.rs:195:14: 195:21 warning: found Rust type `isize` in foreign module, while `libc::c_int` or `libc::c_long` should be used, #[warn(improper_ctypes)] on by default
src/feature/xattr.rs:195         ) -> ssize_t;
                                      ^~~~~~~

It looks like something that might fit the description.

[ogham]

I set up a 32-bit Ubuntu VM and am able to reproduce the "Value too large for defined data type" error. However, I got no compiler warnings.

[ogham]

You were right! It was a temporary variable: self.numeric.format_int(n as isize). That method is in locale which I'll patch to make it accept different number sizes.

[MagaTailor]

Cool, thx!

About the warnings; never mind, that's just probably libstd from 1.5 coupled with the updated libc crate.

[ogham]

Unfortunately, even after fixing that temporary variable, the problem persists. What’s worse, it’s a Rust problem, rather than an exa problem. Even this program prints out false:

fn main() {
    use std::fs::metadata;
    use std::env::args;
    use std::path::Path;

    for arg in args().skip(1) {
        let m = metadata(Path::new(&arg));
        println!("{:?}", m.is_ok());
    }
}

The error isn’t being thrown in exa’s code, it’s in the actual io::Result that gets returned when we try to open a file. So I can’t even work around this in the code.

Some digging brought up this thread on Large File Support from last month (which I really wished I’d seen before, as it has almost my exact same test case failing). I can fix the variable in exa but it will still be stuck on this case until that gets implemented. :(

[MagaTailor]

rust-lang/rust#28978

[cuviper]

You could work around this by using libc::stat64 etc. directly until libstd is fixed.

[MagaTailor]

I've just confirmed the relevant changes do work as expected:

$ ./exa -l bigfile
.rw-r--r-- 3,4G guest 15 Feb  3:58 bigfile