numpy.fromfile() byte offset

[danielhogg]

A major use case for the numpy.fromfile() function is in reading from binary file of some kind. Frequently, binary formats are just arrays of data with a fixed-size header. It would be useful to have an extra optional parameter for the fromfile() function which defines a byte offset for when to start reading. This parameter would default to 0, leaving it backwards-compatible with existing code.

Reproducing code example:

import numpy as np

binary_file = "sample_binary.bin"
my_array = np.fromfile(binary_file, dtype=np.uint8, offset=40)

Currently, the above code will not run, as the 'offset' keyword argument is not supported. In this call, the ndarray object 'my_array' would be the result of the binary file being read in, starting with an offset of 40 bytes.

It is certainly possible to work-around the issue currently, but adding direct support for offsets in the function would make for cleaner code.

[danielhrisca]

@danielhogg
A simple work-around is to manually seek into the file at the required offset

import numpy as np

binary_file = "sample_binary.bin"
with open(binary_file, 'rb') as f:
    f.seek(40)
    my_array = np.fromfile(f, dtype=np.uint8)

Edit: having an offset argument would allow for more explicit control

[mhvk]

This seems like a good idea. I think the default should be None, i.e., the file pointer will not be changed (and is thus at the start when one passes in a string, and wherever a filehandle is when that is passed in).

I think it would involve relatively simple changes in core/src/multiarray/multiarraymodule.c - likely just after creating a duplicate filehandle - this part already is tracking where the original starting position is, and using that position in the actual reading.

One possible complication: for binary files, the offset is very obvious, but this is less true for text files. One may want to allow it only for binary files, and refer to loadtxt for text files.

[danielhrisca]

@danielhogg

🤦‍♂️

It is certainly possible to work-around the issue currently, but adding direct support for offsets in the function would make for cleaner code.

I was too tired last night and completely missed your example.