Image data 1D-array retrieved from get_object less than number of image pixels

[yichong96]

I have written a simple program that allows uploading and reading of image data to minio. When I upload the image to minio, I save the height and width as metadata. However, when I call get_object on the same image and try to form it back to the original image array, I am not able to do so because the number of bytes read from get_object is less than the number of image pixels.

    from minio import Minio
    from minio.error import S3Error
    import json
    import numpy as np
    import argparse
    import glob
    from PIL import Image


    def main():
        client = Minio("localhost:9000",
                access_key="XXX",
                secret_key="XXX",
                secure=False)
    
    parser = argparse.ArgumentParser()
    parser.add_argument("image_filepath", type=str)
    parser.add_argument("-b", "--bucket_name", type=str)
    parser.add_argument("-i", "--operation", required=True)
    FLAGS = parser.parse_args()

    # check if bucket exist
    myBucket = FLAGS.bucket_name
    found = client.bucket_exists(myBucket)
    if found:
        print("Found: " + myBucket)
    else:
        client.make_bucket(myBucket)
        print("Creating " + myBucket +  " bucket")
        
    if FLAGS.operation == "read":
        objects = client.list_objects(myBucket)
        data = []
        for obj in objects:
            print(obj.object_name)
            response = client.get_object(myBucket, obj.object_name)
            headers = response.getheaders()
            data = response.read() # byte array has length <<<< image array unrolled 
            deserialized_bytes = np.frombuffer(data, dtype=np.int8)
            height, width = headers["X-Amz-Meta-Height"], headers["X-Amz-Meta-Width"]
            print(height, width)
            np_image = np.reshape(deserialized_bytes, newshape=(3, int(height), int(width))) # Error here
    
    if FLAGS.operation == "write":
        # upload operation
        count = 0
        for file in glob.glob(FLAGS.image_filepath + "/*.jpg"):
            image = Image.open(file)
            width, height = image.size
            metadata = {"height": height, "width": width}
            object_name = file.split("/")[-1]
            print(file)
            result = client.fput_object(FLAGS.bucket_name, object_name, file, metadata=metadata)
            print("object name added: {}".format(result.object_name))
            count += 1
            
if __name__ == "__main__":
    main()

For example, when I uploaded the image from file, the shape of the image was 64x128 (W,H). However, when I read the same image using get_object the length of the data variable was only 2418 bytes, which is far from the expected 64*128*3 bytes.

Not sure what could be the problem ?

[balamurugana]

Use response like https://github.com/minio/minio-py/blob/master/minio/api.py#L1065

[yichong96]

I am still unable to read the bytes object into the required numpy array. It is strange that the bytes object written to file gives back the original image. However, I don't know why the code below does not work. Not sure if np.from_buffer is expecting something else.

Not an issue with the streaming amount because one pass was enough to get all the data as it was < 1024*1024

    if FLAGS.operation == "read":
        objects = client.list_objects(myBucket)
        for obj in objects:
            print(obj.object_name)
            try: 
                response = client.get_object(myBucket, obj.object_name)
                headers = response.getheaders()
                bytes = b''
                height, width = headers["X-Amz-Meta-Height"], headers["X-Amz-Meta-Width"]
                for data in response.stream(amt=1024*1024):
                    deserialized_bytes = np.frombuffer(data, dtype=np.int8) # bytes still less than image size
                    np_image = np.reshape(deserialized_bytes, newshape=(3, int(height), int(width)))
                    print(np_image.shape)
            finally:
                if response:
                    response.close()
                    response.release_conn()

[bpedersen2]

You are writing a jpeg file (jpeg uses lossfull compression!) and you try to read the compressed data as an uncompressed array.
You need to read it in as a jpeg image in that case.

Or you would need to write the uncompressed data in the first place, so the result of numpy.save(image).

[yichong96]

@bpedersen2 thanks for explanation ! it seems that I would have to write the data into a file first before reading it back in again.