A way to convert/load image from multipart to tensor

[isdzulqor]

Is there a way to directly convert an image from multipart file instead of from path to tensor?

I wanna proceed with the image sent from Rest API to be inferenced.

I only found load from the path which is a string format

vision.Load(path)
imageNet.LoadImage(path)

I'm assuming that writing a multipart file to disk first will be so much costly

[sugarme]

HI @isdzulqor,

Thanks for pointing out and it would be nice to have one API to load image from memory to tensor.
I will create an issue so that we find out time to implement this feature. Of course you are very welcomed to contribute.
With input is image data buf there're 2 options:

• Go: create tensor from data tensor.FromData, then do anything from there (note that API tensor.FromData has NOT been optimized yet in terms performance)
• C++: we use stb_image to handle image in C++. So the idea is handle Go buff data to C++ where C++ using stb_image API and libtorch to create tensor.

If you want to try it on your own project, first one would be more feasible, I guess. Let me know how you go. Thanks.

[isdzulqor]

i'm trying option 1, implement it in the handler like this

func PredictionBuf(w http.ResponseWriter, r *http.Request) {
	contentType := r.Header.Get("Content-Type")
	defer r.Body.Close()

	bodyBytes, err := ioutil.ReadAll(r.Body)
	if err != nil {
		log.Fatal(err)
	}
	buf := bytes.NewBuffer(bodyBytes)

	imageVector := getVectorFromTensor(ts.TensorFrom(buf))
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(imageVector)
}

ts.TensorFrom(buf) returns this error

2021/07/25 10:27:21 Input Data: unsupported data structure or type: ptr

do you have any idea @sugarme ?

[sugarme]

Hi @isdzulqor

Here is a round trip sample. You can adapt it to your own code:

package main

import (
	"fmt"
	"image/png"
	"io"
	"log"
	"os"

	"github.com/sugarme/gotch"
	ts "github.com/sugarme/gotch/tensor"
	"github.com/sugarme/gotch/vision"
)

func main() {
	filename := "./bb.png"

	imgTs, err := vision.Load(filename)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%i\n", imgTs)

	f, err := os.Open(filename)
	if err != nil {
		log.Fatal(err)
	}

	data, shape, err := getImageData(f)
	if err != nil {
		log.Fatal(err)
	}

	x, err := ts.OfDataSize(data, shape, gotch.Uint8)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%i\n", x)

	err = vision.Save(x, "new-bb.png")
	if err != nil {
		log.Fatal(err)
	}
}

func getImageData(r io.Reader) (data []uint8, shape []int64, err error) {
	img, err := png.Decode(r)

	if err != nil {
		return nil, nil, err
	}

	bounds := img.Bounds()
	width, height := bounds.Max.X, bounds.Max.Y
	shape = []int64{3, int64(height), int64(width)}

	var c1, c2, c3 []uint8
	for y := 0; y < height; y++ {
		for x := 0; x < width; x++ {
			r, g, b, _ := img.At(x, y).RGBA()
			c1 = append(c1, uint8(r/257))
			c2 = append(c2, uint8(g/257))
			c3 = append(c3, uint8(b/257))
		}
	}

	data = append(data, c1...)
	data = append(data, c2...)
	data = append(data, c3...)

	return data, shape, nil
}

Let me know how you go. If everything okay, I will create a new API .

[isdzulqor]

@sugarme

very nice. your code is working in my case.

but somehow the performance for using io.Reader is a bit slower. I'm comparing in request hit. not going deeper to compare function by function.