How does decode method on Reader works?

[ArtyomResh]

Hello!
I'm pretty new to Rust, can someone please help me to understand how does decode method on Reader works?

Overall, I'm trying to to pass image from JS to Rust, process it and return back from Rust to JS.
My JS code look like this:

import { Image } from "image_processing";

const input = document.getElementById('image-input')
const image_canvas = document.getElementById('image-canvas')

input.addEventListener('change', async (event) => {
    const buffer = await event.target.files[0].arrayBuffer();
    const view = new Uint8Array(buffer);
    const image = Image.new(view);
    console.log('xxx1', view);

    const imageBlob = image.get_image_blob();
    console.log('xxx2', imageBlob);

    const blob = new Blob(imageBlob, { type: 'image/jpeg'});
    console.log('xxx3', blob);


    image_canvas.width = image.get_width();
    image_canvas.height = image.get_height();
    const imageBitmap = await createImageBitmap(blob, 0, 0, image_canvas.width, image_canvas.height);
    const context = image_canvas.getContext('2d');
    context.drawImage(imageBitmap, 0, 0);
});

And Rust Image.new and Image.get_image_blob functions looks like this:

#[wasm_bindgen]
impl Image {
    pub fn new(image_blob: Vec<u8>) -> Image {
        let reader = Reader::new(io::Cursor::new(image_blob))
            .with_guessed_format()
            .expect("Cursor io never fails");

        let extension = match reader.format() {
            Some(extension) => extension,
            None => panic!("Can't guess image extension")
        };

        let width;
        let height;
        let image = match reader.decode() {
            Ok(image) => {
                let (w, h) = image.dimensions();
                
                width = w;
                height = h;
  
                image
            },
            Err(error) => {
                panic!("Can't get image: {}", error)
            }
        };

        Image {
            image,
            width,
            height,
            extension,
        }
    }
    
    .
    .
    .

    pub fn get_image_blob(&self) -> Vec<u8> {
        match self.extension {
            x if x == ImageFormat::Png => self.image.to_rgba8().to_vec(),
            _ => self.image.to_rgb8().to_vec()
        }
    }
}

When I'm trying to load and return jpeg image in browser console I get this:

I was trying to find source of the problem and It's look like decode somehow changes blob content and I don't know why. Before I process my image with Rust it has right file signature for jpeg: [255, 216, 255, ...] and was 1,5 MB file. But after processing it became [162, 162, 152, ...] and became 15MB file. Can you help me and explain what I'm doing wrong?

Thank you!

[fintelia]

The decode method take a JPEG file and returns the raw pixels as an uncompressed block of memory. In other words, the [162, 162, 152...] means that the top right pixel has value red=162, green=162, and blue=152 or which corresponds to hex 0xA2A298.

[HeroicKatora]

Note that this line self.image.to_rgba8().to_vec() won't encode the image as png. In fact, DynamicImage won't remember the codec that has been used. Instead merely gives you can owned vector of the bytes making up the raw pixel matrix. In terms of DOM you may want to copy these bytes over to a properly sized ImageData.data array instead of trying to assign a file-blob.

[ArtyomResh]

Guys I'm very thankful you for your responses! I have been stuck with this for a week now. It's great that you let newbies like me ask questions about your library in issues!

@fintelia Thank you very much it helped me a lot! I'm pretty new to image processing too, so I'm wondering why decode working as you described. Is it because we need information of raw pixels values to do any operations with image? Is there a way to return compressed values so only 1 element of array represents color of 1 pixel?

@HeroicKatora Thank you, I didn't know about ImageData.data it is great tool! I think you save me another week of research :) I have tried ImageData.data and I noticed that it work perfectly with PNG images, but not with JPEG. JPEG images become bigger, because if i want them to render I should add useless information about alpha channel. What is the best practice to render JPEG image from a raw pixel data?

Can you give me a hint on where else I can ask questions and discuss with people image processing as whole and specifically in Rust and JS? I found that it's hard to find almost any information about this topic, I'm wondering it there any group or chat where people discuss such things?

[fintelia]

Is it because we need information of raw pixels values to do any operations with image?

Yes, exactly.

Is there a way to return compressed values so only 1 element of array represents color of 1 pixel?

That actually wouldn't help much. We return an array of u8's so each element actually takes only one byte, but it takes 4-bytes total to represent a pixel without losing information (one byte for each of the red, green, blue, and alpha channels). Technically we could return an array of 4-byte integers where each element stored a full pixel, but that wouldn't save any space...

[ArtyomResh]

That actually wouldn't help much. We return an array of u8's so each element actually takes only one byte, but it takes 4-bytes total to represent a pixel without losing information (one byte for each of the red, green, blue, and alpha channels). Technically we could return an array of 4-byte integers where each element stored a full pixel, but that wouldn't save any space...

@fintelia Yea, that makes sense. Thank you for that answer! But then can you, please, explain me why initial array (under xxx1 in logs) has different values and quantity of them in contrast to array that I get from decode (under xxx2 in logs)? First one isn't just raw pixels data? Or it's somehow has been optimised? Can I and should I optimise it before passing back to JS?

[fintelia]

The first array is a JPEG file represented as an array of bytes. Image files like JPEG consist of raw pixel data combined with metadata (things like the width and height) so that they can be easy exchanged over the internet or stored on disk. They usually also have some approach for compressing the pixel data so it takes up less space. In the case of JPEG, the technique used is quite complicated but luckily you can just use an implementation some else wrote.

The problem with image formats is that you generally cannot quickly extract individual pixel values from them. The same compression algorithms that shrink the file size also drastically increase the complexity of figuring out individual pixel values. Thus, the typical approach is to immediately decode image files when loading them (which as mentioned earlier converts them into raw arrays of pixels) and then do all further computation on them in decoded form until it is time to save/upload them somewhere in which case they can be encode'ed again and sent to their destination.

[ArtyomResh]

@fintelia Thank for your explanation! Am I right in thinking that to encode JPEG image I need to use image::codecs::jpeg::JpegEncoder.encode?

[fintelia]

@fintelia Thank for your explanation! Am I right in thinking that to encode JPEG image I need to use image::codecs::jpeg::JpegEncoder.encode?

Yep. Either that method, or a higher level one like save_buffer_with_format (which just calls into the encoder internally)

[ArtyomResh]

Thank you! I'm closing this issue, you helped me a lot!