Rendering the image from the clipboard

[FarazPatankar]

Hey y'all, Rust newbie here. For context, I am building a desktop app with Tauri and using arboard to read/write images from/to the clipboard. The writing part works fine but the part I am struggling with is taking the image I read from the clipboard and rendering it on the client so the user can view and edit the image.

Things I've tried in JS:

• Converting the image.bytes into a blob/base64 string.
• Rendering the image.bytes into a canvas.

Things I've tried in Rust:

• Writing the image.bytes to a file using this approach.

In Rust, the file gets saved by when I open it, I just see this.

I think the issue is probably with my understanding of what image.bytes returns and how to use it to actually convert it into a PNG or any other format that I can use to display the image. I know the image itself is in the form of this ImageData struct but I don't really understand much beyond that.

Could someone help walk me through this or give me some pointers so I can further experiment with implementing this feature within my app?

Additional context: My dev environment is MacOS 12.4 and my frontend is written in React.

[ArturKovacs]

I think you should use the Rust ImageData bytes to construct a new JS ImageData: https://developer.mozilla.org/en-US/docs/Web/API/ImageData/ImageData Then pass the JS ImageData to the putImageData function of the canvas: https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/putImageData

…

On Wed, 6 Jul 2022 at 08:54, Faraz Patankar ***@***.***> wrote: Hey y'all, Rust newbie here. For context, I am building a desktop app with Tauri <https://tauri.app/> and using arboard to read/write images from/to the clipboard. The writing part works fine but the part I am struggling with is taking the image I read from the clipboard and rendering it on the client so the user can view and edit the image. Things I've tried in JS: - Converting the image.bytes into a blob/base64 string. - Rendering the image.bytes into a canvas. Things I've tried in Rust: - Writing the image.bytes to a file using this approach <https://www.reddit.com/r/rust/comments/ooloe7/how_to_grab_an_image_from_the_clipboard_and_copy/> . In Rust, the file gets saved by when I open it, I just see this <https://user-images.githubusercontent.com/10681116/177487082-0ed2ef3f-be4c-4586-b5aa-36611734155b.png> . I think the issue is probably with my understanding of what image.bytes returns and how to use it to actually convert it into a PNG or any other format that I can use to display the image. I know the image itself is in the form of this ImageData <https://docs.rs/arboard/2.1.1/arboard/struct.ImageData.html> struct but I don't really understand much beyond that. Could someone help walk me through this or give me some pointers so I can further experiment with implementing this feature within my app? Additional context: My dev environment is MacOS 12.4 and my frontend is written in React. — Reply to this email directly, view it on GitHub <#62>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7PKCBCFFEH52C4IEKB2L3VSUUUFANCNFSM52YSYJIA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[FarazPatankar]

Thanks for the response, @ArturKovacs. I believe I have tried doing what you're suggesting but to no avail. Any chance you could share a code example?

Here's what I've tried to do in JS:

export async function bytesToBlob(
  Image: Image,
): Promise<{ blob: Blob; src: string }> {
  const canvas = document.createElement("canvas");
  const context = canvas.getContext("2d")!;
  canvas.width = Image.width;
  canvas.height = Image.height;
  const imgData = context.createImageData(Image.width, Image.height);
  imgData.data.set(Image.bytes);

  context.putImageData(imgData, 10, 10);

  return new Promise(resolve => {
    canvas.toBlob(file =>
      resolve({
        blob: file!,
        src: URL.createObjectURL(file!),
      }),
    );
  });
}

[ArturKovacs]

First of all, you should decide where you want to render it.

1. If you want to render it to the screen, then you should have a <canvas> element on your frontend and use putImageData of that canvas. If you are using React; you can obtain a reference to the <canvas> with the ref property
2. If you want to render it to a file, then I would NOT use the canvas to do that, instead I would use the image crate in Rust code to render the pixels to a compressed image file.

[FarazPatankar]

So the code I shared just happened to be the last thing I tried. I have tried rendering it to a <canvas> which results in the same image I have shared above.

I have also tried rendering the image to a file using the image crate. Here's the code I ended up with based on the Reddit thread I've shared in the original question:

fn get_image_from_clipboard() {
  let mut ctx = Clipboard::new().unwrap();
    let image = ctx.get_image().unwrap();
    let imgbuf = ImageBuffer::from_raw(
        image.width as u32,
        image.height as u32,
        image.bytes.into_owned(),
    )
    .unwrap();
    let imgbuf = DynamicImage::ImageRgba8(imgbuf);
    imgbuf.save("clipboard-image.png");
}

My end goal is to display the image to the user within the UI so they can edit it. I tried saving the image to file after I couldn't get it to render with JS hoping to read that file on the client but that approach gives me the same output as well.

[ArturKovacs]

I have a feeling that you are copying the file and not the pixel values. The two things are not the same. Note that arboard does not currently support copying and pasting files. Here are two example situation where the user would copy the pixel values (which is supported by arboard)

• When you right click on an image in a browser and then click on "Copy image" (works in Firefox and Chrome)
• When you select an area of an image in an image editor software and press Cmd+C

---

The Rust code you pasted above should work correctly. So if that doesn't work, then the problem might be somewhere else. (Again, maybe you are trying to copy the file)

If I were in your place, I would create a 1 pixel image where you know the exact RGB values of the pixel (with a GIMP or something like that), and copy that image to the clipboard, and then use get_image and print the bytes. If that seems to be correct, then do the same with a 2x2 image where you know the exact pixel values.

For example the 1 pixel image where the pixel is red should have the following bytes:

[255, 0, 0, 255]

[FarazPatankar]

Okay, that makes a lot of sense. I was indeed copying the file as an easy way to test the implementation when in reality, I'd imagine if the user has the file saved, they'd rather drag-n-drop or open the file through the file select UI I have provided.

• Is there a way in which I could detect whether the user is making the same mistake and notify them?
• In your experience, which of the two approaches would be better to implement?
  • Sending the image data to the client, rendering a canvas, and converting it to an image.
  • Saving the file to disk in Rust, reading the file on the client, and then deleting the file.

Edit: Lastly, just to confirm, is there an easier way to render this image on the client that I haven't thought of and is different from the two options I have listed above? Perhaps something that does not involve using the canvas?

[ArturKovacs]

Is there a way in which I could detect whether the user is making the same mistake and notify them?

Even if there is, it wouldn't be a good way of dealing with this problem. The optimal way of dealing with this is to allow users to paste/drag-n-drop files. And actually both the drag-n-drop and the clipboard is the responsibility of that part of the code which manages the windows and input. So in your case handling the clipboard and drag-n-drop should be a feature of Tauri. So I suggest that you open an issue for them if this is not supported by Tauri.

(This also means that optimally, arboard should not be used with graphical applications, because optimally all Rust windowing crates should have a build-in way to manage the clipboard. See also: this PR for winit)

In your experience, which of the two approaches would be better to implement?

This one: "Sending the image data to the client, rendering a canvas, and converting it to an image."

You should not write to the disk unless you want information to be preserved after the application is closed.

Lastly, just to confirm, is there an easier way to render this image on the client that I haven't thought of and is different from the two options I have listed above? Perhaps something that does not involve using the canvas?

If you want to allow the user to edit the image, then I think the best approach is to use the canvas and try to avoid file operations as much as possible.

[charlzyx]

So the code I shared just happened to be the last thing I tried. I have tried rendering it to a <canvas> which results in the same image I have shared above.

I have also tried rendering the image to a file using the image crate. Here's the code I ended up with based on the Reddit thread I've shared in the original question:

fn get_image_from_clipboard() {
  let mut ctx = Clipboard::new().unwrap();
    let image = ctx.get_image().unwrap();
    let imgbuf = ImageBuffer::from_raw(
        image.width as u32,
        image.height as u32,
        image.bytes.into_owned(),
    )
    .unwrap();
    let imgbuf = DynamicImage::ImageRgba8(imgbuf);
    imgbuf.save("clipboard-image.png");
}

My end goal is to display the image to the user within the UI so they can edit it. I tried saving the image to file after I couldn't get it to render with JS hoping to read that file on the client but that approach gives me the same output as well.

i replace from_raw -> from_vec and workd on macos

fn get_image_from_clipboard() {
  let mut ctx = Clipboard::new().unwrap();
    let image = ctx.get_image().unwrap();
    let imgbuf = ImageBuffer::from_vec(
        image.width as u32,
        image.height as u32,
        image.bytes.into_owned(),
    )
    .unwrap();
    let imgbuf = DynamicImage::ImageRgba8(imgbuf);
    imgbuf.save("clipboard-image.png");
}

[complexspaces]

Since it looks like this was figured out and a working solution was found, I'm going to close this in an effort to keep the issue tracker tidy.