Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement screen capture, save, open, and search. #13

Open
octalmage opened this issue Feb 4, 2015 · 30 comments
Open

Implement screen capture, save, open, and search. #13

octalmage opened this issue Feb 4, 2015 · 30 comments

Comments

@octalmage
Copy link
Owner

@octalmage octalmage commented Feb 4, 2015

Just an issue to track progress and ideas.

@octalmage
Copy link
Owner Author

@octalmage octalmage commented Feb 19, 2015

For the bitmap functions, I'll return an object similar to this:

MMBitmapRef createMMBitmap(uint8_t *buffer,
                           size_t width,
                           size_t height,
                           size_t bytewidth,
                           uint8_t bitsPerPixel,
                           uint8_t bytesPerPixel)
{
    MMBitmapRef bitmap = malloc(sizeof(MMBitmap));
    if (bitmap == NULL) return NULL;

    bitmap->imageBuffer = buffer;
    bitmap->width = width;
    bitmap->height = height;
    bitmap->bytewidth = bytewidth;
    bitmap->bitsPerPixel = bitsPerPixel;
    bitmap->bytesPerPixel = bytesPerPixel;

    return bitmap;
}

This way we can pass the necessary

@octalmage
Copy link
Owner Author

@octalmage octalmage commented Feb 19, 2015

With the new syntax this will go in the index.js.

function bitmap (imagedata, width, height) 
{
    this.imagedata = imagedata;
    this.width = width;
    this.height. = height;
}

bitmap.prototype.save = function(path) 
{
    robotjs.saveBitmap(this.imagedata, path);
};

module.exports.screen.capture = function()
{
    b = robotjs.captureScreen();
    return new bitmap(b.imagedata, b.width, b.height);
}

Then you'll use it like this:

var robot = require("robotjs");

var desktop = robot.screen.capture();

desktop.save("screencapture.png");

Note: I wrote this on my phone.

@octalmage
Copy link
Owner Author

@octalmage octalmage commented Feb 19, 2015

Might need to use:

createStringFromMMBitmap
createMMBitmapFromString

To pass the image between V8 and JavaScript.

octalmage added a commit that referenced this issue Mar 29, 2015
@octalmage octalmage self-assigned this Jul 23, 2015
@octalmage octalmage removed their assignment Jul 30, 2015
@Deltatiger
Copy link
Collaborator

@Deltatiger Deltatiger commented Aug 25, 2015

This should be the next priority. Capture screen will allow for more projects to adopt RobotJS. (Also my mini project of a remote desktop )

@octalmage
Copy link
Owner Author

@octalmage octalmage commented Sep 19, 2015

Agreed, I would like to implement this since there's a special way I want it implemented (see bitmap branch). I'll start on this after #87 is merged and RobotJS is completely updated. This is going to take some extra work on Windows due to the libpng and zlib dependencies, which is the main reason I put it off. These libraries are required for disk related functionality, and createMMBitmapFromString.

#64 is another priority for me.

@octalmage octalmage modified the milestone: First Stable Release. Sep 22, 2015
octalmage added a commit that referenced this issue Oct 6, 2015
@Deltatiger
Copy link
Collaborator

@Deltatiger Deltatiger commented Dec 11, 2015

@octalmage I will look into this bug and get it sorted out. Is the version implemented for Linux ?

@octalmage
Copy link
Owner Author

@octalmage octalmage commented Dec 11, 2015

Nice! Yeah it works for Linux too. I'm pretty sure my math is wrong for the buffers though. If I remember correctly it crashed when I passed the image buffer back to C++ from JavaScript in colorAt.

@octalmage octalmage mentioned this issue Jan 6, 2016
1 of 2 tasks complete
@SiriusDG
Copy link

@SiriusDG SiriusDG commented Apr 19, 2016

Is the image save implemented yet? I tried to follow the demo from comments, desktop.save did not yet work. Any simple way to save captured images? This is about all I really need to make great use of this, thanks!

@octalmage
Copy link
Owner Author

@octalmage octalmage commented Apr 19, 2016

The first part is implemented, screen capturing, but not saving yet. We just released the capture code in 0.4.0 this weekend, so it will be a bit longer before the saving code is out.

@octalmage octalmage changed the title Implement screen capture, save, open, and search. Implement screen capture, save, open, and search. Oct 16, 2016
@octalmage octalmage added ready and removed in progress labels Oct 16, 2016
@roccomuso
Copy link

@roccomuso roccomuso commented Nov 16, 2016

+1

@stephenthedev
Copy link

@stephenthedev stephenthedev commented Dec 8, 2016

@octalmage is there an update on this? Has the ready tag since 10/15

@porsager
Copy link

@porsager porsager commented Jan 26, 2017

@octalmage Also curious if this is being worked on or if you'd be ok with a pull request? I'd probably be interested in making the call async. What do you think about that?

@Hum4n01d
Copy link

@Hum4n01d Hum4n01d commented May 1, 2017

Any news?

@happilymarrieddad
Copy link

@happilymarrieddad happilymarrieddad commented May 4, 2017

+1

@octalmage
Copy link
Owner Author

@octalmage octalmage commented May 4, 2017

@porsager I've got screen capture implemented, but that's it. There's logic in there for the rest but it needs some work. See https://github.com/octalmage/robotjs/tree/bitmap-save.

I don't have the time to work on this currently (and won't for a while), but I'm happy to accept and review PRs.

Thanks!

@carlosgalveias
Copy link

@carlosgalveias carlosgalveias commented May 5, 2017

While saving the screen capture is not implemented, here's a workaround i used by using the Jimp package:
https://github.com/oliver-moran/jimp

var Jimp = require('jimp');
var robot = require('robotjs');

var screenCaptureToFile = function(path) {
    return new Promise((resolve, reject) => {
        try {
            var picture = robot.screen.capture();
            var image = new Jimp(picture.width, picture.height, function(err, img) {
                img.bitmap.data = picture.image;
                img.getBuffer(Jimp.MIME_PNG, (err, png) => {
                    image.write(path, resolve);
                });
            });
        } catch (e) {
            console.error(e);
            reject(e);
        }
    })
}

screenCaptureToFile('test.png');

@developer239
Copy link

@developer239 developer239 commented May 10, 2017

Thank you so much.

@omegacoleman
Copy link

@omegacoleman omegacoleman commented Jun 6, 2017

The solution @carlosgalveias given has made two mistakes, the correct version might be:

    var image = jimp(pic.width, pic.height, function(err, img) {
        img.bitmap.data = pic.image;
        img.scan(0, 0, img.bitmap.width, img.bitmap.height, function (x, y, idx) {
          var red   = img.bitmap.data[ idx + 0 ];
          var blue  = img.bitmap.data[ idx + 2 ];
          img.bitmap.data[ idx + 0 ] = blue;
          img.bitmap.data[ idx + 2 ] = red;
        });
        img.write("./" + n + (new Date().getTime()) + '.png');
    });

but this need improving as well. Jimp and RobotJS pick differently from BGR and RGB, so it needs reversing. Actually I think performing an option in robotjs about RGB and BGR is a better idea.

@johnmarlow007
Copy link

@johnmarlow007 johnmarlow007 commented Aug 25, 2017

Hi guys. I just wanted to say "CONGRATULATIONS!" for this project.
I have tried a few things and it works pretty much flowlessly.

Now I too need this screen capture feature with image saved locally for further processing and I highly encourage your efforts to get it working.

I would like to support the project but will need to get deeper in the study of how its done and I hope I can help with my limited knowledge.
Thank you and best of luck !
Andy

@arfost
Copy link

@arfost arfost commented Nov 9, 2017

Hi,

I'm not sure why but the two above functions with jimp didn't work for me, and I needed to save the captured picture, so here is my version of the function using JIMP if it can help someone :)

var Jimp = require('jimp');
var robot = require('robotjs');

var screenCaptureToFile = function (path,  robotScreenPic) {
    return new Promise((resolve, reject) => {
        try {
            var image = new Jimp(robotScreenPic.width, robotScreenPic.height);
            image.scan(0, 0, image.bitmap.width, image.bitmap.height, function (x, y, idx) {
                var color = robotScreenPic.colorAt(x, y);
                var red = parseInt(color[0] + color[1], 16);
                var green = parseInt(color[2] + color[3], 16);
                var blue = parseInt(color[4] + color[5], 16);

                image.bitmap.data[idx + 0] = Number(red);
                image.bitmap.data[idx + 1] = Number(green);
                image.bitmap.data[idx + 2] = Number(blue);
                image.bitmap.data[idx + 3] = 255;
            });
            image.write(path, resolve);

        } catch (e) {
            console.error(e);
            reject(e);
        }
    })
}

Thanks to @carlosgalveias and @omegacoleman for the starter, I couldn't find it without them.
Beware, it can be very slow for big pictures

@vernondegoede
Copy link

@vernondegoede vernondegoede commented Nov 9, 2017

@arfost I experienced the same issue with large pictures (especially retina screens). It took too long to process the picture (about ~600ms), so I ended up using the desktop-screenshot package.

https://github.com/vernondegoede/nonbilight/blob/master/renderer/components/TheaterMode.js#L50

@carlosgalveias
Copy link

@carlosgalveias carlosgalveias commented Nov 10, 2017

In the meantime i found this:
http://tipsandtricks.runicsoft.com/Cpp/BitmapTutorial.html#chapter4

Maybe this can be done natively in C++ so that its faster, in certain cases the JIMP solution is taking 2 seconds.
But getting the output buffer as a RGB instead of a BRG is a must for me.
I would implement it, and tried but crashes all over the place (i'm not a c++ programmer)

@octalmage , what do you think ?

@Deltatiger
Copy link
Collaborator

@Deltatiger Deltatiger commented Nov 10, 2017

This has to be done in C++ to get some speed benefits. Currently the entire screen buffer's memory is copied onto a bitmap object in C++ and then returned to JS. The code snippet posted above iterates over each pixel (I guess) before the image is generated. For a simple 1024 x 768 resolution screen (Which is a very small screen compared to what is actually out there) this is going to run a lot of times.
Instead if RobotJS binary can directly create the specified file in BMP format, it should be way easier and a lot quicker.
And since the data is still being generated as a Bitmap image, creating a BMP file should be easy.

@carlosgalveias
Copy link

@carlosgalveias carlosgalveias commented Nov 13, 2017

Ok, i managed to do it in C++ and now its fast.

Made a fork here https://github.com/carlosgalveias/robotjs in windows that outputs the buffer as a RGB and not as a BRG

Added the function to MMBitmap.c :

MMBitmapRef ConvertBMPToRGBBuffer (MMBitmapRef bitmap) {

assert(bitmap != NULL);
uint8_t *rgbBitmap = NULL;

if (bitmap->imageBuffer != NULL) {
	const size_t bufsize = bitmap->height * bitmap->bytewidth;
	rgbBitmap = malloc(bufsize);
	if (rgbBitmap == NULL) return NULL;
	memcpy(rgbBitmap, bitmap->imageBuffer, bufsize);
}

// find the number of padding bytes
int padding = 0;
int scanlinebytes = bitmap->bytewidth;
while ( ( scanlinebytes + padding ) % bitmap->bytewidth != 0 )     // DWORD = 4 bytes
	padding++;
// get the padded scanline bitmap->bytewidth
int psw = scanlinebytes + padding;

// now we loop trough all bytes of the original buffer, 
// swap the R and B bytes and the scanlines
long bufpos = 0;   
long newpos = 0;
for ( int y = 0; y < bitmap->height; y++ )
	for ( int x = 0; x < bitmap->bytesPerPixel * bitmap->width; x+=bitmap->bytesPerPixel )
	{
		newpos = y * bitmap->bytesPerPixel * bitmap->width + x;     
		rgbBitmap[newpos] = bitmap->imageBuffer[newpos + 2];       
		rgbBitmap[newpos + 2] = bitmap->imageBuffer[newpos];     
	}

// Do not forget to destroy the original bitmap if you dont need it to avoid memory leaks
return createMMBitmap(rgbBitmap,
                      bitmap->width,
                      bitmap->height,
                      bitmap->bytewidth,
                      bitmap->bitsPerPixel,
                      bitmap->bytesPerPixel); }

On screengrab.c i added:

/* Copy the data to our pixel buffer. */

if (bitmap != NULL) {
	bitmap->imageBuffer = malloc(bitmap->bytewidth * bitmap->height);
	memcpy(bitmap->imageBuffer, data, bitmap->bytewidth * bitmap->height);
	bitmapRGB = ConvertBMPToRGBBuffer(bitmap);
}
destroyMMBitmap(bitmap);
@Zazck
Copy link

@Zazck Zazck commented Nov 30, 2017

@arfost process image as buffer(not as array) will speed up your convertion.

function screenCaptureToFile(robotScreenPic) {
  return new Promise((resolve, reject) => {
    try {
      const image = new Jimp(robotScreenPic.width, robotScreenPic.height);
      let pos = 0;
      image.scan(0, 0, image.bitmap.width, image.bitmap.height, (x, y, idx) => {
        /* eslint-disable no-plusplus */
        image.bitmap.data[idx + 2] = robotScreenPic.image.readUInt8(pos++);
        image.bitmap.data[idx + 1] = robotScreenPic.image.readUInt8(pos++);
        image.bitmap.data[idx + 0] = robotScreenPic.image.readUInt8(pos++);
        image.bitmap.data[idx + 3] = robotScreenPic.image.readUInt8(pos++);
        /* eslint-enable no-plusplus */
      });
      resolve(image);
    } catch (e) {
      console.error(e);
      reject(e);
    }
  });
}

it will be 400 times faster

buffer: 50 convert: 21213
buffer: 55 convert: 21277
buffer: 40 convert: 21823
@jtoy
Copy link

@jtoy jtoy commented Mar 3, 2018

is this considered complete?

@ghost
Copy link

@ghost ghost commented Apr 10, 2018

thanks @Zazck for that snippet.

I was having an issue on my computer with the alpha channel not rendering correctly; it was spitting out large black rectangles over certain areas. In case anyone else is having the same issue, this forces the alpha channel to always be 100% during the conversion:

function screenCaptureToFile(robotScreenPic) {
  return new Promise((resolve, reject) => {
    try {
      const image = new Jimp(robotScreenPic.width, robotScreenPic.height);
      let pos = 0;
      image.scan(0, 0, image.bitmap.width, image.bitmap.height, (x, y, idx) => {
        /* eslint-disable no-plusplus */
        image.bitmap.data[idx + 2] = robotScreenPic.image.readUInt8(pos++);
        image.bitmap.data[idx + 1] = robotScreenPic.image.readUInt8(pos++);
        image.bitmap.data[idx + 0] = robotScreenPic.image.readUInt8(pos++);
        pos++
        image.bitmap.data[idx + 3] = 255;
        /* eslint-enable no-plusplus */
      });
      resolve(image);
    } catch (e) {
      console.error(e);
      reject(e);
    }
  });
}
@Sarkell
Copy link

@Sarkell Sarkell commented Apr 11, 2018

@fanfare I didn't come across this issue. And I have been using this solution for a week very intensively

@octalmage It would be wonderful to have this feature ASAP. The solution seems to be found

@will123195
Copy link

@will123195 will123195 commented Jun 12, 2019

This worked for me:

const robot = require('robotjs')
const Jimp = require('jimp')

function captureImage({ x, y, w, h }) {
  const pic = robot.screen.capture(x, y, w, h)
  const width = pic.byteWidth / pic.bytesPerPixel // pic.width is sometimes wrong!
  const height = pic.height
  const image = new Jimp(width, height)
  let red, green, blue
  pic.image.forEach((byte, i) => {
    switch (i % 4) {
      case 0: return blue = byte
      case 1: return green = byte
      case 2: return red = byte
      case 3: 
        image.bitmap.data[i - 3] = red
        image.bitmap.data[i - 2] = green
        image.bitmap.data[i - 1] = blue
        image.bitmap.data[i] = 255
    }
  })
  return image
}

captureImage({ x, y, w, h }).write('capture.png')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.