Program data includes camera picture of program

[maryrosecook]

• Looking for help with handling rotated papers! It currently only handles unrotated papers.
• The camera code stores picture data for each paper in the picture attribute of the program's entry in localStorage. (Alongside points and number.)
• Paper programs written by users can access the picture for a program like this: papers[paperNumber].picture
• Paper programs written by users can display a paper picture like this: ctx.putImageData(papers[paperNumber].picture, 0, 0).

[shaunlebron]

For context, this is mary's demo video.

Thanks for doing this! I looked at this for a bit, and the method of cropping the ROI (region of interest) in OpenCV is restricted to an axis-aligned bounding box, so I'm guessing the picture of a rotated paper will always look something like this:

It might be easiest to provide users with a general utility function instead—to take a region from the camera image of the whole projection surface, and draw it to a target canvas context (paper or supporter):

drawFromCamera(
  ctx,       // canvas context to draw to
  srcRegion, // source coordinates from camera image of whole projection surface
  dstRegion  // destination coordinates to draw to
)
// NOTE: a "region" can be an array of 3 or 4 vertices (triangle or quad for now)

For example, the demo video could be achieved with something like:

const srcIds = [...]; // source animation papers
const dstId = ...; // draw animation to this paper

while (true) {
  for (let srcId of srcIds) {
    drawFromCamera(supporterCtx, papers[srcId].quadPoints, papers[dstId].quadPoints);
    supporterCtx.commit();
    await delay(250);
  }
}

For implementation, the hard work has already been done by patternPath, found in Three.js's canvas renderer. It seems to retrofit the simple canvas API with this arbitrary texture filling function that we need.

I'll try this out in an isolated environment to confirm if the implementation works as expected.

[maryrosecook]

Sounds great! Definitely seems like a better approach. Thanks a lot!

[shaunlebron]

Progress here. It looks like transforming triangles separately doesn't look good. So I'll try to copy the transform method used for each paper context to do proper perspective correction.

[maryrosecook]

Wow cool!

…

On Apr 10, 2018, at 3:15 AM, Shaun Lebron ***@***.***> wrote: Progress here. It looks like transforming triangles separately doesn't look good. So I'll try to copy the transform method used for each paper context to do proper perspective correction. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[shaunlebron]

Works properly now. Will try to plug it in this weekend.

[janpaul123]

Very cool! Thanks both for your contributions. Looking forward to the final PR! 😄

[maryrosecook]

Super cool!

…

On 10 Apr 2018, at 4:16 pm, Shaun Lebron ***@***.***> wrote: Works properly now. Will try to plug it in this weekend. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[shaunlebron]

Won't be able to get to it this weekend, so I left some breadcrumbs here: #41

[janpaul123]

Superseded by #54. Thanks so much, very cool feature. 💯

[maryrosecook]

So excited this feature is in PaperPrograms. Thanks so much for making it general purpose and high quality, @shaunlebron and @janpaul123 !