Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Draw on Slides #83

Closed
LauraRK opened this issue Jan 29, 2021 · 70 comments
Closed

Feature Request: Draw on Slides #83

LauraRK opened this issue Jan 29, 2021 · 70 comments

Comments

@LauraRK
Copy link

@LauraRK LauraRK commented Jan 29, 2021

I would really love the ability to write on my HTML slides as needed while presenting. I saw that Akkana Peck https://shallowsky.com/blog/speaking/drawing-on-slides.html was able to do that with her slide making code and I looked at her js files and tried to copy the appropriate parts and get it to work on my slides but I am not knowledable enough to make it work. I would love the shift D bring up a small ability to draw and then keep scrolling feature. I have tried a variety of solutions to this and none of them work the way I want them to, which is basically I want to draw and move on with out getting asked if I want to save anything or having my drawings stay present on the screen when I move to the next slide. I also like how her solution just has a little object with the colors and not a big obtrusive feature. I will continue trying to make it work but I think this would be a great thing to have in xaringanExtra. Thank you.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

Cool idea, @LauraRK. I actually saw your post on the RStudio Community forum and it got me thinking about this feature. I have some thoughts, but I want to run it by Garrick, since he's both a JS and xaringan expert!

@gadenbuie, do you think it would be possible to define a CSS class that can be added to a slide in the normal fashion, and then when that slide is the visible slide (.remark-visble), use some JS magic to wrap the slide (.remark-slide-container) in a transparent <canvas> and record the mouse movement and mouse down events to draw? Actually, it could probably be more general - the class could be added anywhere (.canvas[] ?) and wrap the div/parent div so you could arbitrarily define the drawing surface. Thoughts?

If you think this is reasonable, I would be interested in making an attempt to implement it as I have been actively working on my JS skills recently.

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 29, 2021

Thank you for maybe taking this on. It might be helpful to see the code @akkana wrote at the bottom of navigate.js in her presto package.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

I'm also very interested in this idea. I'd certainly be willing to test, and perhaps to help. I'm familiar with javascript/css/etc and a user of xaringan, but I've not done any sort of deep dive into how xaringan is written.

It would be especially great if what was written persisted on slides, so that if you leave and return, what was written is still visible. More complicated would be the ability to save all the writing (say for students to return to later) while also being able to clear the writing (say for another presentation of the same slides).

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

I wonder if fabric.js might be useful for this: http://fabricjs.com/

fabric.js is used in this R package: https://cran.r-project.org/web/packages/fabricerin/index.html

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

Thanks for sharing, @rpruim.

Looks like the {fabricerin} package already works with {xaringan} to create a standalone canvas for drawing etc.. I have already hacked together enough JS to add a canvas to the entire visible slide with the press of a button. Perhaps connecting my JS with this package would get some of the way to a working feature.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

That's just the sort of thing I was thinking -- a transparent canvas over the entire slide on which one can draw. The ability to toggle on and off is important especially if the slide has other interactive things on it (links, play buttons, etc.)

Let me now if you get to a point where you need some to test drive something.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

Looks like (at least in the default use) {fabricen} has an erase feature, but not a "clear all" option. The eraser seems to be very small -- same size as the pen? -- so erasing isn't very easy, especially if you want to erase a lot.

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 29, 2021

Thank you both. It is fine with me if I can't save it at this point, although I agree it is a nice long term feature. I would think that perhaps when you print it to pdf they would save if they are written via canvas? That is my next thing to figure out.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

@mattwarkentin , I'm wondering if it makes sense to think about a more general solution -- basically a way to add a transparent canvas element to fill a div (and resize as needed). That div could be a slide, but could also be any other element -- including from HTML that is generated in other ways. {xaringan} could then use this more general solution applied to its slide divs. This is a little out of scope for {xaringan}, but I could see this being generally useful for other types of documents as well.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

Quick note on erasing: {fabricerin} has a gumSize argument that controls the size of the eraser.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

Off to a decent start...

Screen.Recording.2021-01-29.at.1.56.04.PM.mov

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

I would think that perhaps when you print it to pdf they would save if they are written via canvas? That is my next thing to figure out.

@LauraRK, the drawings will stay even when you flip between slides, by default, or can be removed if you toggle off the canvas (I will add a "Clear" button also). The drawings will appear if you decide to "print" the slides to PDF, for example.

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 29, 2021

That is wonderful. Thank you so so much for working on this. This is what I want. 👍

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 29, 2021

When you are ready to have it tested out let me know and hopefully I can figure out how to get it working. I am a bit new to this process flow, using Github, knitting slides, etc.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

I think the basic functionality is ready to be tested. The canvas can be toggled by pressing "d" on any slide. The controls for stroke colour and thickness show up in the top right corner.

For various reason I didn't end up using the {fabricerin} package. Instead I just used the fabric.js library directly, and wrote my own JS/CSS to get things working. This has the side-benefit of also not depending on another R package, should Garrick consider this for inclusion in {xaringanExtra}.

It turns out erasing is tricky. In {fabricerin}, what looks like erasing is actually just painting over your current drawing with the same colour as the canvas background colour. This isn't really viable for xaringan since painting over your drawing will obfuscate the slide content underneath. As of now, you can only clear the entire slides drawing by toggling the canvas off and back on again.

You can try out this development version by running this code in R:

remotes::install_github("mattwarkentin/xaringanExtra")

And all you need to do is add the following code to your xaringan Rmd file:

xaringanExtra::use_fabric()

Please let me know if anything doesn't work as expected, or whether you think any features are missing.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

I haven't tried it yet, but I have a question: if I'm using my ipad, how do I toggle into drawing mode?

I'll be giving it a try shorty.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

Hmm, thats a good question that I don't know the answer to. I suppose that issue applies to all of the behaviours triggered by keystrokes. Perhaps I could write some JS to detect the device and if its mobile, add a button to toggle draw mode. Thoughts?

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

Gave it a quick try. Was able to draw. I like the pen and color adjustment widgets. Things I'm noticing:

  • hitting D a second time clears the writing
  • for a slide built gradually, each portion of the build is a "slide", so when you advance you lose any writing from the previous portion of the slide.
  • You are able to write on top of R code. I made a fork of {fabricerin} to experiment a bit, and one quirk I'm seeing is that I can't write on top of R code for some reason.

Haven't tried it on an ipad yet.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

Since drawing is easiest on a tablet, adding some support for tablets would be really good.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

In case my comment about drawing on top of R code was unclear, here is a screen shot:

image

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 29, 2021

hitting D a second time clears the writing

This is the expected behaviour, for now. Open to suggestions.

for a slide built gradually, each portion of the build is a "slide", so when you advance you lose any writing from the previous portion of the slide.

Good observation. Hmm, I will have to think about whether this can be handled gracefully.

Since drawing is easiest on a tablet, adding some support for tablets would be really good.

Thinking about this some more, I think it would be relatively straightforward to figure out how to toggle draw-mode on a mobile device, but the bigger issue is that screen gestures on a tablet changes slides, so I don't think drawing would play nicely. I need to think about this some more.

In case my comment about drawing on top of R code was unclear

This looks like you're using {fabricerin}, yes? I don't think this is an issue with my implementation in xaringan. At least not in my tests.

Screen.Recording.2021-01-29.at.5.54.53.PM.mov

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

I didn't have any trouble writing on R code with your stuff either.

I was using my slightly hacked version of {fabricerin} -- basically adding a small div hold the controls and then inserting into it the larger canvas rectangle so that you can draw over what comes below. The R code behavior surprised me, but I haven't done more digging to figure out why it behaves that way.

@rpruim
Copy link

@rpruim rpruim commented Jan 29, 2021

I'll have to think a bit more about ipad gestures, but perhaps swipe left and swipe right could be used to advance slides rather than tap -- at least when the fabric stuff is turned on.

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 30, 2021

It works for me :) I have one small request, and that is if perhaps a few colors could be there as a square, or maybe if I reset the color it would stay that color. Right now it is black and almost all my slides are dark blue, so every time I click d I would have to change the color. It is basically exactly what I want though besides that, I love how it is pretty basic and not fancy. I am very grateful. Makes me believe in the good of humanity :) simple, easy, love it 💯 Thank you!!!

@rpruim
Copy link

@rpruim rpruim commented Jan 30, 2021

First, I agree that the basic functionality is there. So that's great.

In terms of colors, two thoughts come to mind:

  • Perhaps there could be global setting for the default color as an argument to use_fabric()? That way the default color could be chosen to reflect the colors used in the slides. Many people will just want one color, but will want to choose it to be visible on their slides. Perhaps this could be overridden on a slide basis (in the RMarkdown).
  • For multicolor stuff (which is probably less important, but some people might really want), it would be nice to have a small number of quickly selectable colors. But I also think we don't want too much clutter. Like @LauraRK , I think simple, basic, and functional is nice.

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 30, 2021

OK, so I did test it on my touch screen and one thing that is causing issues is that when I am writing it thinks I am advancing slides, because I believe a touch also advances slides, so it works when I write with my mouse but not my stylus or finger. Maybe there is a work around if I put in some other option when I use the drawing?

@LauraRK
Copy link
Author

@LauraRK LauraRK commented Jan 30, 2021

I see above reading @rpruim 's comments that it is the tap that is the problem. This is a problem on my touch screen laptop as well and since I want to use the touch screen to write that causes a challenge.

@gadenbuie
Copy link
Owner

@gadenbuie gadenbuie commented Jan 30, 2021

This looks great and is really exciting! Thanks everyone for working on this.

A few thoughts about implementation:

  • Instead of a keyboard shortcut, I'd rather add a button with a pencil icon to the corner of the slide. Clicking the icon enables the writing feature. This makes the interface easier to access in touch devices.
  • We'd give two options for exiting drawing mode:
    • Turn of drawing mode but leave the drawing on the slide
    • Clean the canvas (and exit drawing mode)
  • While drawing is enabled we need to block the remarkjs touch event listeners. I've done this in another extension, I think panelset blocks both touch and drag events.
  • I'd rather pick a reasonable default for the line width and simplify the ui a bit to take out the line width slider and make some room for a color picker (maybe three colors?) We'd pick three good colors (e.g. red, green, blue), but we can let the user set the palette in the use_fabric() function.
  • Regarding continuation slides I think our options are to either attach the canvas to an individual slide -- i.e. a new canvas for each continuation -- or to have one canvas for the entire slide deck. Maybe this could be a user option, but I'd vote for the first. remarkjs treats each partial slide as if it were a completely new slide and it's probably overly complicated to try to sync all the drawings in partial slides.

Thanks again @mattwarkentin for the work on this!

@gadenbuie
Copy link
Owner

@gadenbuie gadenbuie commented Jan 30, 2021

Oh and we're going to need a way to erase lines that isn't painting over existing lines. This might be complicated. I'd aim for the type of eraser where the user draws an "erase" line (which is probably white) and then any existing drawn paths that intersect that line, and the erase line itself, are removed from the canvas.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Jan 30, 2021

Thanks everyone for the great feedback! I will keep working on these changes and see what we can come up with.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

I just installed it, and I see use_fabric() and the help for that function. But the 'd' shortcut isn't doing anything as far as I can tell. Is there another step I'm omitting?

The short-cut is also not listed in the help screen.

I've restarted the R session and rebuilt the slide deck -- no changes.

I can see that canvas stuff is being loaded:

image

packageVersion("xaringanExtra")
## [1] ‘0.2.4’

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Sorry, I should have explained things better. The API has changed a lot since the last version:

  • use_canvas() replaces use_fabric(). This is because fabric.js is no longer used as the backend for a variety of reasons that I will describe when I submit the PR.

  • The toolbox of buttons that control the drawings do not show up when the presentation is first loaded, but they will/should show up (in the top right corner) when you start navigating slides. The "d" keypress only toggles the visibility of the toolbox. To initiate drawing, click the edit/drawing icon.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

I'm not seeing the toolbox, whether or not I hit the D key. I typed use_fabric(), but I meant use_canvas() -- I did get that correct.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Does the toolbox not show up when you change slides?

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

That's correct:
image

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Hmm. Can you let me know which browser you are using? Also, could you check the console for any errors?

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Looks like an undeclared variable:

canvas.js:130 Uncaught ReferenceError: i is not defined
at canvas.js:130
at r (remark-latest.min.js:1)
at HTMLCollection.t.forEach (remark-latest.min.js:1)
at HTMLDocument. (canvas.js:129)

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

This was in chrome. I also tried in firefox.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Okay, this is a bug. It was a last second change I made. Quick fix, one sec.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Looks like you need a two argument function in your forEach()

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Should be fixed. Please re-install and let me know!

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

I had already fixed it locally as well, but I'll reinstall from your repo to double check.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

So is it time to close this and make new issues for specific issues (like the known bug about mis-alignment of pen and drawing)?

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

I think those discussions can happen in the PR thread once I submit it. I just wanted to get yours and @LauraRK's input first.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Things are working locally, but if I push to github and serve with github.io, then I get errors:

GET https://rpruim.github.io/s341/S21/from-class/slides/libs/xaringanExtra-canvas/canvas.css net::ERR_ABORTED 404
Stat341-Day01.html:10 GET https://rpruim.github.io/s341/S21/from-class/slides/libs/xaringanExtra-canvas/canvas.js net::ERR_ABORTED 404
Stat341-Day01.html:11 Uncaught TypeError: window.xaringanExtraCanvas is not a function
at Stat341-Day01.html:11

You can see the slide deck at https://rpruim.github.io/s341/S21/from-class/slides/Stat341-Day01.html

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Did you also upload the libs folder (or whatever you named it)? Seems like the presentation can't find the CSS and JS resources.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Indeed, I missed that when committing. (I had the libs folder from before, but missed adding the new bits.)

After waiting for github.io to catch up, it appears to be working.

Next up: try it on ipad. Stay tuned.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Ipad seems to work.

My initial impression is that the drawing is more pixelated than in the first version that used fabric. Is that true, or just me not remembering correctly?

Seems like the main thing to fix now is the position/scaling issue so that you can draw where you mean to draw. As a temporary fix, is there an ideal aspect ratio to use that makes this least obvious? In particular, would there be a best aspect ratio to use if I wanted things to be full screen (need to figure out how to do that) on the ipad?

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

It may be more pixelated. There may be ways to improve upon this still (seems like there is, from doing some reading).

The benefit of fabric.js is that it turns your drawing into proper paths. This makes for nice looking lines. But the downside is that it is very tricky to erase since the line is a fully connected path, and not a series of connected dots. To erase a path, you would have to simultaneously clip the path and the point of the cursor and then somehow remove the disconnected path. This is why the default "erase" functionality for fabric.js is to just paint over the path with white paint (or whatever colour the canvas is). This won't work for our purposes since that would just obfuscate the slide content underneath the canvas.

Thankfully, the default <canvas> has support for proper erasing. I thought proper erasing was an important enough feature that I made the choice to build my own drawing tool, rather than reverse engineer some erasing tool for fabric.js.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

That makes sense.

It would be nice to get some proper erasing into fabric.js -- that would make everything easier.

Is the scaling issue one that just needs time, or are there some head scratchers there?

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

I think I have found another bug. In Chrome on my mac, when I switch colors, the first stroke remains in the old color. Then next and subsequent ones use the new color.

This doesn't happen on my ipad, but there is also a different color chooser widget. So this might be determined by the browser's color picker.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Is the scaling issue one that just needs time, or are there some head scratchers there?

A bit of a head-scratcher, but I think I am on the right track.

when I switch colours, the first stroke remains in the old colour

I think this happens, I'm guessing, because you started drawing after you selected a new colour, but you hadn't "clicked off" of the colour picker. So in the instant you start drawing the new colour isn't registered yet. But it is registered immediately after so it works by the time you draw a second line. If you just click off of the picker without trying to draw, it should work as expected.

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Regarding colors: I think that color behavior is not ideal -- it feels very unnatural. But it may not be not really be xaringan's problem to fix. As I mentioned, on my ipad, I don't have to do anything to end my color choice, I can just start drawing. So this may be a bug/feature of the color-picker that gets used.

But if there were an easy way to make the intuitive behavior work, that would be nice.

Good luck with the scaling issues. That's really the only remaining thing to have this be useable. The rest is enhancements.

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 4, 2021

Regarding colors: I think that color behavior is not ideal

You are right - and I found a simple way to fix it. It should work as expected if you install the dev version. Thanks for the suggestion!

@rpruim
Copy link

@rpruim rpruim commented Feb 4, 2021

Just gave the color switching a test. Seems to be working. :-)

@Cameron-Fairfield
Copy link

@Cameron-Fairfield Cameron-Fairfield commented Feb 8, 2021

Just tried this today. Mostly working great. Main issue is misaligned cursor and pen outputs. I have to click above and to the left of where I want to draw by quite a bit. If I trace a letter in a title when on my second screen (slightly wider monitor) it appears quite far to the right and below cursor, if I then move window to my laptop with smaller screen the difference is much less significant but still there. I guess that screen size and aspect ratio may be the problem?

@mattwarkentin
Copy link
Contributor

@mattwarkentin mattwarkentin commented Feb 9, 2021

Thanks for trying it out, @Cameron-Fairfield! We are actively working on improving this feature in #87. So stay tuned for a much improved version in the near future.

@gadenbuie
Copy link
Owner

@gadenbuie gadenbuie commented Mar 8, 2021

Closed in #87 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants