Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ipycanvas not working (binary buffers issue?) #8

Closed
martinRenou opened this issue Sep 3, 2021 · 8 comments
Closed

ipycanvas not working (binary buffers issue?) #8

martinRenou opened this issue Sep 3, 2021 · 8 comments

Comments

@martinRenou
Copy link

Thanks for your work bringing widgets to colab! It has been requested many times by users.

I gave ipycanvas a try, and it seems to render the canvas properly. You can try with

!pip install -q ipycanvas

then

from IPython.display import Javascript
from ipycanvas import Canvas

display(Javascript('''
  google.colab.widgets.installCustomManager('https://ssl.gstatic.com/colaboratory-static/widgets/colab-cdn-widget-manager/6a14374f468a145a/manager.min.js');
'''))

canvas = Canvas(height=900)
canvas.fill_rect(0, 0, 50, 50)
canvas

This results in the Canvas widget properly being created in the output:
ipycanvas

Issue

But it's when I try to draw something on the Canvas that things go wrong:

canvas.fill_rect(0, 0, 50, 50)

This results in the following JavaScript error:

Error has occurred while trying to update output. TypeError: Cannot read properties of undefined (reading 'length')
    at bf (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:142)
    at qa.eval [as h] (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:143)
    at sa (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:15)
    at ua.next (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:15)
    at f (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:56)

If I can make a wild guess, I wonder if it could come from around here: https://github.com/googlecolab/colab-cdn-widget-manager/blob/main/src/manager.ts#L237

Context

To give you more context, ipycanvas makes extensive use of comm message for sending "draw calls". So when I do canvas.fill_rect(0, 0, 50, 50) it is sending a comm message (which is mostly a binary buffer) to the front-end. That's why I suspect it's failing in the Comm's on_msg method.

Is there a chance we can access non-minified source of the manager from the cdn? So that I can debug more easily. I tried https://ssl.gstatic.com/colaboratory-static/widgets/colab-cdn-widget-manager/6a14374f468a145a/manager.js but it does not seem to exist.

@blois
Copy link
Collaborator

blois commented Sep 3, 2021

Hi Martin- thanks for the bug report, I'll take a look at it shortly.

The non-minified source can be loaded from https://ssl.gstatic.com/colaboratory-static/widgets/colab-cdn-widget-manager/6a14374f468a145a/manager.dev.js. There are also steps for making changes locally at https://github.com/googlecolab/colab-cdn-widget-manager#development

@martinRenou
Copy link
Author

martinRenou commented Sep 3, 2021

Thanks! That seems so almost too easy to get a dev install 😄 I will give it a try over the week-end

@blois
Copy link
Collaborator

blois commented Sep 3, 2021

This one looks like it'll need a fix in Colab. The issue is that there's a comm message with the data:

{
    "content": {
        "dtype": "uint8",
        "shape": [
            18
        ]
    },
    "method": "custom"
}

Which also includes a single buffer of data. We're assuming elsewhere that every Widget comm message with buffers will also have buffer_paths set. Fixing that assumption appears to address the issue.

image

@blois
Copy link
Collaborator

blois commented Sep 8, 2021

The fix is now live.

@blois blois closed this as completed Sep 8, 2021
@martinRenou
Copy link
Author

Cool! Thanks a lot 😊

@martinRenou
Copy link
Author

I gave it a try today. It seems to now fail differently. The Canvas cannot be displayed anymore and the following JS error shows up:

Error has occurred while trying to update output. Error: Widgets module ipycanvas is not supported

@blois
Copy link
Collaborator

blois commented Sep 10, 2021

One change is that we're moving to a different way to opt-into the custom manager- with the new mechanism you can do:

from google.colab import output
output.enable_custom_widget_manager()

Once in the notebook and it will apply to all widgets in later cells. I didn't think this was live quite yet but looks like you may have gotten it.

Next is that it looks like this may have caused a timing change which I'm curious about- the canvas now needs to be displayed before the fill style is set in order for it to take effect:

from ipycanvas import Canvas
canvas = Canvas(height=900)
display(canvas)
canvas.fill_style = 'red'
canvas.fill_rect(0, 0, 50, 50)

Does ipycanvas persist the drawing in the notebook? If so, do you have a pointer as to how? At a glance it looked like the image buffer is sent outside of the widget state and right now we're only persisting the sate.

@martinRenou
Copy link
Author

Thank you for you response.

Does ipycanvas persist the drawing in the notebook?

It actually doesn't. The canvas widget is not a really good citizen of the widgets ecosystem as it's kind of stateless and relies too much on custom comms.

I've seen similar issue in the Voila front-end, as the Notebook is executed entirely before displaying the widgets. So the draw commands are sent to the void (no widget in the front-end to receive them) and the canvas is rendered blank afterwards.

This is definitely an issue on the ipycanvas side. I will try to fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants