Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support binary data in comm messages via buffers #4208

Merged
merged 9 commits into from
Aug 5, 2024
Merged

Conversation

seeM
Copy link
Contributor

@seeM seeM commented Aug 1, 2024

Addresses #3974 and #4173.

This PR plumbs buffers from jupyter-adapter messages through to the main thread and specifically to our IPyWidgets notebook renderer. This unlocks a bunch of widget types (thanks to @isabelizimm for these examples):

ipywidgets.Image

import ipywidgets
with open('image.png', 'rb') as f: image = f.read()  # you'll need an actual image path
display(ipywidgets.Image(value=image, format='png'))

ipympl (matplotlib's interactive backend)

Note: Save doesn't work yet. We'll need to handle the clicked-data-url webview message in our NotebookOutputWebview.

%matplotlib ipympl
import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots()
x = np.linspace(0, 2*np.pi, 100)
y = np.sin(3*x)
ax.plot(x, y)

ipydatagrid

import pandas as pd
from ipydatagrid import DataGrid
data = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}, index=["One", "Two", "Three"])
w = DataGrid(data, selection_mode="cell", editable=True)
display(w)
# Now change the value of a cell, then run:
w.data
# It should reflect the updated value.
# The next line should update the value in the widget frontend:
w.set_cell_value("A", "One", 3)

bqplot

There appears to be a console warning here, we can address that in a follow-up.

import bqplot.pyplot as bplt
import numpy as np

x = np.linspace(-10, 10, 100)
y = np.sin(x)
axes_opts = {"x": {"label": "X"}, "y": {"label": "Y"}}

fig = bplt.figure(title="Line Chart")
line = bplt.plot(
    x=x, y=y, axes_options=axes_opts
)
bplt.show()

// Don't use a renderer for non-widget MIME types
if (mimeType === 'text/plain' ||
mimeType === 'text/html' ||
mimeType === 'image/png') {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change, ipympl outputs would end up using a simple image/png renderer.

@@ -1,32 +0,0 @@
/*---------------------------------------------------------------------------------------------
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't used anymore.

@@ -1286,6 +1287,7 @@ export class JupyterKernel extends EventEmitter implements vscode.Disposable {
private async sendToSocket(id: string, type: string, dest: JupyterSocket,
parent: JupyterMessageHeader, message: JupyterMessageSpec, metadata: object = {}) {
const msg: JupyterMessage = {
// TODO: Do comms ever try to send buffers back to the kernel?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't seem to be the case in my testing.

nstrayer
nstrayer previously approved these changes Aug 1, 2024
Copy link
Contributor

@nstrayer nstrayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but I am not familiar enough with the architecture decisions to pass judgement on the impact of changing the message structure nesting levels.

@seeM
Copy link
Contributor Author

seeM commented Aug 2, 2024

The smoke tests are failing because I missed a few more places affected by this change... Working on it.

jmcphers
jmcphers previously approved these changes Aug 2, 2024
Copy link
Collaborator

@jmcphers jmcphers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, and the examples worked well in my testing.

image

Copy link
Contributor

@isabelizimm isabelizimm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this fixes the problem observed in #4172 where Positron will crash once before ipydatagrid loads 🎉

@seeM
Copy link
Contributor Author

seeM commented Aug 5, 2024

Thank you for the reviews! I believe I've addressed all of the comments.

@seeM seeM merged commit c8e3bb9 into main Aug 5, 2024
2 checks passed
@seeM seeM deleted the 3974-comm-msg-buffers branch August 5, 2024 12:35
@github-actions github-actions bot locked and limited conversation to collaborators Aug 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants