Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ship vega4-extension by default #6572

Closed
davidanthoff opened this issue Jun 14, 2019 · 33 comments
Closed

Ship vega4-extension by default #6572

davidanthoff opened this issue Jun 14, 2019 · 33 comments

Comments

@davidanthoff
Copy link
Contributor

@davidanthoff davidanthoff commented Jun 14, 2019

I brought this up at the end of #6402, but given that this is really a separate action item from that original issue, I'm opening a new issue here.

I think jupyterlab 1.0.0 should ship the three extensions I mentioned in the title out-of-the box, in addition to the vega5-extension that is already included by default. I think ideally this issue here should be added to the 1.0.0 milestone.

Without that, users will experience a major regression when they update to jupyterlab 1.0.0: all notebooks that they created with earlier versions of jupyterlab that include vega or vega-lite figures will no longer display properly when opened in jupyterlab 1.0.0. I feel that experience would be incompatible with the promise that was made last spring that juptyerlab was ready for end users for their daily work. That messaging implied (to me at least) that minimally I would be able to open files in jupyterlab 1.0.0 that I created with vanilla juptyerlab 0.35.x and they would display correctly.

I believe this affects minimally Python and Julia users: Python users that used altair in their notebooks, and Julia users that used VegaLite.jl. I don't have numbers, but it seems to me that both packages were quite popular over the last year, so this doesn't strike me as a corner case/small user group that would be affected.

This would also be in line with what nteract is doing with their support for vega and vega-lite out-of-the-box.

I don't know how complicated this would be, and I do understand that it is late in the release process for jupyterlab 1.0.0... But on the other hand, I think from an end-user point of view this is really a major regression/bug. But maybe this wouldn't be so complicated to fix? Might it be enough to just move the existing code for vega2-extension, vega3-extension and vega4-extension from https://github.com/jupyterlab/jupyter-renderers into this repository here? And add them to a few config places so that they get shipped out-of-the-box?

Pinging @domoritz and @jakevdp from the vega, vega-lite and altair side of things. Not sure who from the jupyterlab team would be the right point person?

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 14, 2019

Thanks for opening this, @davidanthoff. I think this is a good suggestion.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 14, 2019

I'm supportive of this. Since Jupyterlab loads Vega asynchronously the bundle shouldn't be larger either. I don't think we want to go back to ancient Vega and Vega-Lite versions, though.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 14, 2019

I don't think we want to go back to ancient Vega and Vega-Lite versions, though.

What is 'ancient' in your mind?

@jasongrout jasongrout added this to the 1.0 milestone Jun 14, 2019
@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 14, 2019

Tagging as 1.0 for discussion.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 14, 2019

I don't think anyone needs versions before Vega-Lite 2 and Vega 3 in JupyterLab.

I would say we concentrate on what Altair needs and that is Vega-Lite 2 and Vega-Lite 3. The corresponding Vega extensions in jupyterlab are vega4 and vega5.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 14, 2019

My suggestion would be to bring back the vega4 extension from before #6133. Any objections? If not, I will send a PR.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 14, 2019

So to confirm, we'd just be adding one more extension, the vega4 extension? Would you also update it to pull in the most recent appopriate vega-embed: https://github.com/jupyterlab/jupyter-renderers/blob/a3a5d361eefbe1a8ab5519e7658f7e5bbb74af91/packages/vega4-extension/package.json#L35

@davidanthoff
Copy link
Contributor Author

@davidanthoff davidanthoff commented Jun 14, 2019

My suggestion would be to bring back the vega4 extension

I think that makes sense for another reason: Jupyterlab v0.33.0 was the first release that was not billed as a beta release and declared as ready for general use. That version had vega4 in it, so I think supporting that version that was declared as ready for general use going forward as the oldest release is reasonable.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 15, 2019

Changing title to reflect this.

@jasongrout jasongrout changed the title Ship vega2-extension, vega3-extension and vega4-extension by default Ship vega4-extension by default Jun 15, 2019
domoritz added a commit to domoritz/jupyterlab that referenced this issue Jun 17, 2019
domoritz added a commit to domoritz/jupyterlab that referenced this issue Jun 17, 2019
@ellisonbg
Copy link
Contributor

@ellisonbg ellisonbg commented Jun 18, 2019

A few of us are talking about this at the JLab 1.0 sprint. Questions that are coming up:

  1. What are the downsides, especially in terms of performance or UX of shipping both vega4 and vega5?
  2. If we ship vega4 and vega5 by default now, what happens when vega6 comes out?
  3. Did we move to ship vega5 by default too quickly? In other words, should we only ship vega4?
  4. Is all of this simpler if we don't ship any of them by default?

Of these, getting clarity on 1+2 is especially important to move forward with shipping both vega4 and vega5. Thoughts?

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 18, 2019

  1. As far as I understand it, there are no downsides. The majority of the extension is loaded asynchronously so the impact is minimal. For users the UX will be better since they can use Vega 4 and 5 at the same time.
  2. We ship vega6, I guess.
  3. Then Altair 3 wouldn't work. I think this would be the worst possible situation for users.
  4. I don't think so. JL needs to have a story for updating versions of dependencies and Vega is probably a good test case because you have me and I can make releases and fix incompatibilities for you ;-)

Another option may be a Vega package that supports multiple versions at once but I won't have the cycles to implement this.

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 18, 2019

I wonder if it might be possible to remove the versioning entirely from the vega extension. Can lab extensions access HTTP as part of their handling of different mimetypes? If so could there be a versionless vega etension that uses the vega version listed in the the specification's $schema argument, loaded via the standard CDN links? The downside would be that it would no longer work offline. The upside would be that it would just work and no longer depend on your kernel's Python packages being in sync with your kernel manager's frontend extensions, and that would be a HUGE win as far as usability and maintainability.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 18, 2019

Or the extension could come with one version that works offline and for older versions it fetches via CDN. I like that.

The tricky bit is that Altair sets the mimetype to be a particular version of Vega-Lite rather than just Vega-Lite. This means that the extension won't be loaded. Would it make sense to not output a versioned mimetype from Altair?

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 18, 2019

It would be easy to change Altair to output an unversioned mimetype, if the frontends would support it.

@davidanthoff
Copy link
Contributor Author

@davidanthoff davidanthoff commented Jun 18, 2019

I think having one extension that handles all vega and vega-lite versions would be great. Moving to one version independent MIME type also sounds like a great idea. I think ideally such an extension would a) ship offline versions of all major versions of vega and vega-lite that existed when the extension was last shipped out-of-the-box, and b) had an option to download (and cache for offline use) newer versions if it encounters a spec that had a newer version header.

For 1.0.0, though, I think it would be great if the priority was to ship something that a) displays all existing notebooks that were created with released versions of jupyterlab out of the box (so that means it has to support versioned MIME types that were in use over the last year or so) and b) includes support for vega-lite 3 and vega 5 (which I think means just adding the existing vega4 extension to jupyterlab). It seems that just moving the vega4 extension is the more realistic option in the short term that would also not derail the schedule for jupyterlab 1.0.0 too much?

@ellisonbg
Copy link
Contributor

@ellisonbg ellisonbg commented Jun 18, 2019

Thanks for giving input on this everyone. I don't think relying on a CDN for extensions that are shipped with JupyterLab is going to work. We have a lot of people who run JupyterLab in locked down settings where they aren't going to be able to reach a CDN.

I agree it would be nice to have a single, unversioned mime-type for vega and vega-lite, but given how close we are to 1.0, that will probably have to wait until later. This work will also impact lots of projects (nteract, colab, altair, etc), so it will take some careful planning.

Based on this, my sense is:

  • For the short term, ship the vega4 and vega5 pacakges in JupyterLab 1.0.
  • Long term, create a new unversioned mimetype, a single renderer for that, and a version of vega-embed that can render those things.

Is everyone ok with this plan?

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 19, 2019

@domoritz - to test this out, can you give us a vega 4 file that will not work in vega 5, and a vega 5 file that will not work in vega 4? Then we can be sure we are using the right versions for each file.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 20, 2019

@davidanthoff - the notebooks you posted about over at nteract/nteract#4442 containing vega 4 and vega 5 code - are they invalid for the other version of vega? Just curious.

@davidanthoff
Copy link
Contributor Author

@davidanthoff davidanthoff commented Jun 20, 2019

No, I think they are probably exactly the same JSON, just stored as a different MIME type.

@saulshanabrook
Copy link
Member

@saulshanabrook saulshanabrook commented Jun 20, 2019

@jasongrout You can do this by installing altair<3 and altair>3 and running a simple example like:

import altair as alt
import pandas as pd

source = pd.DataFrame({
    'a': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I'],
    'b': [28, 55, 43, 91, 81, 53, 19, 87, 52]
})

alt.Chart(source).mark_bar().encode(
    x='a',
    y='b'
)

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

No need to install multiple altair versions. You can do something like this with Altair v3:

# Vega-Lite 3 chart. This spec would fail if passed to vega-lite 2 because of the errorbar mark
import altair.vegalite.v3 as alt
from vega_datasets import data

alt.Chart(data.barley.url).mark_errorbar(extent='ci').encode(
  x='yield:Q',
  y='variety:N'
)

and:

# vega-lite 2 chart. This would render under vega-lite 3 because of backward compatibilitiy
import altair.vegalite.v2 as alt
from vega_datasets import data

alt.Chart(data.barley.url).mark_rule().encode(
    x='ci0(yield):Q',
    x2='ci1(yield):Q',
    y='variety:N'
)

I'm having trouble coming up with a spec that would be valid in Vega-Lite 2 but invalid in Vega-Lite 3... the team did a pretty good job of ensuring backward compatibility.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Jun 20, 2019

If we can't find a chart that won't render with vg 3, then perhaps this discussion is moot?

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

I don't think it's moot: we still need a v2 entrypoint, which is provided by the Jupyter extension, or charts created in old versions of Altair will not render. And my knowledge of the schema is not exhaustive, so I can't confidently say there are no examples that render in V2 and not in V3; I just can't come up with one.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 20, 2019

I'm having trouble coming up with a spec that would be valid in Vega-Lite 2 but invalid in Vega-Lite 3... the team did a pretty good job of ensuring backward compatibility.

You're welcome :-)

I think the only backwards incompatible change is that maxTitleLength is now titleLimit. See https://github.com/vega/vega-lite/releases/tag/v3.0.0

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

I think the only backwards incompatible change is that maxTitleLength is now titleLimit

Yeah, I spent some time playing with that, but I haven't figured out how to create a chart where titleMaxLength has any visible effect in VL 2. The docs say it's "Max length for axis title if the title is automatically generated from the field’s description," but I'm not certain what that means (I tried using an aggregation, so that the title would be generated from the aggregation and the field name, but the parameter still has no apparent effect).

And in VL3 it doesn't result in an error, even though it's not part of the schema.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 20, 2019

I'm getting an error

Screen Shot 2019-06-20 at 16 05 27

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

Yeah, there's an error in the vega editor's linter, but not in an embedded chart.

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

Is there a way to make that error show up within the JupyterLab extension? That's what I can't figure out how to do.

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

I'm trying to provide what @jasongrout asked for in #6572 (comment)

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 20, 2019

Oh, the editor used the schema to validate a spec and shows warnings. Vega-Embed doesn't validate inputs.

@domoritz
Copy link
Member

@domoritz domoritz commented Jun 20, 2019

Vega embed will check the $schema and compare the version in it with vl.version or vega.version. I already talked with @saulshanabrook about confirming that the versions are right on Slack.

@jakevdp
Copy link

@jakevdp jakevdp commented Jun 20, 2019

Oh, OK 👍

saulshanabrook added a commit to saulshanabrook/jupyterlab that referenced this issue Jun 26, 2019
This fixes jupyterlab#6709, and is a follow up to jupyterlab#6572. It separates building
vega assets from the rest of our package management. That way, we can
avoid any hoisting issues that we were seeing (possibly from yarnpkg/yarn#5520).

To test this, you can try this notebook which should render
both versions without errors:

https://gist.github.com/saulshanabrook/97c28550ab12e684adc3c325038537ce
@lock
Copy link

@lock lock bot commented Aug 6, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related discussion.

@lock lock bot locked as resolved and limited conversation to collaborators Aug 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants