Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove vega 4 extension #7523

Closed
jasongrout opened this issue Nov 14, 2019 · 8 comments · Fixed by #7650
Closed

Remove vega 4 extension #7523

jasongrout opened this issue Nov 14, 2019 · 8 comments · Fixed by #7650
Assignees
Labels
pkg:vega status:resolved-locked
Milestone

Comments

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Nov 14, 2019

In JupyterLab 2.0, I think it makes sense to finally remove the vega 4 extension and move to vega-lite 4.0 (which should support vega-lite 3 files). See #7428 for many more details and commits about this upgrade.

@jasongrout jasongrout added this to the 2.0 milestone Nov 14, 2019
@jasongrout jasongrout self-assigned this Dec 2, 2019
@domoritz
Copy link
Member

@domoritz domoritz commented Dec 8, 2019

Vega-Lite 4 is out now. We have not sent the announcement yet but it's on npm.

@davidanthoff
Copy link
Contributor

@davidanthoff davidanthoff commented Dec 8, 2019

So I'm quite nervous about this, after reading through all of #7428.

I think one of the core scenarios for JupyterLab notebooks is replication code for scientific papers. In those scenarios, you publish a paper, store a copy of your replication code on a service like zenodo.org, and then folks will try to look at and use that notebook for years to come. I think literally we need to think 5-10 years here, minimally. The cycles in science are just very, very different than in a more normal software environment. I think one of the core promises of Jupyter notebooks (if it wants to target the scientific space) must be that you can still open notebooks from way back in a recent version of JupyterLab, and everything will display just fine and in the same way it did when the notebook was created. Ideally you could also run it, but that is more complicated. But displaying seems like the minimal bar to me.

So these two quotes from @domoritz in the thread in #7428 make me nervous:

I think always keeping the last two versions would be good for users.

In rare cases, we don't do this for example in Vega-Lite 4 where we are removing support for rangeStep. I don't think we can promise backward compatibility for all future versions but we are making an effort to stay compatible.

I just honestly think that doesn't jive with the needs of replication code in the scientific domain...

Wasn't the original plan that JupyterLab would just ship with multiple (major) versions of vega and vega-lite, and would use the appropriate version to render cells in notebooks, based on the vega(-lite) version encoded in the MIME type of the cells? And that old versions wouldn't be removed, but just carried along? That seems to be the most stable approach to guarantee that old notebooks are rendered the same way they were when they were created? In my mind we should make every effort to stick with that plan, it seems like the right approach to me. If the problem was that the older versions of vega and vega-lite were incompatible with newer TS, then maybe we should try to release patch releases of the older versions of vega and vega-lite that work with the newer TS.

There is another story here: the Julia VegaLite.jl package always embeds a vega-lite MIME representation and a PNG representation in notebooks. So we always have one MIME representation that will look exactly correct in every notebook. What would be an unfortunate scenario is one where say JupyterLab 5 used vega-lite 6 to render vega-lite 3 MIME types, but due to some corner incompatible change it rendered it incorrectly. I would almost prefer if in that case the vega-lite 6 renderer just declared that it can't render vega-lite 3 MIME types, and JupyterLab fell back to just showing the PNG version. But really I think what should happen in that case is that JupyterLab 5 should still ship vega-lite 3 and use that to render the plot :)

I think the question of how to show vega and vegalite files is easier, I think just using the latest version of vega(-lite) seems ok for that scenario to me, if there is no version schema embedded in the file itself.

@davidanthoff
Copy link
Contributor

@davidanthoff davidanthoff commented Dec 8, 2019

Having said all of that, from my point a priority would be to get vega-lite 4 support into the shipping product soon, and if that means that for a while support for previous versions is not ideal, that would be ok with me, as long as we could find a medium run strategy to include first-class support for old versions of vega and vega-lite :)

@domoritz
Copy link
Member

@domoritz domoritz commented Dec 8, 2019

In rare cases, we don't do this for example in Vega-Lite 4 where we are removing support for rangeStep. I don't think we can promise backward compatibility for all future versions but we are making an effort to stay compatible.

Just a bit more context. We remove rangeStep from the schema but the compiler translates it automatically to the equivalent properties in Vega-Lite 4. Backwards compatibility is important to us (because of reproducibility as you mentioned). We will always make an effort to maintain backwards compatibility. However, we won't be able to guarantee it if there is a conflict with a new feature. When we break backwards compatibility (for some reason), we will throw a warning/error. In #7428, we were discussing different options and not necessarily a plan.

I would almost prefer if in that case the vega-lite 6 renderer just declared that it can't render vega-lite 3 MIME types, and JupyterLab fell back to just showing the PNG version

I think that's the plan. Only if Vega-Lite is fully backwards compatible, we will accept it.

I hope that my explanations give some context on the commitment to backwards compatibility from the Vega-Lite team. I'm also very happy to discuss better strategies on what we can do from the Vega-Lite side. Having said that, I don't have any say in the versioning strategies for jlab and refer to @jasongrout for comments on that side of things.

@jasongrout
Copy link
Contributor Author

@jasongrout jasongrout commented Dec 14, 2019

Wasn't the original plan that JupyterLab would just ship with multiple (major) versions of vega and vega-lite, and would use the appropriate version to render cells in notebooks, based on the vega(-lite) version encoded in the MIME type of the cells? And that old versions wouldn't be removed, but just carried along?

I never thought we'd ship old versions of vega-lite indefinitely with core. Rather, we'd ship older versions as extensions, and as a one-time exception for jlab 1.0, we shipped vega 4. Anyways, it is a moot point since the old version of vega-lite does not support new Typescript versions, so we can't include it in core.

If the problem was that the older versions of vega and vega-lite were incompatible with newer TS, then maybe we should try to release patch releases of the older versions of vega and vega-lite that work with the newer TS.

That is out of our (JLab's) hands. The Vega project has decided not to ship patch releases of the older versions of vega-lite to support the new Typescript versions.

We (core JLab) are providing whatever backwards compatibility and reproducibility guarantees that vega-lite is itself providing in its latest version. @domoritz and I also discussed how Vega-lite can provide an easy way to upgrade format versions that we can use from JLab. I think that's the most practical position for us to take for core jlab.

There is another story here: the Julia VegaLite.jl package always embeds a vega-lite MIME representation and a PNG representation in notebooks.

Agreed, I think we should do this as our fallback for backwards compatibility (i.e., encode the chart in a format like png that is widely used and has a much, much longer lifespan than the vega format).

@jasongrout
Copy link
Contributor Author

@jasongrout jasongrout commented Dec 14, 2019

I think one of the core scenarios for JupyterLab notebooks is replication code for scientific papers. In those scenarios, you publish a paper, store a copy of your replication code on a service like zenodo.org, and then folks will try to look at and use that notebook for years to come. I think literally we need to think 5-10 years here, minimally. The cycles in science are just very, very different than in a more normal software environment. I think one of the core promises of Jupyter notebooks (if it wants to target the scientific space) must be that you can still open notebooks from way back in a recent version of JupyterLab, and everything will display just fine and in the same way it did when the notebook was created. Ideally you could also run it, but that is more complicated. But displaying seems like the minimal bar to me.

I strongly support the ideals above. If you want to support that sort of lifespan, you have to use (or at least convert to) formats that have that sort of lifespan, and right now my understanding from the vega team is that they do not plan on supporting the vega file format on that lifespan. That means use archival static png, jpg, svg, etc. for the plots. That's why I think the strong story here for backwards compatibility is in mirroring vega plots as a static png.

@davidanthoff
Copy link
Contributor

@davidanthoff davidanthoff commented Dec 17, 2019

It seems that nteract is shipping all previous versions of vega and vega-lite out-of-the box, via the https://github.com/nteract/any-vega package. Could jlab maybe utilize that package as well? Presumably if you were to just use that package, then the TS issue would be moot because that package is just pure JS?

But I should also say that from my point of view it would be higher priority that vega-lite 4 is in jlab 2 when it is released than any of this other stuff.

@jasongrout
Copy link
Contributor Author

@jasongrout jasongrout commented Dec 18, 2019

Could jlab maybe utilize that package as well? Presumably if you were to just use that package, then the TS issue would be moot because that package is just pure JS?

Interesting!

But I should also say that from my point of view it would be higher priority that vega-lite 4 is in jlab 2 when it is released than any of this other stuff.

+1 definitely to shipping vega-lite 4.

@lock lock bot added the status:resolved-locked label Jan 18, 2020
@lock lock bot locked as resolved and limited conversation to collaborators Jan 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pkg:vega status:resolved-locked
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants