Jupyter lite to host Docs

[CSSFrancis]

Describe the functionality you would like to see.

As Binder has basically been overwhelmed now that they can't use googles resources it might be worth looking into using Jupyter lite to host some documentation. I'm not entirely sure if this will work but I think as long as it uses hyperspy with the no-cython key it should be able to handle hosting things.

I was going to play around with it a little bit and see if I can get i t

Describe the context

If this works it would be super cool. It allows you to do things like embed interactive plots.

Additional information

This works by taking the underlying python code and converting it using pyoide. I think this should work as numpy dask and matplotlib all work.

I'll try it out but if someone immediately sees something wrong with this you could save me a couple of hours

[ericpre]

The issue is with numba, which is not supported yet on pyoide and other. It may take a bit of time before it is available, but once it will be, that would be great.
Numba is so useful that it will be difficult to make it an optional dependency to most libraries that we use!

[CSSFrancis]

@ericpre Ahh of course... I totally forgot about numba/ was focusing on the cython code. It might be possible once the EELS, EDS and IO code is split out. Kind of depends if rosettasciio actually needs to be a dependency of hyperspy's.

Maybe we lose too much for what are essentially cool documentation purposes though

[ericpre]

In the past (early days of numba), we had a decorator to make numba optional, maybe it is worth reintroducing it! The cython code is optional, so it can be possible to install it using micropip .

[CSSFrancis]

As a data visualization tool it might be worth it :) I'm kind of in awe of pyoide/ jupyterlite and it would be cool to think about the different things you could do like making interactive plotting tools that run in the browser.

For presenting data I would love to be able to have some data or data representations which you could make interactive without requiring people to download hyperspy.

[hakonanes]

For presenting data I would love to be able to have some data or data representations which you could make interactive without requiring people to download hyperspy.

I'm a bit confused, isn't this what Binder does very well? I wasn't aware of that service being overwhelmed. Can you provide a reference?

[CSSFrancis]

https://blog.jupyter.org/mybinder-org-reducing-capacity-c93ccfc6413f

This goes over it. It still runs in a somewhat limited capacity but there is a pretty big funding gap currently.

[hakonanes]

Right, googling brought me here. Thank you for bringing this to my attention, @CSSFrancis.

Have you observed an effect of the reduced capacity? My only current use of Binder is to offer visitors of kikuchipy's docs a link to run our tutorials on Binder. I always ask Binder to "make a new build" from the recent stable release (which it caches for some time), which was made last week. The first build always takes time, but it built as normal.

[ericpre]

Yes, indeed, the first one in a while normally takes time, but recently, the first one doesn't even complete. If you have been lucky enough to get a binder working, if I were you, I would try my luck with a lottery ticket, because I didn't get a binder running even I tried several times to see how bad the situation was. 😉

https://discourse.jupyter.org/c/binder

[CSSFrancis]

Longer term something like jupyter-lite is maybe more sustainable because it uses compute from within the browser.

It seems like numba would be almost impossible to get working but we could have something like hyperspy-lite which runs without numba.

[CSSFrancis]

Right, googling brought me here. Thank you for bringing this to my attention, @CSSFrancis.

Have you observed an effect of the reduced capacity? My only current use of Binder is to offer visitors of kikuchipy's docs a link to run our tutorials on Binder. I always ask Binder to "make a new build" from the recent stable release (which it caches for some time), which was made last week. The first build always takes time, but it built as normal.

It was pretty bad last week. A couple of times last week I tried pyxems, kikuchipy and hyperspys binder links and none of them built. I was trying to write a presentation and serve it from binder and was pretty surprised when it kept failing.

I did get something to build yesterday but I think I was just running at the perfect time! They have started deleting old caches too after some time so that is kind of hit or miss and it tends to build much more than it used to.

Jupyter-lite is still fairly experimental. But there are some potentially cool things like you could make hyperspy-lite sandbox which you could try loading your data locally. There's pretty limited computing as you are operating in your browser though.

It's one step closer to me writing my dream of writing my Thesis as an interactive jupyter notebook though :)

[hakonanes]

If you have been lucky enough to get a binder working, if I were you, I would try my luck with a lottery ticket

I tried again now, waited for about 20 minutes, and actually got through in the end. Lucky me! So I would buy a lottery ticket if the stores in Norway were not closed for the Pentecost holiday...

A couple of times last week I tried pyxems, kikuchipy and hyperspys binder links and none of them built.

That's no good. Thank you for testing. CPU acceleration with Numba is used wherever there is a benefit in kikuchipy, which is nearly everywhere except the IO and a few utilities... That package cannot use Jupyter lite anytime soon, unfortunately.

What one can hope for is that projects which can use Jupyter lite exchanges MyBinder for Jupyter lite. Then, hopefully, projects which cannot still have a tool for demonstration purposes and a way of serving a trial environment in people's browser. Creating awareness for Jupyter lite is important in this regard, so thanks again, @CSSFrancis.

[CSSFrancis]

I tried again now, waited for about 20 minutes, and actually got through in the end. Lucky me! So I would buy a lottery ticket if the stores in Norway were not closed for the Pentecost holiday...

Yay at least it still kind of works! Binder always did seem like it was too good to be true, some hopefully it can continue in some capacity.

That's no good. Thank you for testing. CPU acceleration with Numba is used wherever there is a benefit in kikuchipy, which is nearly everywhere except the IO and a few utilities... That package cannot use Jupyter lite anytime soon, unfortunately.

The good thing about numba is it is fairly modular, so it probably wouldn't be too hard to release a hyperspy-lite (or kikuchipy-lite/pyxem-lite) version which runs without numba. Compiling with Pyodide comes with a pretty substantial performance hit so in most use cases it is fairly useless if you are trying to do more substantial compute. In that case the added performance hit from numba isn't really the biggest problem.

The question is if that would be useful/worthwhile. I might try to test this with hyperspy just as a proof of concept to see if I can get it to work. Doing it just for documentation purposes seems like overkill but it might be worth it if there are potentially some other interesting things we could do.

I think there are a couple of potentially cool applications to this like making web applications for data visualization based on hyperspy. Beyond that I feel like anything more substantial will fail pretty spectacularly but who knows? Maybe I'm underestimating or overestimating modern day browsers.

What one can hope for is that projects which can use Jupyter lite exchanges MyBinder for Jupyter lite. Then, hopefully, projects which cannot still have a tool for demonstration purposes and a way of serving a trial environment in people's browser. Creating awareness for Jupyter lite is important in this regard, so thanks again, @CSSFrancis.

Yea hopefully people will start to switch. It does seem like that might take some time. Maybe in a couple of months binder may come back :) I really hope that is that the case but I'm not terribly optimistic.