Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JupyterLite-based shell at https://www.sympy.org/en/shell.html #174

Merged
merged 6 commits into from
May 8, 2022

Conversation

ivanistheone
Copy link
Contributor

@ivanistheone ivanistheone commented Apr 19, 2022

This is a derivative PR of the original work by @jptio PR#169.

  • (jeremy) Proof of concept use JupyterLite and Pyodide for the online shell PR169
  • (ivan) Changed iframe src to use jinja variables
  • (aaron) Merge Fixes to mobile menu logic to work on window resize sympy-web-static#13
  • (aaron) Merge make sidebar an overwritable block #172
  • create this PR supersedes PR169
  • add init_printing() to the initial list of executed commands
  • Edit sidebar to add new "about this page" box to explain that this runs entirely in the browser
  • Add support for ?evaluate= query string
  • Add loading overlay to explain it takes a few seconds to load
  • Make example session match the new console style (use Jupyter inputs [1]: [2]: ).
  • Load mathjax in templates/shell.html (so examples in sidebar can be in LaTeX)

@certik
Copy link
Member

certik commented Apr 19, 2022

@ivanistheone thanks! Please ping me once this is ready.

@ivanistheone
Copy link
Contributor Author

I added support for ?evaluate= urls. Here are some test URLs to try:

Still working on webcopy in sidebar, will take of WIP when that is done.

@ivanistheone
Copy link
Contributor Author

ivanistheone commented Apr 19, 2022

@jtpio I ran into an interesting edge case when opening the console in a background tab, jupyterlite seems to think there is no Kernel:

Screen Shot 2022-04-19 at 12 23 10 PM

To reproduce:

Question 1: I was wondering if you might have seen this before and if there is a workaround to avoid this.

@ivanistheone
Copy link
Contributor Author

ivanistheone commented Apr 19, 2022

Hi @jtpio I was looking at the js console while the JupyterLite shell is loading ...

loading 1

loading 2

loading 3

loading 4

loading 5

... and I have another question for you:

Question 2: Is there a way to customize the loading process to skip certain packages (e.g. skip matplotlib)?

Currently it takes 22 seconds to load the SymPy shell on my computer (desktop, medium-fast internet connection), so I was wondering if there might be a way to shave a few seconds off of that by customizing the code in jupyterlite/demo. (Not for this PR but for followup perf. optimization work).

@asmeurer
Copy link
Member

We don't need to block this on anything that can be fixed upstream in jupyterlite, because we can fix those things upstream after this is merged and the site will automatically get the improvements.

@jtpio
Copy link
Contributor

jtpio commented Apr 20, 2022

Question 1: I was wondering if you might have seen this before and if there is a workaround to avoid this.

I might have seen it once before. Looks like this should be reproducible with https://jupyterlite.github.io/demo/repl/ or https://jupyterlite.rtfd.io/en/latest/try/repl?

Looks like a race condition with the kernels not having the time to register themselves before the application starts. Would you mind opening an issue in https://github.com/jupyterlite/jupyterlite ?

Question 2: Is there a way to customize the loading process to skip certain packages (e.g. skip matplotlib)?

Yes there was an attempt at delaying this import in the past but it caused some issues with the recursion limit in Pyodide.
This is currently tracked in jupyterlite/jupyterlite#254.

@jtpio
Copy link
Contributor

jtpio commented Apr 21, 2022

Looks like a race condition with the kernels not having the time to register themselves before the application starts. Would you mind opening an issue in https://github.com/jupyterlite/jupyterlite ?

Opened jupyterlite/jupyterlite#610 to track it upstream.

@jtpio
Copy link
Contributor

jtpio commented May 2, 2022

Opened jupyterlite/jupyterlite#610 to track it upstream.

@ivanistheone This should now be fixed in 0.1.0b6: https://github.com/jupyterlite/jupyterlite/releases/tag/v0.1.0b6

With the demo repo updated to use the latest release: https://jupyterlite.github.io/demo/repl/?toolbar=1&kernel=python

@asmeurer
Copy link
Member

asmeurer commented May 3, 2022

What's the status of this? I'd love to have SymPy Live phased out before we do the next SymPy release since the next SymPy release will also completely remove it from the documentation. At that point, we can shut down the App Engine server, and add a note to the repo that if someone still wants to run it, they should deploy it to the App Engine themselves.

@ivanistheone
Copy link
Contributor Author

What's the status of this?

Yeah sorry for delay. I just have to fix the webcopy then we can merge this, which will conclude PHASE 1. I'll work on this tomorrow and update the demo link for review.

I'd love to have SymPy Live phased out before we do the next SymPy release since the next SymPy release will also completely remove it from the documentation. At that point, we can shut down the App Engine server, and add a note to the repo that if someone still wants to run it, they should deploy it to the App Engine themselves.

As soon as we ship this PR, I'll work on PHASE 3 and can provide and time estimate then. @asmeurer @certik Is there a target date in mind for the next release?

Since the new docs have removed the shell component, live.sympy.org is an independent service so could be [deployed](What's the status of this?

Yeah sorry for delay. I just have to fix the webcopy and we can merge this, which will conclude PHASE 1.

I'd love to have SymPy Live phased out before we do the next SymPy release since the next SymPy release will also completely remove it from the documentation. At that point, we can shut down the App Engine server, and add a note to the repo that if someone still wants to run it, they should deploy it to the App Engine themselves.

As soon as we ship this PR, I'll work on PHASE 3 and can provide and time estimate then. @asmeurer @certik Is there a target date in mind for the next release?

Since the new docs have removed the shell component, live.sympy.org is an independent service so could be launched independently of the next SymPy release.

I'm thiniking at least a few weeks using the (upcoming) http://sympy.org/en/shell.html for testing to make sure it works.

if bool(PHASE2) then PHASE4().) independently of the next SymPy release.

I'm thiniking at least a few weeks using the (upcoming) http://sympy.org/en/shell.html for testing to make sure it works. In other words, if bool(PHASE2) then PHASE4().

@ivanistheone
Copy link
Contributor Author

@jtpio I've been doing some testing on the demo link from various browsers and it works on most of them (on an old iPhone).

Only one that didn't work was Safari (on iOS medium old). Is there something about Safari on iOS that doesn't allow wasm?

doesntloadOniOS

@asmeurer
Copy link
Member

asmeurer commented May 3, 2022

I didn't know about that Google Doc with the phase plan. The plan looks good to me.

As soon as we ship this PR, I'll work on PHASE 3 and can provide and time estimate then. @asmeurer @certik Is there a target date in mind for the next release?

The person to ask here is @oscarbenjamin.

The SymPy release and SymPy Live updates don't necessarily have to happen in tandem, but it's worth noting that the next release of SymPy will remove it from the docs, so we can't shut down the AppEngine at least until then.

Also, how hard would it be to add some text to the AppEngine version that it is being deprecated and redirecting people to the new version? That way there will be some warning for people before it is shut down completely.

- Changed iframe src to use jinja variables
- Updated sidebar webcopy
- Added temporary instructions to run using SHIFT+ENTER and play button
  on mobile
- Added js code to handle evaluare query string params
- Added extrahead block in base.html (in order to allow inclusion of
  page-specific javascripts
- Wrapped all text in trans-endtrans blocks
- add init_printing() to the initial list of executed commands
- add loading overlay for 18 seconds (avg. loading time is 22 secs, but
  maybe someone is on fast internet and beast silicon)
@ivanistheone ivanistheone changed the title [WIP] Jupyterlite shell prime JupyterLite-based shell at https://www.sympy.org/en/shell.html May 4, 2022
@jtpio
Copy link
Contributor

jtpio commented May 4, 2022

Thanks @ivanistheone!

Only one that didn't work was Safari (on iOS medium old). Is there something about Safari on iOS that doesn't allow wasm?

Were you able to get some information from the dev tools console? Curious if this might be an issue with WebAssembly in general, or maybe Pyodide.

In any case this would not be a blocker for this PR and could be investigated independently.

@ivanistheone
Copy link
Contributor Author

ivanistheone commented May 4, 2022

Okay all PHASE 1 work done; @asmeurer @certik ready for review.

Summary of changes:

  • original SypPy-in-JupyterLab work from PR#169
  • Followup work (see commit message for details: 600b517

Screenshot:
webcopy screenshot

If you have edit-suggestions for webcopy of sidebar, please edit the text in the google doc and I will update the html in templates/shell.html based on the gdoc.

@ivanistheone
Copy link
Contributor Author

Were you able to get some information from the dev tools console? Curious if this might be an issue with WebAssembly in general, or maybe Pyodide.

@jtpio I managed to connect to get a Safari debug console from destkop; I put some screenshots of errors here:
jupyterlite/jupyterlite#632

@asmeurer
Copy link
Member

asmeurer commented May 4, 2022

On iOS Safari (latest iOS), I get this error

Traceback (most recent call last):

  File /lib/python3.10/site-packages/IPython/core/interactiveshell.py:3397 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Input In [1] in <cell line: 1>
    from sympy import *

  File /lib/python3.10/site-packages/sympy/__init__.py:73 in <module>
    from .polys import (Poly, PurePoly, poly_from_expr, parallel_poly_from_expr,

  File /lib/python3.10/site-packages/sympy/polys/__init__.py:75 in <module>
    from .polyfuncs import (symmetrize, horner, interpolate,

  File /lib/python3.10/site-packages/sympy/polys/polyfuncs.py:11 in <module>
    from sympy.polys.specialpolys import (

  File /lib/python3.10/site-packages/sympy/polys/specialpolys.py:7 in <module>
    from sympy.functions.elementary.miscellaneous import sqrt

  File /lib/python3.10/site-packages/sympy/functions/__init__.py:10 in <module>
    from sympy.functions.combinatorial.numbers import (carmichael, fibonacci, lucas, tribonacci,

  File /lib/python3.10/site-packages/sympy/functions/combinatorial/numbers.py:27 in <module>
    from sympy.functions.elementary.trigonometric import sin, cos, cot

  File /lib/python3.10/site-packages/sympy/functions/elementary/trigonometric.py:3302
    return sqrt(arg**2)/arg*(S.Pi/2 - acot(1/sqrt(arg**2 - 1)))
                                                             ^
SyntaxError: invalid syntax

@asmeurer
Copy link
Member

asmeurer commented May 4, 2022

What is the caching story here? For me, whenever I reload the page, it always takes a few seconds to load. Is the "loading" progress bar just tracking downloading packages or is it also including import time? I don't think we need to have one for import time.

Also, if it is possible, it would better to have the loading bar work in a way that lets people enter code while it is loading and have it run as soon as the loading happens.

@asmeurer
Copy link
Member

asmeurer commented May 4, 2022

Is it hard to change the submit button from shift-enter to just enter? I suppose that would break multiline code, unless it uses something similar to IPython where it adds a newline if the code is multiline and submits otherwise.

@ivanistheone
Copy link
Contributor Author

On iOS Safari (latest iOS), I get this error
Traceback (most recent call last):
File /lib/python3.10/site-packages/sympy/functions/elementary/trigonometric.py:3302
return sqrt(arg2)/arg*(S.Pi/2 - acot(1/sqrt(arg2 - 1)))
SyntaxError: invalid syntax

I ran into a similar error in Opera on iOS and opened an issue here: jupyterlite/jupyterlite#633


What is the caching story here?

Good question. I also get the same loading times whether I CTRL+R (refresh) or CTRL+SHIFT+R (clear cache and refresh). I defer to @jtpio to say if there is something we can do. Not a blocker for this PR, but would be nice to bring down the loading time in future updates.


Also, if it is possible, it would better to have the loading bar work in a way that lets people enter code while it is loading and have it run as soon as the loading happens.

The loading overlay is a complete hack (hard coded 18 sec delay). Basically, I figured since users have to wait O(20 sec) for the shell to become interactive, we better block them for some time to let them know it's loading. Should we reduce this to 10 seconds? (after the loading screen is gone, you can enter commands as per your suggestion)


Is it hard to change the submit button from shift-enter to just enter?

I was looking into that today. I created ivanistheone/live from the jupyterlite/demo template and I was looking into the various config options, but I don't know how to customize keypress yet.

@jtpio
Copy link
Contributor

jtpio commented May 5, 2022

I was looking into that today. I created ivanistheone/live from the jupyterlite/demo template and I was looking into the various config options, but I don't know how to customize keypress yet.

@ivanistheone I opened sympy/live#1 to use Enter for executing code, and also enable some app optimizations by only deploying the repl app.

@jtpio
Copy link
Contributor

jtpio commented May 5, 2022

but would be nice to bring down the loading time in future updates.

Yes I'm hoping we can add the import hooks back upstream in JupyterLite: jupyterlite/jupyterlite#254
So packages like matplotlib would not be imported at startup, but only when needed. This should already help reduce the loading time.

@ivanistheone
Copy link
Contributor Author

Update from this morning.

Thank's to Jeremy's PRs sympy/live#1, sympy/live#2, sympy/live#3, we now have custom JupyterLite with SUBMIT on ENTER key bindings (and COMMAND+ ENTER for newlines).

You can see it here: https://ivanistheone.github.io/live/repl/?toolbar=1&kernel=python but we want this to run here: https://sympy.github.io/live/repl/?toolbar=1&kernel=python before updating any links.

I would like to tansfer ownership of https://github.com/ivanistheone/live to https://github.com/sympy today, but I will need someone from the sympy org to give me collaborator access to the repo after the transfer so I can continue working on it.

@asmeurer @certik Could we do this today, if not I will continue working on ivanistheone for now.

@ivanistheone
Copy link
Contributor Author

Updates:

Demo link as usual: https://ivanistheone.github.io/sympy.github.com/en/shell.html

Waiting for further webcopy edits/suggestions and/or testing. We also discussed getting rid of the loading overlay completely, but I think it's useful to have and 3 secs is not long. If no further changes, then we ship it!

@asmeurer
Copy link
Member

asmeurer commented May 6, 2022

The disappearing message means the content shifts up. I would rather just have it below the content and not disappearing.

@asmeurer
Copy link
Member

asmeurer commented May 6, 2022

sympy.org/live now just redirects to the notebook rather than the whole page. If we are going to have that as a page, we should make it link to the actual live page, or redirect to live.sympy.org.

@asmeurer
Copy link
Member

asmeurer commented May 6, 2022

I still prefer to not have the loading overlay. I can see it load underneath the loading indicator before it finishes, meaning I have to wait 3 seconds even if I wouldn't have to. Having a loading indicator is fine, but my opinion is it would be better to have it in some way that does not disable access to the content.

@asmeurer
Copy link
Member

asmeurer commented May 6, 2022

I'm +1 for shipping this once you think it is ready. You should have push access to this repo.

@ivanistheone
Copy link
Contributor Author

I still prefer to not have the loading overlay.

Gone.

The disappearing message means the content shifts up. I would rather just have it below the content and not disappearing.

Moved.

For the record, I think the disappearing message on top would have been better UX (more prominent to tell user what is going on), but I don't feel strongly about it.
removed dissapearing message on top
Fixed message below is good enough... and we can tweak it further if need be.

sympy.org/live now just redirects to the notebook rather than the whole page. If we are going to have that as a page, we should make it link to the actual live page, or redirect to live.sympy.org.

Is https://sympy.org/live linked to from somewhere currently? I've added a TODO to look into this in next phase.

@ivanistheone
Copy link
Contributor Author

I think this one is ready to merge! I see the Merge pull request button is available to me, so I can merge myself if there are no further changes requested.

Demo here as usual: https://ivanistheone.github.io/sympy.github.com/en/shell.html

Copy link
Member

@certik certik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I played with it, it seems to work well. It's a little slow to load, but once it loads, it's quick.

The code seems ok.

Let's wait for @asmeurer to give it a final approval though. But as far as I am concerned, this is good to go.

Thanks @ivanistheone for your hard work!

@asmeurer
Copy link
Member

asmeurer commented May 8, 2022

Yes, let's go ahead and merge this. We can make improvements as new pull requests.

@ivanistheone ivanistheone merged commit 3848dd0 into sympy:sources May 8, 2022
@ivanistheone
Copy link
Contributor Author

ivanistheone commented May 8, 2022

Okay merged.


When testing the evaluate? query strings, I noticed a problem with the redirect sympy.github.io --> www.sympy.org that was dropping repeated query string parameters. E.g. sympy.github.io?code=a&code=b will get redirected to www.sympy.org?code=a dropping the code=b.

I pushed a hotfix PR #175 changing the URL we use for the shell to be www.sympy.org so now there are no more redirects and functionality works as expected.


On an unrelated note: I noticed a non-critical error in the JS console (it doesn't seem to break any functionality; I opened an issue for it on jupyterlite repo jupyterlite/jupyterlite#635 in case want to track it down)

@asmeurer
Copy link
Member

asmeurer commented May 9, 2022

Great. Can you send an announcement to the mailing list?

@ivanistheone
Copy link
Contributor Author

ivanistheone commented May 10, 2022

Can you send an announcement to the mailing list?

I was going to send an email announcement + request for testing, but I ran into an issue jupyterlite/jupyterlite#636 so I won't post today. Hopefully it's just a transient error.

I will send announcement when fixed.

Flagging jupyterlite/jupyterlite#636 for @jtpio to triage appropriately.

@asmeurer
Copy link
Member

asmeurer commented May 10, 2022

OK. We don't need to have things in a perfect state to send an announcement about the soft launch, especially since the site is live anyway, so people are already going to start going to it. And it was already announced on Twitter.

@ivanistheone
Copy link
Contributor Author

Issue#636 was transient; everything works fine now.

I sent the announcement to the mailing list.

Thanks everyone who has been part of this so far. I'm very glad we're progressing toward latest-SymPy-live-shell-in-the-browser-with-almost-no-maintenance future.

@asmeurer
Copy link
Member

I haven't heard any negative feedback on this yet. I guess we should move live.sympy.org to it as well. Then once we do a SymPy release with the new Sphinx theme that removes the Live extension, we can shut Live down (or at least remove the billing).

@asmeurer
Copy link
Member

Maybe doing this just requires updating the DNS for live.sympy.org?

@asmeurer
Copy link
Member

For example, here is how it can be done with namecheap:

Screen Shot 2022-07-26 at 3 12 47 PM

See http://live.asmeurer.com/. Except I don't know how to make it use https. I think NumFOCUS handles the sympy.org DNS. @certik do you know what needs to be done to change this?

@ivanistheone
Copy link
Contributor Author

Hi @asmeurer ; Is there a reason you're looking for a new plan? The previously agreed upon approach was to wait to use the https://sympy.org/en/shell.html for a testing period, then deploy the repo https://github.com/sympy/live as github pages then point live.sympy.org to the github pages (PHASE4).

I haven't pushed on this recently because I've asked several people to test https://sympy.org/en/shell.html and I've had very few good reviews. Desktop browsers work fine, but it doesn't seem to be usable on mobile (not loading at all on older browsers and takes too long on modern). This seemed like a show stopper to me, that's why I haven't pushed on the migration recently...

Perhaps we can agree that supporting mobile users and legacy browsers are out of scope? If we agree on that then, I can resume work on the roadmap we discussed before: PHASE3 and PHASE4.

Also, I'm curious. Dos someone know what's the current GAE hosting bill for running live.sympy.org ?
Is there a way to access server logs to see what percentage of user-agents are mobile phones? (perhaps there are not so many mobile users, so it won't be letting anyone down).

@asmeurer
Copy link
Member

I'm not pushing for a new plan. But I do think this has been live long enough that we can consider the testing period to be over.

It looks like Live is costing us about $20-30 a month (Gamma is much more). It's hard to tell, but I think the costs have declined a little bit since deploying this, but not a whole lot. I expect most users of Live are coming from the docs.

The reason I am pushing for this is that I want to shut down Live. As soon as we do the next SymPy release (which will be soon), it will remove the Live extension from the docs, so the only thing remaining will be live.sympy.org.

I have full access to GAE, but I have no idea how to extract that information from the logs (the Google Cloud console is very confusing). I think the information is there (user agent), but I don't know how to easily extract it and I don't really have the time to figure it out. I'm happy to give you permissions on GAE if you want to look into it yourself.

@certik
Copy link
Member

certik commented Jul 27, 2022

Yes, I know how to change the dns records. Just ping me when you need to change them and to what.

The SymPy Live works reasonably well. I just tested it on Firefox Apple M1, and on Firefox iPhone, both works.

@ivanistheone
Copy link
Contributor Author

Okay I have an update on the user agent distribution (based on data from GAE access log).

/1/ First off, and perhaps most importantly, the service receives approx. 1 request per minute on average, which means live.sympy.org is not that popular, and we won't be letting thousands of people down if something breaks with the switch WASM. (We should really promote live.sympy.org shell more though—SymPy is such a good thing for learning math, and the live shell doesn't require any installation... I'm adding a mental note to do more promotion post-launch. I guess the new cleaned up docs will help too...)

As for mobile users, I did a rudimentary analysis on the access log from the past six months found that:

/2/ Mobile users are only 6.6% of traffic:
six point six percent mobile

/3/ Within the mobile users, most of them use recent versions of Android and iOS:
mostly modern mobile OSs

All this means my fears about "letting users down" were unfounded and we can go ahead with the rest of the migration of live.sympy.org. I'll look into the github actions scripts to make this happen and I'll ping y'all here again when there is something to test.

@asmeurer
Copy link
Member

/1/ First off, and perhaps most importantly, the service receives approx. 1 request per minute on average, which means live.sympy.org is not that popular, and we won't be letting thousands of people down if something breaks with the switch WASM. (We should really promote live.sympy.org shell more though—SymPy is such a good thing for learning math, and the live shell doesn't require any installation... I'm adding a mental note to do more promotion post-launch. I guess the new cleaned up docs will help too...)

Hopefully that also means we won't be letting too many people down by removing it from the docs as well. Actually SymPy Gamma is much more popular (at least based on the billing rates), which is unfortunate because it's not so straightforward to replace it with a pyiodide equivalent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants