Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy both py and pyc sources #3407

Closed
rth opened this issue Jan 3, 2023 · 2 comments
Closed

Deploy both py and pyc sources #3407

rth opened this issue Jan 3, 2023 · 2 comments

Comments

@rth
Copy link
Member

rth commented Jan 3, 2023

I think the outcome of #3269 is that while .pyc sources are a huge load time improvement, for some use-cases (e.g. teaching) full errors tracebacks is more critical than load speed. So the ideal case would be to distribute both versions.

As mentioned by @hoodmane in #3051 (comment)

Ideally I think it ought to be a setting for loadPyodide whether to use the pyc files or the full ones. We should also probably look further into the possibilities for fixing up tracebacks, if it's reasonable we might offer an API for it.

For the topic of fixing the tracebacks with the pyc version, it still means that the source code of the py version needs to be deployed somewhere. I mean even if it turns out that the best way is some other format for source code (e.g. so a particular line could be fetched via an HTTP range request when the exception happens), in end I feel like aside from this traceback loading mechanism, there should be a way to look at actual source code we distribute if there is any doubt for debugging without having to rebuild anything.

There are several ways both versions could be deployed,

  1. Using a different suffix in the indexURL e.g. (v0.22.0/full/ vs v0.22.0/pyc/). There could be an option in loadPyodide to make switching easier.
  2. Using the same indexURL, where both versions are deployed. The problem here is that for packages, if wheels are in the same folder, it's difficult to differentiate pyc compiled wheel from the original wheel (e.g. for numpy it would be the same filename), also one would need two repodata.json. So in the end IMO option 1 would be easier.

For package wheels #3253 it would be trivial to build both at the same time. For CPython itself #3166 I guess we could also do it in the build core. The question is though which version we test (testing both would be too slow) and how we make sure that the version we don't test is not broken.

@ryanking13
Copy link
Member

I also think option 1 is better.

The question is though which version we test (testing both would be too slow) and how we make sure that the version we don't test is not broken.

If we would want to test only one version, I think testing .pyc compiled version is more reasonable. If it works on .pyc version, I think it will hopefully also work on .py version, but the opposite might not be true. For example, if a package has a hardcoded path to .py file inside the source code, it might not work in compiled versions.

@rth
Copy link
Member Author

rth commented Mar 29, 2023

Resolved in #3701

@rth rth closed this as completed Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants