Strange micropip failure when loading wheel from custom URL #1653

deeplook · 2021-06-23T07:55:37Z

I'm trying to install a soon-to-be-open-sourced package with zero dependencies in JupyterLite like this, but am getting a weird TypeError: exceptions must derive from BaseException. Any idea what I can check further?

import micropip
url = ("https://example.com/foobar-py2.py3-none-any.whl")
await micropip.install(url)

Traceback (most recent call last):
  File "<console>", line 6, in <module>
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 178, in install
    await gather(*wheel_promises)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 123, in _install_wheel
    wheel = await _get_url(url)
  File "/lib/python3.8/site-packages/micropip.py", line 33, in _get_url
    resp = await fetch(url)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
TypeError: exceptions must derive from BaseException

The text was updated successfully, but these errors were encountered:

hoodmane · 2021-06-23T12:47:17Z

Well it looks like fetch(url) threw a weird object that Pyodide was unable to recognize as an exception. First try await fetch("https://example.com/foobar-py2.py3-none-any.whl") in Javascript and in Pyodide and double check that those succeed.

deeplook · 2021-06-23T18:47:34Z

Ok, thanks! So, await micropip.fetch(url) returns a shorter traceback with the same exception:

Traceback (most recent call last):
  File "<console>", line 6, in <module>
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
TypeError: exceptions must derive from BaseException

While in a Javascript kernel fetch("<SAME_URL>") returns [object Object].

hoodmane · 2021-06-23T18:53:41Z

Try monkey patching fetch before loading Pyodide:

window.orig_fetch = fetch;
window.fetch = async function(...args){
	console.log("fetch called with args:", fetch);
	let resp;
	try {
		resp = await orig_fetch(...args);
	} catch(e) {
		console.log("fetch threw:", e);
		throw e;
	}
	console.log("fetch returned:", resp);
	return resp;
}

What does it say?

deeplook · 2021-06-23T18:55:52Z

[object Object]

fetch called with args: async function(...args){
	console.log("fetch called with args:", fetch);
	let resp;
	try {
		resp = await orig_fetch(...args);
	} catch(e) {
		console.log("fetch threw:", e);
		throw e;
	}
	console.log("fetch returned:", resp);
	return resp;
}
fetch threw: TypeError: NetworkError when attempting to fetch resource.

hoodmane · 2021-06-23T18:56:22Z

So it did await fetch(fetch)??

hoodmane · 2021-06-23T18:57:50Z

And it seems to have thrown a perfectly reasonable error object in response, so I don't know why Pyodide isn't converting it to a valid Python exception...

hoodmane · 2021-06-23T18:58:11Z

Very mysterious.

hoodmane · 2021-06-23T19:00:59Z

Could you provide a reproduction? This behavior seems to suggest a serious bug -- logic errors aside I would hope that it could fail a bit more gracefully....

deeplook · 2021-06-23T19:01:08Z

You can try with the URL behind this short link: http://s.here.com/TeGPQR which is publically reachable.

hoodmane · 2021-06-23T19:02:43Z

So I just try to load that wheel on https://pyodide-cdn2.iodide.io/v0.17.0/full/console.html?

hoodmane · 2021-06-23T19:03:45Z

Okay it reproduces thanks. I'll investigate and get back to you.

deeplook · 2021-06-23T19:04:17Z

Thanks!

hoodmane · 2021-06-26T18:55:18Z

So the problem is that fetch is throwing a network error which for some reason the reflection code in jsproxy.c fails to detect as an error object, but only in firefox. In chrome it works fine. The reason the NetworkError is being thrown in the first place is probably because of a problem with coors. Anyways this bug is also only present in v0.17.0, on main Firefox handles this correctly. I tweaked the logic in hiwire_is_error recently due to a different bug and that tweak seems to have fixed this.

Closed as fixed on main.

deeplook · 2021-06-29T10:26:23Z

Thanks for your efforts! If I understand correctly you didn't observe any issue on Chrome. I gave it a try, too, using Version 90.0.4430.212 (Official Build) (x86_64) on macOS 10.14.6 and I get his even more impressive traceback below. Can you shed some more light on this, please?

Traceback (most recent call last):
  File "<console>", line 5, in <module>
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 178, in install
    await gather(*wheel_promises)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 123, in _install_wheel
    wheel = await _get_url(url)
  File "/lib/python3.8/site-packages/micropip.py", line 33, in _get_url
    resp = await fetch(url)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
JsException: TypeError: Failed to fetch
PythonError: Traceback (most recent call last):
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/pyolite/interpreter.py", line 25, in run
    result = await self.run_complete
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/pyodide/console.py", line 253, in load_packages_and_run
    raise e
  File "/lib/python3.8/site-packages/pyodide/console.py", line 247, in load_packages_and_run
    result = await eval_code_async(
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 419, in eval_code_async
    return await CodeRunner(
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 278, in run_async
    res = await res
  File "<console>", line 5, in <module>
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 178, in install
    await gather(*wheel_promises)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 123, in _install_wheel
    wheel = await _get_url(url)
  File "/lib/python3.8/site-packages/micropip.py", line 33, in _get_url
    resp = await fetch(url)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
JsException: TypeError: Failed to fetch

    at new_error (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.js:9:175413)
    at wrap_exception (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[297]:0x1a6c94)
    at FutureDoneCallback_call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[379]:0x1aaab3)
    at byn$fpcast-emu$FutureDoneCallback_call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[14420]:0x7a257c)
    at _PyObject_MakeTpCall (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[760]:0x1cabed)
    at __static_1198 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[2906]:0x2c54f1)
    at byn$fpcast-emu$__static_1198 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[22640]:0x7bf866)
    at __static_678 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[1621]:0x21220e)
    at byn$fpcast-emu$__static_678 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[22120]:0x7bdce6)
    at PyVectorcall_Call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[763]:0x1cb006)

hoodmane · 2021-06-29T12:16:43Z

Yeah on tip of tree I think there is no problem, on v0.17.0 there is one. JsException: TypeError: Failed to fetch is the correct response (unlike TypeError: exceptions must derive from BaseException). I believe that the fetch request is failing due to COORS. When I tested it on my computer in Javascript fetch(yoururl) raises TypeError: Failed to fetch in Chrome and TypeError: NetworkError when attempting to fetch resource. in Firefox.

Does fetch(yoururl) in Javascript succeed for you?

deeplook · 2021-06-29T16:49:28Z

In a JS kernel on Chrome I get for both, fetch(myurl) as well as fetch("https://spatial.utk.edu/download/pyshp-2.1.3-py3-none-any.whl") the result [object Object].

hoodmane · 2021-06-29T16:50:30Z

Hmmmmm. Okay, I think I was never reproducing this then.

hoodmane · 2021-06-29T16:53:10Z

So can you reproduce it in https://pyodide-cdn2.iodide.io/v0.17.0/full/console.html and or https://pyodide-cdn2.iodide.io/dev/full/console.html? There I get COORS errors for your URLs I think. To allow the request to succeed do you need special HTTP headers?

deeplook · 2021-06-29T17:01:39Z

Both give me JsException: TypeError: Failed to fetch for both URLs.

deeplook · 2021-06-29T17:04:17Z

No special headers needed.

hoodmane · 2021-06-29T17:05:31Z

So if you use fetch directly in the browser console it works?

hoodmane · 2021-06-29T17:06:35Z

For f = await fetch("https://spatial.utk.edu/download/pyshp-2.1.3-py3-none-any.whl") Firefox says:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://spatial.utk.edu/download/pyshp-2.1.3-py3-none-any.whl. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

Uncaught (in promise) TypeError: NetworkError when attempting to fetch resource.

deeplook · 2021-06-29T17:12:10Z

My Firefox Developer Edition doesn't...

hoodmane added bug Something isn't working micropip labels Jun 24, 2021

hoodmane changed the title ~~Getting TypeError for zero dependency package~~ Strange micropip failure when loading custom wheel URL Jun 24, 2021

hoodmane changed the title ~~Strange micropip failure when loading custom wheel URL~~ Strange micropip failure when loading wheel from custom URL Jun 24, 2021

hoodmane added core / type conversions and removed micropip labels Jun 25, 2021

hoodmane closed this as completed Jun 26, 2021

deeplook mentioned this issue Jun 28, 2021

Install on Pyodide/JupyterLite opengeos/leafmap#55

Closed

hoodmane reopened this Jun 29, 2021

rth mentioned this issue Mar 30, 2023

requests not working in Firefox koenvo/pyodide-http#22

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strange micropip failure when loading wheel from custom URL #1653

Strange micropip failure when loading wheel from custom URL #1653

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 26, 2021 •

edited

deeplook commented Jun 29, 2021

hoodmane commented Jun 29, 2021

deeplook commented Jun 29, 2021

hoodmane commented Jun 29, 2021

hoodmane commented Jun 29, 2021

deeplook commented Jun 29, 2021

deeplook commented Jun 29, 2021

hoodmane commented Jun 29, 2021

hoodmane commented Jun 29, 2021

deeplook commented Jun 29, 2021

Strange micropip failure when loading wheel from custom URL #1653

Strange micropip failure when loading wheel from custom URL #1653

Comments

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 23, 2021

hoodmane commented Jun 23, 2021

deeplook commented Jun 23, 2021

hoodmane commented Jun 26, 2021 • edited

deeplook commented Jun 29, 2021

hoodmane commented Jun 29, 2021

deeplook commented Jun 29, 2021

hoodmane commented Jun 29, 2021

hoodmane commented Jun 29, 2021

deeplook commented Jun 29, 2021

deeplook commented Jun 29, 2021

hoodmane commented Jun 29, 2021

hoodmane commented Jun 29, 2021

deeplook commented Jun 29, 2021

hoodmane commented Jun 26, 2021 •

edited