Update upstream URL for libjpeg. NFC

[sbc100]

The old URL seems to be generating Forbidden!.

I wonder if we should start maintaining our own mirror to avoid
this kind of thing.

[sbc100]

Landing TBR to unblock CI

[kripken]

Yes, if there is no unreliable upstream repo for this, we should create a mirror in emscripten-ports and use that.

[rth]

Thanks!

The actual error for this was,

system_libs:INFO: retrieving port: libjpeg from https://dl.bintray.com/homebrew/mirror/jpeg-9c.tar.gz
[...]
urllib.error.HTTPError: HTTP Error 403: Forbidden

in case someone searches for it on Github and is not able to find the relevant issue.

[rth]

Are you sure this fixes it though @sbc100 ? I'm still getting the same error with the new URL if requests is not installed (which worked fine previously I think), and I can also reproduce it from a Python REPL directly,

>>> from tools import system_libs
>>> system_libs.Ports().fetch_project('libjpeg', 'https://www.ijg.org/files/jpegsrc.v9c.tar.gz', 'jpeg-9c')
system_libs:INFO: retrieving port: libjpeg from https://www.ijg.org/files/jpegsrc.v9c.tar.gz
Traceback (most recent call last):
  File "/src/emsdk/emsdk/upstream/emscripten/tools/system_libs.py", line 1707, in retrieve
    import requests
ModuleNotFoundError: No module named 'requests'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/src/emsdk/emsdk/upstream/emscripten/tools/system_libs.py", line 1756, in fetch_project
    retrieve()
  File "/src/emsdk/emsdk/upstream/emscripten/tools/system_libs.py", line 1712, in retrieve
    f = urlopen(url)
  File "/usr/local/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

If requests is installed this indeed fetches the .tar.gz correctly.

[sbc100]

No requests is not needed.

The error here is "urllib.error.HTTPError: HTTP Error 403: Forbidden".

Can you run wget https://www.ijg.org/files/jpegsrc.v9c.tar.gz or curl https://www.ijg.org/files/jpegsrc.v9c.tar.gz or load that URL in your browser?

[rth]

All of curl, wget and that URL in the browser work.

The issue seems to be with urllib specifically which is used in fetch_project,

>>> from urllib.request import urlopen
>>> res = urlopen("https://www.ijg.org/files/jpegsrc.v9c.tar.gz")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

This happens both in CI and locally so it shouldn't be location dependent.

If a different user-agent is passed in headers, then the download works. So they seem to be blocking the urllib user agent.

[sbc100]

How odd... seems like maybe misconfigured webserver?

[sbc100]

I'll mirror the archive on google cloud storage

[rth]

They probably added it to the deny list for some reason mesonbuild/libjpeg#9 (comment) . Thanks!