how to change default mirror url pypi.python.org/simple/ with devpi-server

[robertding]

i start devpi-server version 4.3.0 as cache server
with cmd devpi-server --serverdir /root/packages/ --port 5555 --host=0.0.0.0 --start
how can i change the default cache mirror?

[fschulze]

What do you mean by "default cache mirror"? Do you mean the URL pip uses, or the URL for the PyPI cache?

[robertding]

sorry i didn't make it clear. i want to change the default url pypi.python.org/simple/ to another mirror url like mirrors.aliyun.com when a package not accessable in my devpi server.

[robertding]

@fschulze sorry i dont exactly know the differnet between pip uses(client side usage?) and PyPi cache.
I tried to modify the pip.conf in my devpi server, but it didn't work. The devpi server still tring to get missing package from pypi.python.org which is far slow in my country.
I tried to modify the source code config.py under devpi_server, still not working. Thanks for your help.

[fschulze]

You can change mirror_url on the root/pypi index. Something like this should work (from my head, untested): devpi index root/pypi mirror_url="...". It must point to the simple index of the mirror.

[fschulze]

Did my answer solve your issue?

[blackholll]

i have the same requirement
is this contol download package from ?
in you reply, how to "change mirror_url on the root/pypi index"?

server/devpi_server/extpypi.py line 118?

@fschulze

class PyPIStage(BaseStage): def __init__(self, xom, username, index, ixconfig): super(PyPIStage, self).__init__(xom, username, index, ixconfig) self.httpget = self.xom.httpget # XXX is requests/httpget multi-thread safe? self.cache_expiry = self.ixconfig.get( 'mirror_cache_expiry', xom.config.args.mirror_cache_expiry) self.xom = xom if xom.is_replica(): url = xom.config.master_url self.mirror_url = url.joinpath("%s/+simple/" % self.name).url else: url = URL(self.ixconfig['mirror_url']) self.mirror_url = url.asdir().url

[blackholll]

and i have try to change mirror_url in main.py/_pypi_ixconfig_default/, it doesn't work

[fschulze]

You don't have to change code, use the devpi client to set the option on the index. For example to change the url for root/pypi:

devpi index root/pypi mirror_url="..."

[blackholll]

@fschulze i think you have not understand my mean. normally, when someone install a package that haven‘t cached in devpi server, the devpi server when download the package from pipy.python.org. but we are in china, the speed is slow. so i expect the devpi server download package from pypi.doubanio.com/simple but not pypi.python.org. this should be work on devpi server but not client.

[fschulze]

That's exactly what I'm saying. If you run devpi index root/pypi you should get something like this:

% devpi index root/pypi
https://d.rzon.de:8141/root/pypi:
  type=mirror
  volatile=False
  mirror_url=https://pypi.python.org/simple/
  mirror_web_url_fmt=https://pypi.python.org/pypi/{name}
  title=PyPI

Now you run devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/" and you should get something like:

% devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/"
/root/pypi changing mirror_url: https://pypi.doubanio.com/simple/
https://d.rzon.de:8141/root/pypi:
  type=mirror
  volatile=False
  mirror_url=https://pypi.doubanio.com/simple/
  mirror_web_url_fmt=https://pypi.python.org/pypi/{name}
  title=PyPI

Now packages will be downloaded from the mirror instead of regular pypi.

[blackholll]

@fschulze
I run devpi server by supervisor, and the command is :" /data/virtualenv/devpi/bin/devpi-server --port 404%(process_num)d --serverdir /data/project/devpiserver --outside-url=http://pypi.xxxxx.com".
how to run devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/"? there has no args like 'mirror_url'.

[fschulze]

You need devpi-client for that, it's not a command line option of the server, but a setting in the database of the server, which you can manipulate with devpi-client. See documentation, starting with https://devpi.net/docs/devpi/devpi/stable/+d/quickstart-releaseprocess.html

[blackholll]

ths very much, it works

[pkuwwt]

Not work any more?

devpi index root/pypi mirror_url=“...” get

PATCH http://127.0.0.1:3141/root/pypi
403 Forbidden: Access was denied. To this resource
Unauthorized: PyPIVIEW failed permission check

[pkuwwt]

Oh, I should

devpi-server --passwd root
devpi-server --start 
devpi use root/pypi
devpi login root
devpi index root/pypi mirror_url="https://pypi.tuna.tsinghua.edu.cn/simple"

[pkuwwt]

Then there is another problem:

WARN: devpi-client-4-2.0 got an unversioend reply, assuming API-VERSION 2 (as implemented by devpi-server-2.0)
PATCH http://127.0.0.1:3141/root/pypi
500 Internal Server Error: Internal Server Error

The server encountered an unexpected internal server error

(Generated by waitress)

My devpi-server is 4.8.0

[fschulze]

You should see the traceback for the 500 Internal Server Error in the devpi-server logs. In your case they should be located in ~/.devpi/server/. Could you paste that traceback here?

[pkuwwt]

@fschulze no, there is no logs in ~/.devpi/server/. Do you mean .xproc/devpi-server/xprocess.log:

2019-03-18 23:34:27,730 INFO  [req17] GET /root/pypi
2019-03-18 23:34:27,978 INFO  [req18] PATCH /root/pypi
2019-03-18 23:34:27,980 ERROR Exception when serving /root/pypi
Traceback (most recent call last):
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 13, in _error_handler
    response = request.invoke_exception_view(exc_info)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/view.py", line 769, in invoke_exception_view
    raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/waitress/channel.py", line 336, in service
    task.service()
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/waitress/task.py", line 175, in service
    self.execute()
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/waitress/task.py", line 452, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 146, in __call__
    return self.app(environ, start_response)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 270, in __call__
    response = self.execution_policy(environ, self)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 279, in default_execution_policy
    return request.invoke_exception_view(reraise=True)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/view.py", line 768, in invoke_exception_view
    reraise_(*exc_info)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/compat.py", line 179, in reraise
    raise value
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 277, in default_execution_policy
    return router.invoke_request(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 249, in invoke_request
    response = handle_request(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 170, in request_log_handler
    response = handler(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 206, in request_tx_handler
    response = handler(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 43, in excview_tween
    response = _error_handler(request, exc)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 17, in _error_handler
    reraise(*exc_info)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/compat.py", line 179, in reraise
    raise value
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/view.py", line 657, in _call_view
    response = view_callable(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/viewderivers.py", line 325, in secured_view
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/viewderivers.py", line 401, in viewresult_to_response
    result = view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/viewderivers.py", line 116, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 640, in index_modify
    ixconfig = stage.get()
AttributeError: 'PyPIStage' object has no attribute 'get'

[fschulze]

Please try with 4.8.1. There is a fix included which I thought was already released with 4.8.0, but which I forgot.

[gary02]

Hi, I have run devpi index root/pypi mirror_url="https://pypi.doubanio/com/simple"，

$ devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/"                                                                                                                                                                   
/root/pypi mirror_url=https://pypi.doubanio.com/simple/
http://127.0.0.1:3141/root/pypi?no_projects=:
  type=mirror
  volatile=False
  mirror_url=https://pypi.doubanio.com/simple/
  mirror_web_url_fmt=https://pypi.org/project/{name}/
  title=PyPI

but devpi-server still fetch packages from files.pythonhosted.org.

This is the log:

2019-12-19 12:47:58,500 INFO  [req2] GET /laixi/stag/django/
2019-12-19 12:47:58,503 INFO  [req3] GET /laixi/stag/+simple/django/
2019-12-19 12:47:58,623 WARNI [req3] [Rtx780] serving cached links for https://pypi.doubanio.com/simple/django/ because returned serial -1, cache_serial 6073711 is better!
2019-12-19 12:47:58,750 WARNI [req3] [Rtx780] serving cached links for https://pypi.doubanio.com/simple/django/ because returned serial -1, cache_serial 6073711 is better!
2019-12-19 12:47:58,888 INFO  [req4] GET /root/pypi/+f/402/5317ca01f75fc/Django-2.2.6-py3-none-any.whl
2019-12-19 12:47:59,746 INFO  [req4] [Wtx780] reading remote: https://files.pythonhosted.org/packages/b2/79/df0ffea7bf1e02c073c2633702c90f4384645c40a1dd09a308e02ef0c817/Django-2.2.6-py3-none-any.whl, target root/pypi/+f/402/5317ca01f75fc/Django-2.2.6-py3-none-any.whl

And my env:

Python 3.6.8

devpi-client==5.1.1                                                                                                                                                                                                                                            
devpi-common==3.4.0                                                                                                                                                                                                                                            
devpi-server==5.3.1                                                                                                                                                                                                                                            
devpi-web==4.0.1

[fschulze]

The mirror doesn't send the x-pypi-serial header, because of that it uses old data (indicated by the serving cached links lines. You have to remove the root/pypi index and re-create it with devpi index -c root/pypi type=mirror mirror_url=https://pypi.doubanio.com/simple/ so it uses your mirror from the start.

[gary02]

The mirror doesn't send the x-pypi-serial header, because of that it uses old data (indicated by the serving cached links lines. You have to remove the root/pypi index and re-create it with devpi index -c root/pypi type=mirror mirror_url=https://pypi.doubanio.com/simple/ so it uses your mirror from the start.

Thank you for your reply!
I just find that --extra-index-url option in pip can meet my needs too.

[yunfan]

@fschulze i had met this issue too, and from the discussions above i learnt much, but the index re-create you mentioned raised error for the target index root/pypi exists, so how to remove the index root/pypi i had searched the quickstart and havnt found a hints on this

also i want to ensure that my demands is possible

first i want to run a devpi-server which indicate the upstream server to a none-official pypa but a mirrors, because like the above guys, i am chinese :[

second i want the server i had just setted could be used like another mirror to pip, for eg,
pip search -i http://mydevpi-server:8080/simple devpi-server

is that possible or should i find other alternatives?

[fschulze]

@fschulze i had met this issue too, and from the discussions above i learnt much, but the index re-create you mentioned raised error for the target index root/pypi exists, so how to remove the index root/pypi i had searched the quickstart and havnt found a hints on this

You can remove the index with devpi index --delete root/pypi. You have to be logged in as root.

also i want to ensure that my demands is possible

first i want to run a devpi-server which indicate the upstream server to a none-official pypa but a mirrors, because like the above guys, i am chinese :[

If you don't have any data in your server yet, you can also use the --no-root-pypi option when running devpi-init. Then you can manually add the mirror index: devpi index -c root/pypi type=mirror mirror_url=....

second i want the server i had just setted could be used like another mirror to pip, for eg,
pip search -i http://mydevpi-server:8080/simple devpi-server

is that possible or should i find other alternatives?

For pip search you need to use the URL to the index itself, not it's "simple" URL. For a locally running devpi this would work like this: pip search -i http://localhost:3141/user/index/ ...
You can also put this into a pip.conf:

[global]
index_url = http://localhost:3141/user/index/+simple/
[search]
index = http://localhost:3141/user/index/

[yunfan]

@fschulze thanks very much i had finally make it work

[johncf]

You can also put this into a pip.conf:

[global]
index_url = http://localhost:3141/user/index/+simple/
...

This is a bit unrelated... Is the best-practice for specifying index url with the final +simple part? It works even when I keep specify it as http://localhost:3141/user/index. (I have devpi-web installed too. Is that why it works?)

[fschulze]

@johncf It works without +simple, but it causes a redirect every time and only specific user agents are redirected, so it might not work with every tool.