Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to change default mirror url pypi.python.org/simple/ with devpi-server #433

Closed
robertding opened this issue Jun 7, 2017 · 27 comments
Closed

Comments

@robertding
Copy link

@robertding robertding commented Jun 7, 2017

i start devpi-server version 4.3.0 as cache server
with cmd devpi-server --serverdir /root/packages/ --port 5555 --host=0.0.0.0 --start
how can i change the default cache mirror?

@fschulze
Copy link
Contributor

@fschulze fschulze commented Jun 7, 2017

What do you mean by "default cache mirror"? Do you mean the URL pip uses, or the URL for the PyPI cache?

@robertding
Copy link
Author

@robertding robertding commented Jun 7, 2017

sorry i didn't make it clear. i want to change the default url pypi.python.org/simple/ to another mirror url like mirrors.aliyun.com when a package not accessable in my devpi server.

@robertding
Copy link
Author

@robertding robertding commented Jun 7, 2017

@fschulze sorry i dont exactly know the differnet between pip uses(client side usage?) and PyPi cache.
I tried to modify the pip.conf in my devpi server, but it didn't work. The devpi server still tring to get missing package from pypi.python.org which is far slow in my country.
I tried to modify the source code config.py under devpi_server, still not working. Thanks for your help.

@fschulze
Copy link
Contributor

@fschulze fschulze commented Jun 7, 2017

You can change mirror_url on the root/pypi index. Something like this should work (from my head, untested): devpi index root/pypi mirror_url="...". It must point to the simple index of the mirror.

@fschulze
Copy link
Contributor

@fschulze fschulze commented Sep 4, 2017

Did my answer solve your issue?

@blackholll
Copy link

@blackholll blackholll commented Oct 13, 2017

i have the same requirement
is this contol download package from ?
in you reply, how to "change mirror_url on the root/pypi index"?

server/devpi_server/extpypi.py line 118?

@fschulze

class PyPIStage(BaseStage): def __init__(self, xom, username, index, ixconfig): super(PyPIStage, self).__init__(xom, username, index, ixconfig) self.httpget = self.xom.httpget # XXX is requests/httpget multi-thread safe? self.cache_expiry = self.ixconfig.get( 'mirror_cache_expiry', xom.config.args.mirror_cache_expiry) self.xom = xom if xom.is_replica(): url = xom.config.master_url self.mirror_url = url.joinpath("%s/+simple/" % self.name).url else: url = URL(self.ixconfig['mirror_url']) self.mirror_url = url.asdir().url

@blackholll
Copy link

@blackholll blackholll commented Oct 13, 2017

and i have try to change mirror_url in main.py/_pypi_ixconfig_default/, it doesn't work

@fschulze
Copy link
Contributor

@fschulze fschulze commented Oct 13, 2017

You don't have to change code, use the devpi client to set the option on the index. For example to change the url for root/pypi:

devpi index root/pypi mirror_url="..."
@blackholll
Copy link

@blackholll blackholll commented Oct 14, 2017

@fschulze i think you have not understand my mean. normally, when someone install a package that haven‘t cached in devpi server, the devpi server when download the package from pipy.python.org. but we are in china, the speed is slow. so i expect the devpi server download package from pypi.doubanio.com/simple but not pypi.python.org. this should be work on devpi server but not client.

@fschulze
Copy link
Contributor

@fschulze fschulze commented Oct 15, 2017

That's exactly what I'm saying. If you run devpi index root/pypi you should get something like this:

% devpi index root/pypi
https://d.rzon.de:8141/root/pypi:
  type=mirror
  volatile=False
  mirror_url=https://pypi.python.org/simple/
  mirror_web_url_fmt=https://pypi.python.org/pypi/{name}
  title=PyPI

Now you run devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/" and you should get something like:

% devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/"
/root/pypi changing mirror_url: https://pypi.doubanio.com/simple/
https://d.rzon.de:8141/root/pypi:
  type=mirror
  volatile=False
  mirror_url=https://pypi.doubanio.com/simple/
  mirror_web_url_fmt=https://pypi.python.org/pypi/{name}
  title=PyPI

Now packages will be downloaded from the mirror instead of regular pypi.

@blackholll
Copy link

@blackholll blackholll commented Oct 15, 2017

@fschulze
I run devpi server by supervisor, and the command is :" /data/virtualenv/devpi/bin/devpi-server --port 404%(process_num)d --serverdir /data/project/devpiserver --outside-url=http://pypi.xxxxx.com".
how to run devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/"? there has no args like 'mirror_url'.

@fschulze
Copy link
Contributor

@fschulze fschulze commented Oct 16, 2017

You need devpi-client for that, it's not a command line option of the server, but a setting in the database of the server, which you can manipulate with devpi-client. See documentation, starting with https://devpi.net/docs/devpi/devpi/stable/+d/quickstart-releaseprocess.html

@blackholll
Copy link

@blackholll blackholll commented Oct 17, 2017

ths very much, it works

@robertding robertding closed this Mar 7, 2018
@pkuwwt
Copy link

@pkuwwt pkuwwt commented Mar 6, 2019

Not work any more?

devpi index root/pypi mirror_url=“...” get

PATCH http://127.0.0.1:3141/root/pypi
403 Forbidden: Access was denied. To this resource
Unauthorized: PyPIVIEW failed permission check
@pkuwwt
Copy link

@pkuwwt pkuwwt commented Mar 6, 2019

Oh, I should

devpi-server --passwd root
devpi-server --start 
devpi use root/pypi
devpi login root
devpi index root/pypi mirror_url="https://pypi.tuna.tsinghua.edu.cn/simple"
@pkuwwt
Copy link

@pkuwwt pkuwwt commented Mar 6, 2019

Then there is another problem:

WARN: devpi-client-4-2.0 got an unversioend reply, assuming API-VERSION 2 (as implemented by devpi-server-2.0)
PATCH http://127.0.0.1:3141/root/pypi
500 Internal Server Error: Internal Server Error

The server encountered an unexpected internal server error

(Generated by waitress)

My devpi-server is 4.8.0

@fschulze
Copy link
Contributor

@fschulze fschulze commented Mar 7, 2019

You should see the traceback for the 500 Internal Server Error in the devpi-server logs. In your case they should be located in ~/.devpi/server/. Could you paste that traceback here?

@pkuwwt
Copy link

@pkuwwt pkuwwt commented Mar 18, 2019

@fschulze no, there is no logs in ~/.devpi/server/. Do you mean .xproc/devpi-server/xprocess.log:

2019-03-18 23:34:27,730 INFO  [req17] GET /root/pypi
2019-03-18 23:34:27,978 INFO  [req18] PATCH /root/pypi
2019-03-18 23:34:27,980 ERROR Exception when serving /root/pypi
Traceback (most recent call last):
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 13, in _error_handler
    response = request.invoke_exception_view(exc_info)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/view.py", line 769, in invoke_exception_view
    raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/waitress/channel.py", line 336, in service
    task.service()
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/waitress/task.py", line 175, in service
    self.execute()
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/waitress/task.py", line 452, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 146, in __call__
    return self.app(environ, start_response)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 270, in __call__
    response = self.execution_policy(environ, self)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 279, in default_execution_policy
    return request.invoke_exception_view(reraise=True)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/view.py", line 768, in invoke_exception_view
    reraise_(*exc_info)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/compat.py", line 179, in reraise
    raise value
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 277, in default_execution_policy
    return router.invoke_request(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 249, in invoke_request
    response = handle_request(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 170, in request_log_handler
    response = handler(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 206, in request_tx_handler
    response = handler(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 43, in excview_tween
    response = _error_handler(request, exc)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 17, in _error_handler
    reraise(*exc_info)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/compat.py", line 179, in reraise
    raise value
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/view.py", line 657, in _call_view
    response = view_callable(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/viewderivers.py", line 325, in secured_view
    return view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/viewderivers.py", line 401, in viewresult_to_response
    result = view(context, request)
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/pyramid/viewderivers.py", line 116, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "/home/ubuntu/devpi_env/lib/python3.5/site-packages/devpi_server/views.py", line 640, in index_modify
    ixconfig = stage.get()
AttributeError: 'PyPIStage' object has no attribute 'get'
@fschulze
Copy link
Contributor

@fschulze fschulze commented Mar 19, 2019

Please try with 4.8.1. There is a fix included which I thought was already released with 4.8.0, but which I forgot.

@gary02
Copy link

@gary02 gary02 commented Dec 19, 2019

Hi, I have run devpi index root/pypi mirror_url="https://pypi.doubanio/com/simple"

$ devpi index root/pypi mirror_url="https://pypi.doubanio.com/simple/"                                                                                                                                                                   
/root/pypi mirror_url=https://pypi.doubanio.com/simple/
http://127.0.0.1:3141/root/pypi?no_projects=:
  type=mirror
  volatile=False
  mirror_url=https://pypi.doubanio.com/simple/
  mirror_web_url_fmt=https://pypi.org/project/{name}/
  title=PyPI

but devpi-server still fetch packages from files.pythonhosted.org.

This is the log:

2019-12-19 12:47:58,500 INFO  [req2] GET /laixi/stag/django/
2019-12-19 12:47:58,503 INFO  [req3] GET /laixi/stag/+simple/django/
2019-12-19 12:47:58,623 WARNI [req3] [Rtx780] serving cached links for https://pypi.doubanio.com/simple/django/ because returned serial -1, cache_serial 6073711 is better!
2019-12-19 12:47:58,750 WARNI [req3] [Rtx780] serving cached links for https://pypi.doubanio.com/simple/django/ because returned serial -1, cache_serial 6073711 is better!
2019-12-19 12:47:58,888 INFO  [req4] GET /root/pypi/+f/402/5317ca01f75fc/Django-2.2.6-py3-none-any.whl
2019-12-19 12:47:59,746 INFO  [req4] [Wtx780] reading remote: https://files.pythonhosted.org/packages/b2/79/df0ffea7bf1e02c073c2633702c90f4384645c40a1dd09a308e02ef0c817/Django-2.2.6-py3-none-any.whl, target root/pypi/+f/402/5317ca01f75fc/Django-2.2.6-py3-none-any.whl

And my env:

Python 3.6.8

devpi-client==5.1.1                                                                                                                                                                                                                                            
devpi-common==3.4.0                                                                                                                                                                                                                                            
devpi-server==5.3.1                                                                                                                                                                                                                                            
devpi-web==4.0.1
@fschulze
Copy link
Contributor

@fschulze fschulze commented Dec 19, 2019

The mirror doesn't send the x-pypi-serial header, because of that it uses old data (indicated by the serving cached links lines. You have to remove the root/pypi index and re-create it with devpi index -c root/pypi type=mirror mirror_url=https://pypi.doubanio.com/simple/ so it uses your mirror from the start.

@gary02
Copy link

@gary02 gary02 commented Dec 19, 2019

The mirror doesn't send the x-pypi-serial header, because of that it uses old data (indicated by the serving cached links lines. You have to remove the root/pypi index and re-create it with devpi index -c root/pypi type=mirror mirror_url=https://pypi.doubanio.com/simple/ so it uses your mirror from the start.

Thank you for your reply!
I just find that --extra-index-url option in pip can meet my needs too.

@yunfan
Copy link

@yunfan yunfan commented Feb 5, 2020

@fschulze i had met this issue too, and from the discussions above i learnt much, but the index re-create you mentioned raised error for the target index root/pypi exists, so how to remove the index root/pypi i had searched the quickstart and havnt found a hints on this

also i want to ensure that my demands is possible

first i want to run a devpi-server which indicate the upstream server to a none-official pypa but a mirrors, because like the above guys, i am chinese :[

second i want the server i had just setted could be used like another mirror to pip, for eg,
pip search -i http://mydevpi-server:8080/simple devpi-server

is that possible or should i find other alternatives?

@fschulze
Copy link
Contributor

@fschulze fschulze commented Feb 5, 2020

@fschulze i had met this issue too, and from the discussions above i learnt much, but the index re-create you mentioned raised error for the target index root/pypi exists, so how to remove the index root/pypi i had searched the quickstart and havnt found a hints on this

You can remove the index with devpi index --delete root/pypi. You have to be logged in as root.

also i want to ensure that my demands is possible

first i want to run a devpi-server which indicate the upstream server to a none-official pypa but a mirrors, because like the above guys, i am chinese :[

If you don't have any data in your server yet, you can also use the --no-root-pypi option when running devpi-init. Then you can manually add the mirror index: devpi index -c root/pypi type=mirror mirror_url=....

second i want the server i had just setted could be used like another mirror to pip, for eg,
pip search -i http://mydevpi-server:8080/simple devpi-server

is that possible or should i find other alternatives?

For pip search you need to use the URL to the index itself, not it's "simple" URL. For a locally running devpi this would work like this: pip search -i http://localhost:3141/user/index/ ...
You can also put this into a pip.conf:

[global]
index_url = http://localhost:3141/user/index/+simple/
[search]
index = http://localhost:3141/user/index/
@yunfan
Copy link

@yunfan yunfan commented Feb 7, 2020

@fschulze thanks very much i had finally make it work

@johncf
Copy link

@johncf johncf commented Apr 24, 2020

You can also put this into a pip.conf:

[global]
index_url = http://localhost:3141/user/index/+simple/
...

This is a bit unrelated... Is the best-practice for specifying index url with the final +simple part? It works even when I keep specify it as http://localhost:3141/user/index. (I have devpi-web installed too. Is that why it works?)

@fschulze
Copy link
Contributor

@fschulze fschulze commented Apr 24, 2020

@johncf It works without +simple, but it causes a redirect every time and only specific user agents are redirected, so it might not work with every tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.