-
Notifications
You must be signed in to change notification settings - Fork 942
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Artifactory not compatible with new /simple file links. #3275
Comments
@thiagofigueiro thank you for the report. this breakage is likely due to not following the redirect being served. You should be seeing a 303 See Other from pypi.python.org/simple/ urls. |
Internal mirror details Server: Artifactory Professional 4.8.0 rev 40210 Remaining configuration using defaults. I attempted changing the remote URL to https://pypi.org/simple/ but still received the same error. |
We may need to be issuing a 302... perhaps 303s aren't well supported? |
Does Artificatory have a |
@ewdurbin is there something I can do on my side to avoid the redirect? I obviously want to move to the new mirror ASAP but, more importantly, I'd like to have a working service while I figure-out the differences and reconfigure our internal mirror. |
@thiagofigueiro we've disabled it for now. we're going to try to filtering on User-Agent |
@ewdurbin I can confirm it started working again. Many thanks! If you would like me to help with tests, please reach-out to me: thiagocsf@gmail.com. |
URL detail: Downloading http://INTERNAL_HOSTNAME:80/artifactory/api/pypi/pypi-mirror/packages/c5/b8/b0a563f18548168492afa9786f2b9d26774d75b578969488effa1f4061c0/django_extensions-1.6.7-py2.py3-none-any.whl |
@thiagofigueiro we've moved from a 303 to a 302 and are opting all Artifactory User-Agents out of the split. If possible, setting a |
@ewdurbin No obvious way to do this with the version we're currently using in production so it will take me a little bit to set-up the test. I'm downloading the latest available from JFrog and will update this comment once I'm done. Instance public IP: 184.72.69.31 Update: aaaaaand Artifactory OSS doesn't have PyPi support. Pro Trial licence achieved. |
@thiagofigueiro if you can provide an IP address... we could give your installation the 302 redirect. |
Thanks for the quick fix. We're trying it out now. Our users were impacted today as well, and after some confusion, they finally contacted me, and then I found this thread that seems to match what we were seeing. We also use Artifactory. |
Our case seems to have cleared as well now. Artifactory as caching proxy behind the firewall. It's not clear to me what failed, though. I can't yet find errors. |
@mmielke-ciena to be explicit we are currently specifically excluding Artifactory User-Agents from our load test, so the issue is likely to arise if we stop doing so. We've moved from a 303 to a 302... and hope that Artifactory supports this status code. I'll update this issue before we remove the exception for Artifactory. |
@ewdurbin test instance ready. It's currently working:
Could you please activate the redirect for 184.72.69.31? |
@thiagofigueiro 184.72.69.31 should be receiving the 302 redirects now. |
Excellent - thanks. Can confirm it broke on the latest version of Artifactory Pro.
Will find-out how to set |
@thiagofigueiro if you can open an issue with Artifactory, let me know where it is and I'll slide in and contribute to helping find a solution. |
Artifactory issue opened: #78509 |
where is their tracker @thiagofigueiro ? |
I didn't get a link in the email - only the number. I have pointed them to this thread - hopefully they'll chime-in. Meanwhile, this is what I see in the logs. It seems they do follow the redirect.
i.e.:
|
@thiagofigueiro can you configure artifactory to use https://pypi.org directly? |
I did.
No record of redirect in the log when I use the new mirror:
If you would like access to this instance please give me an IP address. |
@thiagofigueiro and things are working as expected or are you still in a broken state? |
Still broken. Same error from pip:
|
Artifactory seems to be unable to parse the HTML correctly for individual packages. It can read the package listing on https://pypi.org/simple since I saw it download the individual package metadata, but then failed to present any packages in the listing. |
The relevant parsing information is that artifactory does not permit files from outside of the source server:
Thus generating empty listings. |
Ah, so it seems to be upset that we moved file hosting off of the domain and onto files.pythonhosted.org? |
The move to hosting package content on files.pythonhosted.org is intentional and will be happening. We'll need Artifactory to release fixes which respect the URLs provided. |
@thiagofigueiro do you have a link to the JFrog/Artifactory ticket? |
Hi, @ewdurbin - I had answered this before, you probably missed it: #3275 (comment) and the message right after it. tl;dr: JFrog don't provide a link to the ticket. :( The CSR responding to the email failed to understand what I was explaining and I don't think they will find their way to this github issue. |
Hi, @ewdurbin and @thiagofigueiro, thank you for reporting this matter we, JFrog, are aware to it. Please find the JIRA ticket, you are more than welcome to watch it: https://www.jfrog.com/jira/browse/RTFACT-16223 Ori |
@ori-yitzhaki thank you for following up, I'd like to note that this may be a higher priority issue than it is filed in JFrog-Jira. We'll be shipping the new /simple via warehouse in the next couple weeks and will not be able to guard Artifactory installations once that is complete. Target right now is April 8th. |
JFrog has made contact and I'm meeting with them to make sure they fully understand the changes that pypi.org will bring. Will update with summary of the discussion. |
I would like to thank you for reaching out and agree to support Artifactory users on pypi.python.org for a while. I would like to ask you to set the target for the migration of pypi.python.org => pypi.org to April 22nd. By that time we will have the support for pypi.org. 10x |
Oh! To summarize meeting: JFrog team confirmed the changes they saw with the warehouse simple index implementation and asked for anything else to be aware of. TLS v1.0/v1.1 deprecation was all that I had to add. They mentioned needing some more time to ship code changes and allow users to handle that. I directed them to have that conversation here and it seems they have! |
I said that we can technically delay the redirects from pypi.python.org to pypi.org that are causing trouble, but only for a short time, but we need consensus on how long. We currently exclude user agents for Artifactory from our temporary redirects and can support the same for the main rollout. |
Makes sense to me. |
we actually discuss that User-Agent exclusions for the redirect will last for a couple of months @ewdurbin can you come with a date which until it you will support the User-Agent exclusions for the redirect? |
@ori-yitzhaki If Artifactory can fully support using Currently (as of today) we are planning to fully shutdown |
Customers will have to coordinate such a change with their security departments (firewall) in larger organizations it might take a bit more time and we will appreciate if we can provide them sufficient notice |
I don't believe we are going to be able to keep the Keeping the exclusion past April 30th, 2018 would mean having to continue to run the legacy PyPI backend, @ewdurbin would have to speak more to that, but that's a much larger ask than keeping the exclusion in place until the 22nd. |
The discussion I recall didn’t include anything beyond April 30, I tried to make clear that the initially
requested April 22 date was technically straightforward, but may have misunderstood.
Indeed, holding on to the old backends after that date for a single client of the index is not something we can support.
|
@dstufft thanks for the information. |
@ori-yitzhaki We can keep the User-Agent exclusion until we shut down the legacy backends, that is currently scheduled for April 30th but that date is subject to change if needed (but I suspect it won't be). |
@ori-yitzhaki hey! so the day is here! pypi.org has shipped and we're planning to shutdown legacy on April 30th, 2018. Can you confirm that JFrog/Artifactory is underway with patches? |
The April 10th update in the JIRA ticket by @yuvalreches says:
The redirect has already started (on Monday, April 16th), and the User-Agent exclusion will work until April 30th. Could you please update the JIRA ticket to reflect @ewdurbin's explanation above, and advise your users to update Artifactory and change their PyPI remote repo URL fields now rather than waiting till April 22nd? Thank you. |
@ewdurbin Artifactory 5.10.3 and 5.9.5, 5.8.9 are available for download and supporting pypi.org 😃 |
Thank you @yuvalreches! @ori-yitzhaki @yuvalreches do you have a public support document, blog post, or similar that we can reference after the shutdown for artifactory users? |
@ewdurbin you can use this one: |
@yuvalreches excellent! thank you! Given that Artifactory 5.10.3, 5.9.5, and 5.8.9 are released and we have the necessary reference material for users impacted on Monday, I'm going to close this issue! |
We have an internal mirror of https://pypi.python.org that stopped working as the load testing started: https://status.python.org/incidents/0gmdf90kkt8n
I am currently investigating and collecting logs to further report but would like to create an issue in case other people come across the same problem.
If this turns-out to be my internal problem (I hope so!), my apologies in advance. I will update this issue as I have more information.
The text was updated successfully, but these errors were encountered: