Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 issues (and possible solutions) #950

Open
drzraf opened this issue Sep 17, 2018 · 13 comments
Labels

Comments

@drzraf
Copy link

@drzraf drzraf commented Sep 17, 2018

After long talks with my hosting provider, it seems clear that packagist.org routing IPv6 is at cause.
This already caused numerous issues in the past like

Could the owner of packagist.org have a contact with it's provider (OVH).
Ticket 72669662 has been filled there about IPv6 to packagist.org.
At least in some of the above tickets, it seems that OVH hosting is unable to provide IPv6 connectivity between 2 of its own datacenters.

@sudofox

This comment has been minimized.

Copy link

@sudofox sudofox commented Sep 27, 2018

Ah, so this is why I can't update my dependencies. Any way to force IPv4 usage within composer itself?

Edit: I tossed this into /etc/hosts to get it to work:

aburk@aburk:~$ dig +short repo.packagist.org
142.44.164.255

# echo "142.44.164.255 repo.packagist.org" >> /etc/hosts
@phishfood420

This comment has been minimized.

Copy link

@phishfood420 phishfood420 commented Sep 30, 2018

Think I have to agree. Packagist/Composer keeps saying the "fix" is to practically disable your box from using ipv6 but a search of everyone having issues pretty much shows it's multiple people, from multiple areas, on multiples of different installs and networks and colo centers, all having problems.

And keeping in mind these folks were probably all using ipv6 before and didn't have issues except for packagist until they came along and found this.

Sounds like the fix isn't as composer suggests (make your box prefer ipv4) but it's time for them to remove their ipv6 hostname records until OVH or someone else can figure out why they have ipv6 issues.

@infomaniac50

This comment has been minimized.

Copy link

@infomaniac50 infomaniac50 commented Oct 18, 2018

ARIN is out of IPv4 addresses so forcing your network to use it exclusively is not the answer. OVH has been this way since forever and it seems they are not committed to IPv6. https://otacon22.com/2016/02/21/two-hosting-providers-ipv6-setups-compared-ovh-online-net/

I'm not too familiar with how composer pulls code. It seems all packagist does is store json files that point to github/bitbucket or where ever the code is stored. A public S3 bucket would work very well for this. Using CloudFlare or CloudFront may work well too. You wouldn't have to dump your current infrastructure. You would just setup your origin server as the backend for the CDN network.

I'm curious to know how much traffic you guys see in a month.

@fdelapena

This comment has been minimized.

Copy link

@fdelapena fdelapena commented Oct 24, 2018

IPv6 is highly broken in OVH. I've moved all my VPS to somewhere else and all my issues got solved. Just don't use OVH, they don't care about IPv6 in 2018.

@drzraf

This comment has been minimized.

Copy link
Author

@drzraf drzraf commented Oct 25, 2018

btw OVH supports told (in the 72669662 ticket) that the problem was not on their side... (so supposedly coming from the network configuration of the VPS client)
@Seldaek ?

@Seldaek

This comment has been minimized.

Copy link
Member

@Seldaek Seldaek commented Oct 25, 2018

Hey folks, I am well aware of the issue, and nagging OVH as well. It's very hard to move forward with this though it seems on their end. I don't know enough about network routing to really say why.

However on our end, options are fairly limited as well. Moving to AWS would cost us quite a bit and CloudFront doesn't let you invalidate things fast enough anyway AFAIK. Disabling IPv6 entirely is not possible either as some people have IPv6-only machines and they would then be unable to reach us.

The best option I see if OVH can't fix this is to migrate to Fastly as they have enough cache control to be workable for us. I'll try to experiment more with that.

In the meantime adding a static IP for repo.packagist.org to your hosts file is probably the best option. Or alternatively disabling IPv6 routing as per https://getcomposer.org/doc/articles/troubleshooting.md#operation-timed-out-ipv6-issues-

@sudofox

This comment has been minimized.

Copy link

@sudofox sudofox commented Oct 25, 2018

@Seldaek

This comment has been minimized.

Copy link
Member

@Seldaek Seldaek commented Oct 30, 2018

OVH claims they fixed a few things in their north american v6 network 4-5 days ago. Does anyone see an improvement? If not please run the following commands and post the outputs here (they might take a few minutes to run):

ping6 repo-ca-bhs-1.packagist.org

mtr -r -P 443 -T -c 100 repo-ca-bhs-1.packagist.org

mtr -r -c 100 repo-ca-bhs-1.packagist.org

If you are not in north america, please let me know if you're having IPv6 issues as well, and show me the output of ping6 repo.packagist.org.

@drzraf

This comment has been minimized.

Copy link
Author

@drzraf drzraf commented Oct 30, 2018

Here are my tests (box checked for successful ping6):

$ mtr -6 -r -P 443 -T -c 100 repo-ca-bhs-1.packagist.org # from BHS3

HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- [removed]                  0.0%   100    0.3   0.3   0.2   3.5   0.2
  2.|-- 2607:5300:201::fe          0.0%   100    0.3   0.3   0.2   0.5   0.0
  3.|-- 2607:5300:201:c18:ffff:ff  0.0%   100    0.3   0.5   0.3   3.2   0.3
  4.|-- 2607:5300:0:1:1:c18:2:1    0.0%   100    0.5   0.5   0.4   0.9   0.0
  5.|-- 2607:5300:0:1:1:c8:2:0     0.0%   100    0.5   0.5   0.4   5.2   0.5
  6.|-- 2607:5300:201:c8::1f       0.0%   100    0.4   0.4   0.4   0.9   0.0
  7.|-- 2607:5300:201::83          0.0%   100    0.6   0.5   0.4   3.5   0.3
  8.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0

It happens that among the AAAA records for packagist.org come:

  • 2607:5300:201:2100::7:2273
  • 2607:5300:201:2100::7:2274
  • 2001:41d0:601:2000::536
  • 2001:41d0:601:2000::497

The two last ones (Warsaw ?) are reachable from BHS3.

IMHO the link OVH BHS1 <-- ipv6 --> OVH BHS3 is the broken one.

@infomaniac50

This comment has been minimized.

Copy link

@infomaniac50 infomaniac50 commented Oct 31, 2018

It seems to work Ok from AT&T 2600:1700::/28 AS7018.

I only got 2607:5300:201:2100::7:2273 in the AAAA query for packagist.org. No lost packets from ping -6.

@deviantintegral

This comment has been minimized.

Copy link

@deviantintegral deviantintegral commented May 1, 2019

I've been having ipv6 connectivity issues to packagist for the past several months. So far, I haven't noticed trouble to any other hosts. Here's my ping6 / mtr logs, which I think indicate OVH still has a problem.

$ ping6 repo-ca-bhs-1.packagist.org
PING6(56=40+8+8 bytes) 2607:f2c0:f00e:7a00:41b6:a241:6603:4065 --> 2607:5300:201:2100::7:2273

--- repo-ca-bhs-1.packagist.org ping6 statistics ---
100 packets transmitted, 0 packets received, 100.0% packet loss

$ mtr -r -P 443 -T -c 100 repo-ca-bhs-1.packagist.org
Start: 2019-05-01T09:07:53-0400
HOST: acutus-498.local            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- blackbox.lan               0.0%   100    0.4   0.5   0.3   0.8   0.1
  2.|-- 2607:f2c0:8006:2::1        0.0%   100    8.4  11.1   8.2  75.1   8.9
  3.|-- ae0-2150-bdr01-tor.teksav  0.0%   100    8.4  10.5   8.1  45.2   5.4
  4.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  5.|-- 2607:5300::252f            5.0%   100  8736. 1914.  16.5 8736. 1870.1
  6.|-- vl100.bhs-d1-a75.qc.ca    51.0%   100   16.3  16.1  15.8  16.6   0.2
  7.|-- be7.bhs-z2g1-a75.qc.ca    44.0%   100   16.1  16.0  15.7  17.0   0.3
  8.|-- 2607:5300::1:1:c8:2:0      0.0%   100   16.0  16.0  15.7  17.7   0.3
  9.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
 10.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
 11.|-- 2607:5300:201:2100::7:227 47.0%   100   16.5  18.5  15.6  73.6  10.3

$ mtr -r -c 100 repo-ca-bhs-1.packagist.org
Start: 2019-05-01T09:09:42-0400
HOST: acutus-498.local            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- blackbox.lan               0.0%   100    0.3   0.3   0.2   0.6   0.1
  2.|-- 2607:f2c0:8006:2::1        0.0%   100    8.3   9.1   8.1  41.6   3.6
  3.|-- ae0-2150-bdr01-tor.teksav  0.0%   100    9.8   9.7   8.1  51.4   5.1
  4.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
  5.|-- 2607:5300::252f           24.0%   100   17.5  21.4  16.4  98.2  15.4
  6.|-- vl100.bhs-d1-a75.qc.ca     0.0%   100   15.9  16.0  15.5  17.3   0.3
  7.|-- be5.bhs-z2g1-a75.qc.ca     0.0%   100   15.6  15.9  15.6  17.6   0.2
  8.|-- 2607:5300::1:1:c8:1:0      0.0%   100   15.8  15.8  15.6  16.9   0.2
  9.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0
@hp197

This comment has been minimized.

Copy link

@hp197 hp197 commented May 31, 2019

Also have the same issue
I think it heavily depends from where you are connecting (it sounds like a cdn issue).
This is the answer from .nl (and is broken):

dig repo.packagist.org AAAA

; <<>> DiG 9.10.6 <<>> repo.packagist.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15984
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;repo.packagist.org.		IN	AAAA

;; ANSWER SECTION:
repo.packagist.org.	120	IN	AAAA	2001:41d0:801:1000::895

;; Query time: 43 msec
;; SERVER: 2001:470:7c9e:101::1#53(2001:470:7c9e:101::1)
;; WHEN: Fri May 31 21:07:44 CEST 2019
;; MSG SIZE  rcvd: 75
curl -v --connect-timeout 5 [2001:41d0:801:1000::27d]
*   Trying 2001:41d0:801:1000::27d...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7fe809805400)
* Connection timed out after 5002 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 5002 milliseconds
@mwgamble

This comment has been minimized.

Copy link

@mwgamble mwgamble commented Jun 20, 2019

+100 Fastly. They have been 100% rock-solid and during the almost five years of experience with them.

Fastly are used to hosting package repositories, as they already handle PyPI traffic.

@alcohol alcohol pinned this issue Jul 18, 2019
@alcohol alcohol changed the title fix the IPv6 at packagist.org IPv6 issues (and possible solutions) Jul 18, 2019
@alcohol alcohol added the Support label Jul 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.