New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent connectivity issues from AppVeyor build environment #3961

Closed
FeodorFitsner opened this Issue May 15, 2017 · 21 comments

Comments

Projects
None yet
7 participants
@FeodorFitsner

FeodorFitsner commented May 15, 2017

Hi guys,

Starting from last Thursday, May 11 we started seeing a lot of timeouts while restoring NuGet packages on build worker VMs in our primary environment. The environment is located in Rackspace DFW3 data center, San Antonio, TX. We are not experiencing the issues on VMs running in Google Compute Engine cloud.

Could it be that requests from our environment are being throttled/dropped somehow (it uses a single outgoing IP address)? Could it be related to recent issues in NuGet's CDN provider? Do you see dropped requests on your side coming from our IP?

Any help with this ongoing issue is highly appreciated!

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner May 15, 2017

WinMTR report for that location:

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             148.62.16.3 -    0 |  109 |  109 |    0 |   17 |  226 |    2 |
|    aggr172b-1-core10.dfw3.rackspace.net -    0 |  109 |  109 |    0 |    1 |    7 |    1 |
|   core10-corea.corea.dfw3.rackspace.net -    0 |  109 |  109 |    2 |    3 |    8 |    3 |
|           be31.dcpe2.dfw1.rackspace.net -    0 |  109 |  109 |    2 |    3 |    7 |    6 |
|           be1-mspe4.dfw30.rackspace.net -    0 |  109 |  109 |    4 |    4 |    9 |    4 |
|      de-cix.dfw.verizondigitalmedia.com -    0 |  109 |  109 |    4 |    6 |   28 |    6 |
|                          152.195.72.129 -    0 |  109 |  109 |    4 |    8 |   54 |    6 |
|                            72.21.81.200 -    0 |  109 |  109 |    3 |    4 |   10 |    4 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

FeodorFitsner commented May 15, 2017

WinMTR report for that location:

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             148.62.16.3 -    0 |  109 |  109 |    0 |   17 |  226 |    2 |
|    aggr172b-1-core10.dfw3.rackspace.net -    0 |  109 |  109 |    0 |    1 |    7 |    1 |
|   core10-corea.corea.dfw3.rackspace.net -    0 |  109 |  109 |    2 |    3 |    8 |    3 |
|           be31.dcpe2.dfw1.rackspace.net -    0 |  109 |  109 |    2 |    3 |    7 |    6 |
|           be1-mspe4.dfw30.rackspace.net -    0 |  109 |  109 |    4 |    4 |    9 |    4 |
|      de-cix.dfw.verizondigitalmedia.com -    0 |  109 |  109 |    4 |    6 |   28 |    6 |
|                          152.195.72.129 -    0 |  109 |  109 |    4 |    8 |   54 |    6 |
|                            72.21.81.200 -    0 |  109 |  109 |    3 |    4 |   10 |    4 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
@skofman1

This comment has been minimized.

Show comment
Hide comment
@skofman1

skofman1 May 16, 2017

Contributor

@ryuyu , take a look.
@FeodorFitsner, we don't throttle requests, so this is most likely CDN related. is the issue still in progress?

Contributor

skofman1 commented May 16, 2017

@ryuyu , take a look.
@FeodorFitsner, we don't throttle requests, so this is most likely CDN related. is the issue still in progress?

@skofman1 skofman1 added this to the S118 - 2017.5.08 milestone May 16, 2017

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner May 16, 2017

Thanks for looking into that! Yes, it's still happening.

FeodorFitsner commented May 16, 2017

Thanks for looking into that! Yes, it's still happening.

@gep13

This comment has been minimized.

Show comment
Hide comment
@gep13

gep13 May 16, 2017

@FeodorFitsner @skofman1 has this issue been fixed? Looks like builds that were failing for me, and now passing again.

gep13 commented May 16, 2017

@FeodorFitsner @skofman1 has this issue been fixed? Looks like builds that were failing for me, and now passing again.

@pascalberger

This comment has been minimized.

Show comment
Hide comment
@gep13

This comment has been minimized.

Show comment
Hide comment
@gep13

gep13 May 16, 2017

@pascalberger yip, seems to still be failing for me as well. Had one build pass, which then it failed again 😢

gep13 commented May 16, 2017

@pascalberger yip, seems to still be failing for me as well. Had one build pass, which then it failed again 😢

@skofman1

This comment has been minimized.

Show comment
Hide comment
@skofman1

skofman1 May 16, 2017

Contributor

@FeodorFitsner , @gep13 , @pascalberger , sorry to hear this wasn't resolved yet. We are communicating with our CDN provider, and will update you on the progress.
I assume you are all using AppVeyor hosted in San Antonio, TX, Correct?

Contributor

skofman1 commented May 16, 2017

@FeodorFitsner , @gep13 , @pascalberger , sorry to hear this wasn't resolved yet. We are communicating with our CDN provider, and will update you on the progress.
I assume you are all using AppVeyor hosted in San Antonio, TX, Correct?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner May 16, 2017

Yes, it's Rackspace DFW3 data center in San Antonio, TX.

FeodorFitsner commented May 16, 2017

Yes, it's Rackspace DFW3 data center in San Antonio, TX.

@gep13

This comment has been minimized.

Show comment
Hide comment
@gep13

gep13 May 16, 2017

@skofman1 said...
I assume you are all using AppVeyor hosted in San Antonio, TX, Correct?

Yes, that is correct. Both @pascalberger and myself are running builds on AppVeyor.

gep13 commented May 16, 2017

@skofman1 said...
I assume you are all using AppVeyor hosted in San Antonio, TX, Correct?

Yes, that is correct. Both @pascalberger and myself are running builds on AppVeyor.

@gep13

This comment has been minimized.

Show comment
Hide comment
@gep13

gep13 May 16, 2017

@skofman1 just to let you know, our builds now seems to be running correctly 👍

gep13 commented May 16, 2017

@skofman1 just to let you know, our builds now seems to be running correctly 👍

@ryuyu

This comment has been minimized.

Show comment
Hide comment
@ryuyu

ryuyu May 16, 2017

Contributor

Hey! Glad to hear everything is working ok.
Just for some extra information here, we heard back from our CDN provider. According to them, they were seeing a few latency spikes in the path inside of rackspace network:

  1. be41.corea.dfw1.rackspace.net 0.0% 210 2.8 2.6 2.5 3.9 0.1
  2. po1.corea-core9.core9.dfw3.rackspace.net 0.0% 210 4.7 4.7 4.6 9.0 0.3
  3. core9-aggr172a-1.dfw3.rackspace.net 0.0% 209 4.3 36.7 4.1 261.4 66.0 <<<<<<

As far as we can tell, it appears to have been an intermittent network issue with rackspace.

Please let us know and reopen the issue if you see more problems like this going forward.

Thanks!

Contributor

ryuyu commented May 16, 2017

Hey! Glad to hear everything is working ok.
Just for some extra information here, we heard back from our CDN provider. According to them, they were seeing a few latency spikes in the path inside of rackspace network:

  1. be41.corea.dfw1.rackspace.net 0.0% 210 2.8 2.6 2.5 3.9 0.1
  2. po1.corea-core9.core9.dfw3.rackspace.net 0.0% 210 4.7 4.7 4.6 9.0 0.3
  3. core9-aggr172a-1.dfw3.rackspace.net 0.0% 209 4.3 36.7 4.1 261.4 66.0 <<<<<<

As far as we can tell, it appears to have been an intermittent network issue with rackspace.

Please let us know and reopen the issue if you see more problems like this going forward.

Thanks!

@ryuyu ryuyu closed this May 16, 2017

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner May 16, 2017

Yeah, it's working again! 😄 Thank you very much guys for fixing that!

FeodorFitsner commented May 16, 2017

Yeah, it's working again! 😄 Thank you very much guys for fixing that!

@skofman1

This comment has been minimized.

Show comment
Hide comment
@skofman1

skofman1 May 17, 2017

Contributor

@FeodorFitsner , glad to hear everything works for you! Please note that no fix was performed on our side, and the issue originated in the rackspace network.

Contributor

skofman1 commented May 17, 2017

@FeodorFitsner , glad to hear everything works for you! Please note that no fix was performed on our side, and the issue originated in the rackspace network.

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner May 17, 2017

@skofman1 so there was a fix on their side?

FeodorFitsner commented May 17, 2017

@skofman1 so there was a fix on their side?

@skofman1

This comment has been minimized.

Show comment
Hide comment
@skofman1

skofman1 May 17, 2017

Contributor

@FeodorFitsner , I assume this is the case, since there was no action on our end.

Contributor

skofman1 commented May 17, 2017

@FeodorFitsner , I assume this is the case, since there was no action on our end.

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner commented May 17, 2017

OK, thanks!

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jul 17, 2017

Hi guys,

We are experiencing connectivity issues to api.nuget.org from AppVeyor environment again. Could you please take a look? There was service disruption on nuget CDN provider - maybe this affecting the connectivity.

FeodorFitsner commented Jul 17, 2017

Hi guys,

We are experiencing connectivity issues to api.nuget.org from AppVeyor environment again. Could you please take a look? There was service disruption on nuget CDN provider - maybe this affecting the connectivity.

@gep13

This comment has been minimized.

Show comment
Hide comment
@gep13

gep13 commented Jul 17, 2017

@skofman1 skofman1 reopened this Jul 19, 2017

@skofman1 skofman1 assigned skofman1 and unassigned ryuyu Jul 19, 2017

@brienRosenquist

This comment has been minimized.

Show comment
Hide comment
@brienRosenquist

brienRosenquist Jul 19, 2017

I am also experiencing the same issue...packet loss when connecting out to https://api.nuget.org. When we run a continuous ping from one of our servers, we can see that there are timeouts occurring. A side by side ping test out to google 8.8.8.8 and did not see any timeouts.

The following traceroute was executed multiple times and we see timeouts occurring at 72.21.81.200 which belongs to Edgecast:

PS C:\Users\Rack> tracert api.nuget.org

Tracing route to cs9.wpc.v0cdn.net [72.21.81.200]
over a maximum of 30 hops:

1 2 ms 2 ms 1 ms 72.32.186.2
2 2 ms 2 ms 2 ms aggr107a-core3.dfw1.rackspace.net [72.3.129.76]
3 3 ms 2 ms 3 ms Po1.core3.CoreA.dfw1.rackspace.net [72.32.111.0]
4 3 ms 3 ms 3 ms be31.dcpe2.dfw1.rackspace.net [74.205.108.120]
5 5 ms 5 ms 4 ms 10.25.1.103
6 4 ms 6 ms 5 ms ae-22.a01.dllstx04.us.bb.gin.ntt.net [129.250.207.117]
7 4 ms 4 ms 4 ms ae-0.edgecast-networks.dllstx04.us.bb.gin.ntt.net [128.242.179.166]
8 3 ms 3 ms * 72.21.81.200
9 3 ms 3 ms 3 ms 72.21.81.200

brienRosenquist commented Jul 19, 2017

I am also experiencing the same issue...packet loss when connecting out to https://api.nuget.org. When we run a continuous ping from one of our servers, we can see that there are timeouts occurring. A side by side ping test out to google 8.8.8.8 and did not see any timeouts.

The following traceroute was executed multiple times and we see timeouts occurring at 72.21.81.200 which belongs to Edgecast:

PS C:\Users\Rack> tracert api.nuget.org

Tracing route to cs9.wpc.v0cdn.net [72.21.81.200]
over a maximum of 30 hops:

1 2 ms 2 ms 1 ms 72.32.186.2
2 2 ms 2 ms 2 ms aggr107a-core3.dfw1.rackspace.net [72.3.129.76]
3 3 ms 2 ms 3 ms Po1.core3.CoreA.dfw1.rackspace.net [72.32.111.0]
4 3 ms 3 ms 3 ms be31.dcpe2.dfw1.rackspace.net [74.205.108.120]
5 5 ms 5 ms 4 ms 10.25.1.103
6 4 ms 6 ms 5 ms ae-22.a01.dllstx04.us.bb.gin.ntt.net [129.250.207.117]
7 4 ms 4 ms 4 ms ae-0.edgecast-networks.dllstx04.us.bb.gin.ntt.net [128.242.179.166]
8 3 ms 3 ms * 72.21.81.200
9 3 ms 3 ms 3 ms 72.21.81.200

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jul 20, 2017

@brienRosenquist are you guys Rackspace customers too?

FeodorFitsner commented Jul 20, 2017

@brienRosenquist are you guys Rackspace customers too?

@brienRosenquist

This comment has been minimized.

Show comment
Hide comment
@brienRosenquist

brienRosenquist commented Jul 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment