really bad performance using AppVeyor feeds with > 2.17.3 #1317

Closed
ctaggart opened this Issue Dec 19, 2015 · 15 comments

Comments

Projects
None yet
3 participants
@ctaggart
Contributor

ctaggart commented Dec 19, 2015

We have 60 dependencies in a paket.depdendencies with 55 of them from NuGet Gallery and 5 of them our AppVeyor feed. Paket 2.17.3 took 45 seconds to do a paket update. Paket 2.18.0 took 9 minutes.

Paket 2.17.3 will tries 1 request per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/package-versions/Auth0?includePrerelease=true HTTP/1.1
Authorization: Basic 
Accept: application/atom+json,application/json
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:26:07 GMT
Content-Length: 2

[]

Paket 2.18.0 and above makes 4 requests per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20NormalizedVersion%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 400 Bad Request
Cache-Control: private
Content-Length: 290
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:04 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">No property 'NormalizedVersion' exists in type 'Appveyor.Models.Package' at position 18.</message>
</error>
GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20Version%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic 
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 565
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/8.5
DataServiceVersion: 2.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<feed xml:base="https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
  <title type="text">Packages</title>
  <id>https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/Packages</id>
  <updated>2015-12-19T05:29:05Z</updated>
  <author>
    <name />
  </author>
  <link rel="self" title="Packages" href="Packages" />
</feed>
GET https://ci.appveyor.com/nuget/ourfeed/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic 
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: no-cache
Content-Length: 248
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">Resource not found for the segment 'Packages'.</message>
</error>
GET https://ci.appveyor.com/nuget/ourfeed/odata/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic 
Accept-Charset: UTF-8
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
Accept: application/atom+xml, application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: private
Content-Length: 245
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:06 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">Resource not found for the segment 'odata'.</message>
</error>
@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 19, 2015

Member

Thanks for reporting this.

Member

forki commented Dec 19, 2015

Thanks for reporting this.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 19, 2015

Member

Ok the fix that I have in mind will hopefully make 0 calls. Let me see if I
find some time on Monday.
On Dec 19, 2015 06:53, "Cameron Taggart" notifications@github.com wrote:

We have 60 dependencies in a paket.depdendencies with 55 of them from
NuGet Gallery and 5 of them our AppVeyor feed. Paket 2.17.3 took 45 seconds
to do a paket update. Paket 2.18.0 took 9 minutes.

Paket 2.17.3 will tries 1 request per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/package-versions/Auth0?includePrerelease=true HTTP/1.1
Authorization: Basic
Accept: application/atom+json,application/json
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:26:07 GMT
Content-Length: 2

[]

Paket 2.18.0 and above makes 4 requests per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20NormalizedVersion%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 400 Bad Request
Cache-Control: private
Content-Length: 290
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:04 GMT

No property 'NormalizedVersion' exists in type 'Appveyor.Models.Package' at position 18.

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20Version%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 565
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/8.5
DataServiceVersion: 2.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

<title type="text">Packages</title> https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/Packages 2015-12-19T05:29:05Z

GET https://ci.appveyor.com/nuget/ourfeed/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: no-cache
Content-Length: 248
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

Resource not found for the segment 'Packages'.

GET https://ci.appveyor.com/nuget/ourfeed/odata/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic
Accept-Charset: UTF-8
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
Accept: application/atom+xml, application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: private
Content-Length: 245
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:06 GMT

Resource not found for the segment 'odata'.


Reply to this email directly or view it on GitHub
#1317.

Member

forki commented Dec 19, 2015

Ok the fix that I have in mind will hopefully make 0 calls. Let me see if I
find some time on Monday.
On Dec 19, 2015 06:53, "Cameron Taggart" notifications@github.com wrote:

We have 60 dependencies in a paket.depdendencies with 55 of them from
NuGet Gallery and 5 of them our AppVeyor feed. Paket 2.17.3 took 45 seconds
to do a paket update. Paket 2.18.0 took 9 minutes.

Paket 2.17.3 will tries 1 request per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/package-versions/Auth0?includePrerelease=true HTTP/1.1
Authorization: Basic
Accept: application/atom+json,application/json
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:26:07 GMT
Content-Length: 2

[]

Paket 2.18.0 and above makes 4 requests per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20NormalizedVersion%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 400 Bad Request
Cache-Control: private
Content-Length: 290
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:04 GMT

No property 'NormalizedVersion' exists in type 'Appveyor.Models.Package' at position 18.

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20Version%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 565
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/8.5
DataServiceVersion: 2.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

<title type="text">Packages</title> https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/Packages 2015-12-19T05:29:05Z

GET https://ci.appveyor.com/nuget/ourfeed/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: no-cache
Content-Length: 248
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

Resource not found for the segment 'Packages'.

GET https://ci.appveyor.com/nuget/ourfeed/odata/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic
Accept-Charset: UTF-8
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
Accept: application/atom+xml, application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: private
Content-Length: 245
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:06 GMT

Resource not found for the segment 'odata'.


Reply to this email directly or view it on GitHub
#1317.

forki added a commit that referenced this issue Dec 19, 2015

forki added a commit that referenced this issue Dec 19, 2015

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 19, 2015

Member

#1319 should make things much better. I hope I find good Internet connection this evening to release a prerelease.
But review would be very welcome.

Member

forki commented Dec 19, 2015

#1319 should make things much better. I hope I find good Internet connection this evening to release a prerelease.
But review would be very welcome.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 19, 2015

Member

Ok a prerelease version is now available. Please let me know if it works for you.

// @vasily-kirichenko I remember you also had more than one feed and saw slower updates. Maybe this helps for you too.

There is still one new issue open to make "paket install" faster as well. Hopefully find some time on Monday.

Member

forki commented Dec 19, 2015

Ok a prerelease version is now available. Please let me know if it works for you.

// @vasily-kirichenko I remember you also had more than one feed and saw slower updates. Maybe this helps for you too.

There is still one new issue open to make "paket install" faster as well. Hopefully find some time on Monday.

@vasily-kirichenko

This comment has been minimized.

Show comment
Hide comment
@vasily-kirichenko

vasily-kirichenko Dec 19, 2015

Contributor

I have very slow "install" since 2.24.0, updates are even slower.

Contributor

vasily-kirichenko commented Dec 19, 2015

I have very slow "install" since 2.24.0, updates are even slower.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 19, 2015

Member

I will check that too
On Dec 19, 2015 7:12 PM, "Vasily Kirichenko" notifications@github.com
wrote:

I have very slow "install" since 2.24.0, updates are even slower.


Reply to this email directly or view it on GitHub
#1317 (comment).

Member

forki commented Dec 19, 2015

I will check that too
On Dec 19, 2015 7:12 PM, "Vasily Kirichenko" notifications@github.com
wrote:

I have very slow "install" since 2.24.0, updates are even slower.


Reply to this email directly or view it on GitHub
#1317 (comment).

@ctaggart

This comment has been minimized.

Show comment
Hide comment
@ctaggart

ctaggart Dec 19, 2015

Contributor

Thank you. I will test this when I'm in the office on Monday.

Contributor

ctaggart commented Dec 19, 2015

Thank you. I will test this when I'm in the office on Monday.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 20, 2015

Member

@vasily-kirichenko please check the new alpha version if you find some time. It does some things a little bit smarter. I hope it helps.

Member

forki commented Dec 20, 2015

@vasily-kirichenko please check the new alpha version if you find some time. It does some things a little bit smarter. I hope it helps.

forki added a commit that referenced this issue Dec 20, 2015

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 21, 2015

Member

since I also noticed a caching issue I had to invalidate all cache files.

Member

forki commented Dec 21, 2015

since I also noticed a caching issue I had to invalidate all cache files.

@ctaggart

This comment has been minimized.

Show comment
Hide comment
@ctaggart

ctaggart Dec 22, 2015

Contributor

This is not fixed in 2.37.1. There are still 3 requests that fail after the first call returns an empty list of packages. There are 3 extra unnessary failed requests for each package. The good news is that the paket update was only 2 minutes & 23 seconds. It looks like the cache fix helped.

Contributor

ctaggart commented Dec 22, 2015

This is not fixed in 2.37.1. There are still 3 requests that fail after the first call returns an empty list of packages. There are 3 extra unnessary failed requests for each package. The good news is that the paket update was only 2 minutes & 23 seconds. It looks like the cache fix helped.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 22, 2015

Member

What calls?
On Dec 22, 2015 04:54, "Cameron Taggart" notifications@github.com wrote:

This is not fixed in 2.37.1. There are still 3 requests that fail after
the first call returns an empty list of packages. There are 3 extra
unnessary failed requests for each package. The good news is that the paket
update was only 2 minutes & 23 seconds. It looks like the cache fix
helped.


Reply to this email directly or view it on GitHub
#1317 (comment).

Member

forki commented Dec 22, 2015

What calls?
On Dec 22, 2015 04:54, "Cameron Taggart" notifications@github.com wrote:

This is not fixed in 2.37.1. There are still 3 requests that fail after
the first call returns an empty list of packages. There are 3 extra
unnessary failed requests for each package. The good news is that the paket
update was only 2 minutes & 23 seconds. It looks like the cache fix
helped.


Reply to this email directly or view it on GitHub
#1317 (comment).

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 22, 2015

Member

Ok applied another fix. Did it help?

Member

forki commented Dec 22, 2015

Ok applied another fix. Did it help?

@ctaggart

This comment has been minimized.

Show comment
Hide comment
@ctaggart

ctaggart Dec 22, 2015

Contributor

Yes! There is only one call per package to AppVeyor now and the responses are all HTTP 200s. The paket update time dropped to 51 seconds for the same project with version 2.38.0.

Related, but not required for this ticket:
image

3 requests are being made to nuget.org for each package that it doesn't have. #1320 would make this irrelevant for me.

Contributor

ctaggart commented Dec 22, 2015

Yes! There is only one call per package to AppVeyor now and the responses are all HTTP 200s. The paket update time dropped to 51 seconds for the same project with version 2.38.0.

Related, but not required for this ticket:
image

3 requests are being made to nuget.org for each package that it doesn't have. #1320 would make this irrelevant for me.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Dec 23, 2015

Member

Could you please retry with 2.39.1 (twice)? The second update run should only use the correct feeds.

Member

forki commented Dec 23, 2015

Could you please retry with 2.39.1 (twice)? The second update run should only use the correct feeds.

@vasily-kirichenko

This comment has been minimized.

Show comment
Hide comment
@vasily-kirichenko

vasily-kirichenko Dec 24, 2015

Contributor

2.39.1.0: 40 seconds first run, 15 seconds subsequent runs. Much faster, still slower than 2.23 (6 seconds).

Contributor

vasily-kirichenko commented Dec 24, 2015

2.39.1.0: 40 seconds first run, 15 seconds subsequent runs. Much faster, still slower than 2.23 (6 seconds).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment