Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

knife supermarket list taking too long #8958

Closed
vsingh-msys opened this issue Oct 3, 2019 · 5 comments · Fixed by #8971

Comments

@vsingh-msys
Copy link
Contributor

commented Oct 3, 2019

Description

knife supermarket list fetching the list from default supermarket https://supermarket.chef.io URL. Seems the default items count set to 10 and no way we can change it.

so it total items ~3,860 so it sending 3860/10 = 386 requests to get the data. After fetching all the data it's showing to the user which take around ~15mins

Eg.

knife supermarket list
start_time: 2019-10-03 16:14:33 +0530
------
end_time: 2019-10-03 16:31:49 +0530
Total time: 17.2687 mins

Client Output

TRACE: Initiating GET to https://supermarket.chef.io/api/v1/cookbooks?items=10&start=0
TRACE: ---- HTTP Request Header Data: ----
TRACE: Accept: application/json
TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
TRACE: X-REMOTE-REQUEST-ID: 7c23aa39-f5c9-4d8d-8a78-54fe92aa1c4f
TRACE: ---- End HTTP Request Header Data ----
TRACE: ---- HTTP Status and Header Data: ----
TRACE: HTTP 1.1 200 OK
TRACE: cache-control: max-age=0, private, must-revalidate
TRACE: content-encoding: gzip
TRACE: content-type: application/json; charset=utf-8
TRACE: date: Thu, 03 Oct 2019 10:28:40 GMT
TRACE: etag: W/"b63af66a9a9cfaff2782f977db888127"
TRACE: server: nginx
TRACE: x-content-type-options: nosniff
TRACE: x-frame-options: SAMEORIGIN
TRACE: x-request-id: 365623e7-97d3-4a4c-8317-9ffa914d346f
TRACE: x-runtime: 0.041249
TRACE: x-xss-protection: 1; mode=block
TRACE: content-length: 498
TRACE: connection: Close
TRACE: ---- End HTTP Status/Header Data ----
TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response
TRACE: Content-Length validated correctly.
TRACE: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
TRACE: Decompressing gzip response
TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
TRACE: Initiating GET to https://supermarket.chef.io/api/v1/cookbooks?items=10&start=10
TRACE: ---- HTTP Request Header Data: ----
TRACE: Accept: application/json
TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
TRACE: X-REMOTE-REQUEST-ID: 7c23aa39-f5c9-4d8d-8a78-54fe92aa1c4f
TRACE: ---- End HTTP Request Header Data ----
TRACE: ---- HTTP Status and Header Data: ----
TRACE: HTTP 1.1 200 OK
TRACE: cache-control: max-age=0, private, must-revalidate
TRACE: content-encoding: gzip
TRACE: content-type: application/json; charset=utf-8
TRACE: date: Thu, 03 Oct 2019 10:28:42 GMT
TRACE: etag: W/"2e4a2ba66a08eead0237b6fbf9d0b2a8"
TRACE: server: nginx
TRACE: x-content-type-options: nosniff
TRACE: x-frame-options: SAMEORIGIN
TRACE: x-request-id: df424c0b-cfd8-468a-be8f-754650cbfa2c
TRACE: x-runtime: 0.045889
TRACE: x-xss-protection: 1; mode=block
TRACE: content-length: 562
TRACE: connection: Close
TRACE: ---- End HTTP Status/Header Data ----
TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response
TRACE: Content-Length validated correctly.
TRACE: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
TRACE: Decompressing gzip response
TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
TRACE: Initiating GET to https://supermarket.chef.io/api/v1/cookbooks?items=10&start=20
TRACE: ---- HTTP Request Header Data: ----
TRACE: Accept: application/json
TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
TRACE: X-REMOTE-REQUEST-ID: 7c23aa39-f5c9-4d8d-8a78-54fe92aa1c4f
TRACE: ---- End HTTP Request Header Data ----
TRACE: ---- HTTP Status and Header Data: ----
TRACE: HTTP 1.1 200 OK
TRACE: cache-control: max-age=0, private, must-revalidate
TRACE: content-encoding: gzip
TRACE: content-type: application/json; charset=utf-8
TRACE: date: Thu, 03 Oct 2019 10:28:43 GMT
TRACE: etag: W/"d1585a2f14f179d10995b50093c8cbea"
TRACE: server: nginx
TRACE: x-content-type-options: nosniff
TRACE: x-frame-options: SAMEORIGIN
TRACE: x-request-id: 838bdf94-919f-4d1f-9e7c-24730406fa97
TRACE: x-runtime: 0.061106
TRACE: x-xss-protection: 1; mode=block
TRACE: content-length: 537
TRACE: connection: Close
TRACE: ---- End HTTP Status/Header Data ----
TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response
TRACE: Content-Length validated correctly.
TRACE: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
TRACE: Decompressing gzip response
TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
TRACE: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
TRACE: Initiating GET to https://supermarket.chef.io/api/v1/cookbooks?items=10&start=30
TRACE: ---- HTTP Request Header Data: ----
TRACE: Accept: application/json
TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
TRACE: X-REMOTE-REQUEST-ID: 7c23aa39-f5c9-4d8d-8a78-54fe92aa1c4f
TRACE: ---- End HTTP Request Header Data ----
@lamont-granquist

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2019

that dates back to chef 0.8.12 and the pagination should just get removed.

@tas50

This comment has been minimized.

Copy link
Member

commented Oct 3, 2019

I'm not sure if we want to actually make a response for 3800 items, but clearly the 10 item pagination is WAY to low. We could probably easily make that 100 or 200 items and be just fine.

@lamont-granquist

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

berks and policyfiles and everything pull all of it every time though. if you do 100 item pagination then you'll just wind up doing 38 requests and take all those round trips instead of 1. i can't really think of any way that is better on the server either. the pagination is also likely buggy since there's no query id to get consistent pagination on every request.

@vsingh-msys

This comment has been minimized.

Copy link
Contributor Author

commented Oct 4, 2019

Yup @lamont-granquist I have tested with items count 1000 and without limit, it's taking around 12-15 secs only in both the cases. So we can pull the total records in one go that will work also but for the future, I think we should keep the limit with some minor improvements?

@vsingh-msys vsingh-msys referenced this issue Oct 9, 2019
3 of 10 tasks complete
@robbkidd

This comment has been minimized.

Copy link
Member

commented Oct 9, 2019

For your situational awareness: berks pulls its information from the /universe endpoint which is a single JSON response of all-the-things for dep-solving purposes; it does not paginate through total-cookbook-count / page-size results.

As noted in this issue, knife supermarket list does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.