[release-v1.6.1] Added pagination to the 'current_package_list_with_r…

…esources' action.

Using this action to retrieve the whole list of datasets in even a moderately-sized catalogue
is extremely slow.  (In the order of 5-6 datasets/sec on my development machine).  Aside from
this being slow, this means that since one of our supported deployments is proxying through
nginx, nginx will timeout the request before it is completed.

So as a worksround for now, the datasets can be paginated through multiple smaller requests.
Meaning at least the connection won't timeout; even if the total time retrieving the datasets
is approximately the same.

ckan/logic/action/get.py

@@ -62,6 +62,7 @@ def current_package_list_with_resources(context, data_dict):
     model = context["model"]
     user = context["user"]
     limit = data_dict.get("limit")
+    page = int(data_dict.get('page', 1))
 
     check_access('current_package_list_with_resources', context, data_dict)
 
@@ -71,7 +72,8 @@ def current_package_list_with_resources(context, data_dict):
 
     query = query.order_by(model.package_revision_table.c.revision_timestamp.desc())
     if limit:
-        query = query.limit(limit)
+        query = query.limit(int(limit))
+        query = query.offset((page-1)*limit)
     pack_rev = query.all()
     return _package_list_with_resources(context, pack_rev)
 

doc/apiv3.rst

@@ -55,7 +55,7 @@ Logic Action                           Parameter keys
 ====================================== ===========================
 site_read                              (none)                      
 package_list                           (none)
-current_package_list_with_resources    limit
+current_package_list_with_resources    limit, page
 revision_list                          (none)
 package_revision_list                  id
 group_list                             all_fields