Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

package-search: improving performance on fetching facet display name #2691

Closed
wants to merge 5 commits into from

Conversation

alexandru-m-g
Copy link
Contributor

In our CKAN project, HDX, we show all facets ( as opposed to just the popular ones ) on the search page. We have 241 groups and 110 orgs so the time spent for fetching the display_name for the orgs and groups can be significant. Therefore, instead of fetching the title for each org and group I've changed the code to do only one db query and to fetch only the needed columns.

On HDX

On my laptop ( i5, 16GB RAM, SSD ) I get the following time with the unchanged pacakge_search() function:

  • 2015-10-18 20:07:16,683 INFO [ckan.lib.base] /dataset render time 2.159 seconds

After the change:

  • 2015-10-18 20:05:28,595 INFO [ckan.lib.base] /dataset render time 0.930 seconds

On "Core" CKAN

I get the following time with the unchanged pacakge_search() function:

  • on /dataset 2015-10-18 20:44:50,578 INFO [ckan.lib.base] /dataset render time 0.347 seconds
  • on /dataset?_organization_limit=0&_groups_limit=0 2015-10-18 20:47:28,893 INFO [ckan.lib.base] /dataset render time 0.417 seconds

And after the change:

  • on /dataset 2015-10-18 20:40:33,138 INFO [ckan.lib.base] /dataset render time 0.205 seconds
  • /dataset?_organization_limit=0&_groups_limit=0 2015-10-18 20:43:14,784 INFO [ckan.lib.base] /dataset render time 0.221 seconds

In debug mode ( pydevd ), the differences are much higher: around 3 seconds on HDX and 1 second on core ckan

Also I removed a reference to the pkg variable that is no longer used since "trusting solr".

@alexandru-m-g
Copy link
Contributor Author

Weird that this test failed: ckanext/datastore/tests/test_create.py on python 2.7. It didn't fail on python 2.6 and before, the same python 2.7 test, worked: https://travis-ci.org/ckan/ckan/jobs/86087573. Also locally the test passes.
Will close this pull request and will try again.

@alexandru-m-g alexandru-m-g deleted the master branch August 5, 2017 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant