Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Indexing speedup #5939

Merged
merged 4 commits into from Jul 16, 2019
Merged

Indexing speedup #5939

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b38423d
fix indexing speedup
dojutsu-user Jul 16, 2019
cbbbb42
remove if
dojutsu-user Jul 16, 2019
a6b8a1a
use next()
dojutsu-user Jul 16, 2019
985488d
add logging
dojutsu-user Jul 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
33 changes: 21 additions & 12 deletions readthedocs/search/management/commands/reindex_elasticsearch.py
View file
Open in desktop
Expand Up @@ -10,7 +10,6 @@

from ...tasks import (index_objects_to_es, switch_es_index, create_new_es_index,
index_missing_objects)
from ...utils import get_chunk

log = logging.getLogger(__name__)

Expand All @@ -19,17 +18,27 @@ class Command(BaseCommand):

@staticmethod
def _get_indexing_tasks(app_label, model_name, index_name, queryset, document_class):
total = queryset.count()
chunks = get_chunk(total, settings.ES_TASK_CHUNK_SIZE)

for chunk in chunks:
data = {
'app_label': app_label,
'model_name': model_name,
'document_class': document_class,
'index_name': index_name,
'chunk': chunk
}
chunk_size = settings.ES_TASK_CHUNK_SIZE
qs_iterator = queryset.only('pk').iterator()
is_iterator_empty = False

data = {
'app_label': app_label,
'model_name': model_name,
'document_class': document_class,
'index_name': index_name,
}

while not is_iterator_empty:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually you can just use islice from itertools

from itertools import islice

objects_id = list(islice(qs_iterator, chunk_size))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And you can use https://docs.djangoproject.com/en/2.2/ref/models/querysets/#values-list to only get an iterator with only the ids

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this approach we will have a list of all the ids at once.
I think it is better to iterate over on generator object?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It returns a QuerySet of values, it's lazy https://stackoverflow.com/questions/37140426/does-django-queryset-values-list-return-a-list-object

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohhh...
But we also want to logging based on the pk.
Sorry... But I'm not able to think on how to use this method and to log also.
Do you have something on your mind?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can get the last pk from the list last_pk = objects_id[-1] if objects_id else 0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not so important anyway, current logic works too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the current implementation. It feels more explicit to me.

objects_id = []

try:
for _ in range(chunk_size):
objects_id.append(next(qs_iterator).pk)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd also like some logging here to be able to track progress. Something like if pk % 5000 = 0: log.info('Total: pk')

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 👍

except StopIteration:
is_iterator_empty = True

data['objects_id'] = objects_id
yield index_objects_to_es.si(**data)

def _run_reindex_tasks(self, models, queue):
Expand Down
8 changes: 0 additions & 8 deletions readthedocs/search/utils.py
View file
Open in desktop
Expand Up @@ -94,14 +94,6 @@ def get_project_list_or_404(project_slug, user, version_slug=None):
return project_list


def get_chunk(total, chunk_size):
"""Yield successive `chunk_size` chunks."""
# Based on https://stackoverflow.com/a/312464
# licensed under cc by-sa 3.0
for i in range(0, total, chunk_size):
yield (i, i + chunk_size)


def _get_index(indices, index_name):
"""
Get Index from all the indices.
Expand Down