Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexing speedup #5939

Merged

Conversation

@dojutsu-user
Copy link
Member

@dojutsu-user dojutsu-user commented Jul 16, 2019

No description provided.

@dojutsu-user dojutsu-user added this to In progress in In-doc search UI via automation Jul 16, 2019

try:
for _ in range(chunk_size):
objects_id.append(qs_iterator.__next__().pk)
Copy link
Member

@stsewd stsewd Jul 16, 2019

You can just call next(qs_iterator).pk


try:
for _ in range(chunk_size):
objects_id.append(qs_iterator.__next__().pk)
Copy link
Contributor

@davidfischer davidfischer Jul 16, 2019

This section could probably be simplified by using next() with a default value.

Copy link
Member Author

@dojutsu-user dojutsu-user Jul 16, 2019

What should be the default?

Copy link
Member Author

@dojutsu-user dojutsu-user Jul 16, 2019

I think default is not required, current logic required to catch the exception.

'index_name': index_name,
}

while not is_iterator_empty:
Copy link
Member

@stsewd stsewd Jul 16, 2019

Actually you can just use islice from itertools

from itertools import islice

objects_id = list(islice(qs_iterator, chunk_size))

Copy link
Member

@stsewd stsewd Jul 16, 2019

And you can use https://docs.djangoproject.com/en/2.2/ref/models/querysets/#values-list to only get an iterator with only the ids

Copy link
Member Author

@dojutsu-user dojutsu-user Jul 16, 2019

With this approach we will have a list of all the ids at once.
I think it is better to iterate over on generator object?

Copy link
Member

@stsewd stsewd Jul 16, 2019

Copy link
Member Author

@dojutsu-user dojutsu-user Jul 16, 2019

Ohhh...
But we also want to logging based on the pk.
Sorry... But I'm not able to think on how to use this method and to log also.
Do you have something on your mind?

Copy link
Member

@stsewd stsewd Jul 16, 2019

you can get the last pk from the list last_pk = objects_id[-1] if objects_id else 0

Copy link
Member

@stsewd stsewd Jul 16, 2019

Not so important anyway, current logic works too

Copy link
Member

@ericholscher ericholscher Jul 16, 2019

I prefer the current implementation. It feels more explicit to me.


try:
for _ in range(chunk_size):
objects_id.append(next(qs_iterator).pk)
Copy link
Member

@ericholscher ericholscher Jul 16, 2019

I'd also like some logging here to be able to track progress. Something like if pk % 5000 = 0: log.info('Total: pk')

Copy link
Member Author

@dojutsu-user dojutsu-user Jul 16, 2019

Done 👍

Copy link
Member

@ericholscher ericholscher left a comment

👍 I will get this wrapped up and deployed to web03 today to start the reindex.

@ericholscher ericholscher merged commit a457fc0 into readthedocs:gsoc-19-indoc-search Jul 16, 2019
@dojutsu-user dojutsu-user deleted the indexing-speedup branch Jul 16, 2019
@dojutsu-user dojutsu-user moved this from In progress to Done in In-doc search UI Jul 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants