Populating index

[caseydm]

I'm populating an index with around 12 million records. For some reason, the process stopped at around 9 million records. I'm curious, what happens when I run python manage.py search_index --populate --models employee.Employee. Will it skip records that already exist in the index? Will it be a faster process to get to those remaining 3 million records? Or will it take the same amount of time since it needs to iterate through every record?

[sabricot]

Yes it will iterate though every records, this could be a good thing to add a way to filter/order via the command for this kind of problem. There is a PR (#58) but i didn't really look into it yet.

[caseydm]

Ok thanks. I am having trouble indexing all of my 12 million records. It seems to stop around 9.5 million. I wonder if using queryset has anything to do with it? Should I write my own bulk update function or is it already doing that in the background?

[caseydm]

Hi all. As an update, I realized the reason the index was stopping short was due to a detached Heroku dyno timing out after 24 hours. I increased the pagination size and was able to complete the job within 24 hours.

I am really interested in a way to limit the index job size. I have someone adding around 8,000 records per week, so if I could just set a job to run each night and index the last 10,000 records that would be perfect. Anybody think of a way to easily implement that?

[cocoakekeyu]

Is that same as #118?

[sabricot]

Yes, close in favor of #118