Process import dataset and result in separate methods

[thauk-copperleaf]

This refactoring allows imports to be done asynchronously (in our case, using celery). We've been using this for months now, and I thought it was finally time to push the changes back upstream.

[thauk-copperleaf]

@bmihelac I'm not merging this immediately because I'd like to get your opinion first :)

[bmihelac]

@thauk-copperleaf I had take just a quick look for now and would dedicate more time later today.

Do I get this right - when async importing, you overrideprocess_dataset that hands out job to celery and immediately return some result that process_result displays?

[thauk-copperleaf]

Yes, process_dataset() divides the dataset into chunks, creates a group of celery task signatures for each chunk, and then uses chord to run the group. The immediate return value is an ImportResult model object (internal to us) that has a TaskState that defines if it's finished or not. (How that works is outside the scope of the library.)

[thauk-copperleaf]

We subclass ImportExportMixin and override its process_result() method in order to handle both asynchronous and synchronous imports.

[bmihelac]

Everything looks good to me.

[mateuspadua]

Hello everyone, there are some example how to import data using Celery? I didn't find anything in documentation. Thanks :)
http://django-import-export.readthedocs.io/en/stable/search.html?q=celery&check_keywords=yes&area=default#

[thauk-copperleaf]

@mateuspadua You'll have to build the tooling on your own. There's no library to provide the integration out of the box.

[mateuspadua]

@thauk-copperleaf thanks for reply, :)