MR_import slow perfomance when importing a lot of objects (1000+) #183

Closed
Fl0p opened this Issue Jun 14, 2012 · 1 comment

Comments

Projects
None yet
3 participants
@Fl0p

Fl0p commented Jun 14, 2012

Bottle neck here :

NSManagedObject+MagicalDataImport.m line:230

  • (id) MR_importFromObject:(id)objectData inContext:(NSManagedObjectContext *)context;
    {
    NSAttributeDescription *primaryAttribute = [[self MR_entityDescription] MR_primaryAttributeToRelateBy];

    id value = [objectData MR_valueForAttribute:primaryAttribute];

    NSManagedObject *managedObject = [self MR_findFirstByAttribute:[primaryAttribute name] withValue:value inContext:context];
    if (managedObject == nil)
    {
    managedObject = [self MR_createInContext:context];
    }

    [managedObject MR_importValuesForKeysWithObject:objectData];

    return managedObject;

    }

Every import perform a fetch request with predicate ( MR_findFirstByAttribute )
if fetch returns nil new object in context will be created
this may took about 10-50 milliseconds (when current table have some data in it)
when i try to import 1000 + object it it may take 10-50 seconds
In my case - list of about 1500 objects with some repeats objects and indexed primary key it took 27-29 seconds

So need to optimize array importing ( MR_importFromArray )

Best way to optimize :

  1. from input array create a set SET_A (by primary key) without duplicates and solve/merge duplicated entitles
  2. fetch all existing objects from database SET_B
  3. find intersection of SET_B and SET_A by primary key = SET_C
  4. update existing entitles from SET_C by data from objects of SET_A
  5. find difference SET_A - SET_C by primary key = SET_D
  6. import SET_D without checking (create new entity for every object without findFirst request)

waiting for update :)

@blackgold9

This comment has been minimized.

Show comment Hide comment
@blackgold9

blackgold9 Oct 9, 2012

Contributor

Sorry to be so late, but a good thing to do is to pre-fetch the objects you will need with one query.
That way it goes to disk once, and you can access the results to make your changes. I've had huge success with that.

Contributor

blackgold9 commented Oct 9, 2012

Sorry to be so late, but a good thing to do is to pre-fetch the objects you will need with one query.
That way it goes to disk once, and you can access the results to make your changes. I've had huge success with that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment