Duplicate Proto Records

[lprhodes]

Hi,

uniquelyAddNewProtoRecord:toExistingResponseGroups: ironically adds multiple instances of the same object to the protoRecords array.

This means when protoRecordWithRecordResponseObject:entity:existingResponseGroups is called, both the uniquelyAddNewProtoRecord and completeRelationshipProtoRecordMappingToProtoRecord methods end up adding the same object to the array.

[cnstoll]

Please see my latest pull request. Is that what you meant?

Can you also clarify what you mean by duplicates? Are these duplicates in some array somewhere? Is that array actually being used for anything? Could it be deleted or bypassed?

When I read duplicate I freak out a bit because I think you mean duplicates in the database. I have a lot of tests that are telling me that's not happening. So I just want to make sure we're saying the same thing about what's going on here.

Thanks,

• Conrad

[lprhodes]

The protoRecords is the array that had duplicates in - it was the same pointer for both objects. With my amend to use a prototypeDictionary it was easily fixed with the following code:

• (void)addProtoRecord:(MMRecordProtoRecord *)protoRecord {
  if (![self.prototypeDictionary objectForKey:protoRecord.primaryKeyValue]) {
  [self.protoRecords addObject:protoRecord];
  }
  }

As a side note - I did manage to get duplicates in my database at one point - with the issue that caused me to think the result and failure block was being called…multiple calls of the exact same request were occurring simultaneously. Now I'm not calling it multiple times so it won't occur any more.

[lprhodes]

Forget my side note - one key was an NSString and one was an NSNumber so comparePrimaryKeyValues wasn't working. It could be worth putting an NSAssert in there - my mogenerator is broken for some reason.

[cnstoll]

Hrm, interesting. Yeah, that's a weird one. Part of me wants to just always normalize that value to a string and do string comparison...but that may not be correct in all cases.

So is the issue above fixed in your pull request?

[lprhodes]

String comparison would be slower though surely? I fixed it in the following commit:
lprhodes@74362d1

[cnstoll]

Yeah it's not ideal that's for sure. What you're describing though sounds like a problem in your data set. These are duplicates for one type of entity right? But that the objects coming back in JSON have either a number or a string value? I think if this happened for us I would a) want to make sure the data is consistent, and b) maybe add a check in the marshaler/representation layer to make sure.

[lprhodes]

The duplicate was in fact for a single record I was importing (describing a user)…there was one being imported and two proto records. The key was the username.

[cnstoll]

This was actually a pretty major deal for performance. I didn't realize it until the other day in the dub dub labs. This is fixed by my WWDC Performance pull request. Assigning to myself and to the 1.1.0 milestone.

[cnstoll]

@urbanappetite do you mind checking the performance branch again to see if it's not saving records properly for you? I've tested it on all the example apps as well as my test suite and it's working fine on all of those.

[lprhodes]

I'll give it a look tomorrow - Glastonbury festival this weekend. Thanks!

Sent from my iPhone

On 30 Jun 2013, at 03:55, Conrad Stoll notifications@github.com wrote:

@urbanappetite do you mind checking the performance branch again to see if it's not saving records properly for you? I've tested it on all the example apps as well as my test suite and it's working fine on all of those.

—
Reply to this email directly or view it on GitHub.

[cnstoll]

Wow, that looks awesome. I love Mumford and Sons!

I did fix a bug in this branch this morning that may have been causing the issue that was giving you trouble. Just try it again and let me know how it goes.

Thanks,

• Conrad

[lprhodes]

Hey,

There's still an issue (the previous issue which was that it wasn't completing at all is fixed).

I have two entities that have a many to many relationship to one another. I've simplified the JSON below. The relationship is on "tableIdentifiers". Only one relationship is ever made between Table 1 and Table 2 by MMRecord rather than the 3 that should occur in the example below.

Table 1:
[
{
"tableIdentifiers" : [
"brand",
"metric",
"context"
],
"name" : "Some Name",
"key" : 5
}
]

Table 2:
[
{
"code" : "brand"
},
{
"code" : "context"
},
{
"code" : "metric"
}
]

[cnstoll]

Hey Luke,

Help me make sure I understand the example. You're saying that there are two entities, Table 1 and Table 2, but that a tableIdentifiers relationship on Table 1 should contain three Table 2 objects. Table 2 objects have a primary key called "code" and in this case will contain brand, metric, and context?

Did I get that all right?

That's actually very similar to one of the examples I am testing with. Let me do a bit more digging on my end.

Thanks,

• Conrad

[lprhodes]

Exactly, I think the many-to-many relationship is likely to be the culprit given that relationships are still made.

[cnstoll]

Ok. Pretty sure I found the problem. I'll push up an update later that should fix that issue and you can take another look.

[cnstoll]

Ok, changes pushed. Let's try that one more time :)

[lprhodes]

That's fixed, thanks!

FYI - With the small test dataset I'm using I get 45 seconds import on the master branch, 5 seconds on the performance branch. I got it down to 10 seconds just with the dictionary suggestion - to halve the time again is brilliant thanks!

[cnstoll]

Awesome! All merged into Master now. I'm going to wait to rev the podspec for a few more days to see if we can't get deletion in there too. Thanks again!

[lprhodes]

FYI - I've posted a radar requesting the ability for core data to batch insert rather than inserting pone record at a time. It may be worth posting a similar radar to help increase the requests weight.

[lprhodes]

Thanks for all your work!