Multiple Threads access #30
Comments
|
Nope, nothing to do at all. In fact in the unit tests there are lots of examples of up to 50 threads all reading/writing/deleting at the same time. There is a critical section around a write op to a database, but these will not block reads. And transactions are partitioned by thread as well, so unless there is a huge op within a large transaction, you should not see, much in the way of blocking. Any additional information you could provide would be helpful. Is there anything special about what you are persisting? e.g. large BLOB objects, or very wide objects? |
|
It seems I am not able to find a reason, but I suspect is somehow related to the framework (at least maybe to the way I am using it). As a background - I am sending a few server requests from different threads to fetch some data (using AFNetworking). When the results come, I insert/update them in the database with SharkORM. The problem is that the app simply freeze (like entering in an endless loop) - but not always at the same step and what is most frustrating - it happens a few times per each 100 runs (or so). I will try to note down where it happens each time but unfortunately I don't know how I can reproduce it properly. Any thoughts are highly appreciated. PS - it's not like a temporary block - it freezes forever, no matter what I do. I am storing only simple values - like NSNumber, NSString, BOOL, double and a few relations between tables. Is it possible that I could enter in a deadlock or something (like referentiating A from B and also B from A)? |
|
I don't think there is a deadlock - because in that case it would freeze all the time at the same place/moment. |
|
Hi Sorry for not getting back to you yesterday, I would agree. The deadlocks always show themselves waiting for a semaphore, and we only have the one critical section as far as I can remember around write operations. You have done quite a lot of investigation already. And we also use AFNetworking in the same kind of way. We modified AFNetworking to always use background threads and not all crashing in on the main thread, but that would be the only difference. Are you seeing any errors on the delegate? Locked SQL file, index corruption, anything like that? |
|
Also, i'd never expect a freeze on your example of thread 4. I'd be looking at what priority those dispatch items were, and if they got suspended by GCD for higher priority blocks? But i'm really scraping the barrel for ideas there. |
|
Thank you for your reply. I am attaching 2 images with what I think it might be relevant.
|
|
Thanks for the info, it's very weird. As the "setPropertyIMP", is only on a single object, and shares nothing with other objects or the ORM. There is quite literally no reason why it would ever block there. What created the block and dispatched it? Was it an AFNetworking response block? Could you try dispatching one of those blocks on a different queue: dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{ I appreciate that this is normally a nightmare to try and break your app apart to make a bit of it async. But might I suggest using an NSLock to hold a semaphore until the dispatched block has completed. I'm just interested to see if you break out of this dispatch queue, whether or not it starts to work. It might give us at least a direction to head in. P.S. does this happen on an entirely repeatable basis, or is it intermittent? |
|
Hy @Parkyprg, Is there anything specific you would like me to look at, or a scenario I could setup that you think would be the best possible chance to replicate your issue? I have a couple of free hours today that can be dedicated to trying to work out this issue. Thanks |
|
Hi Adrian, I managed to reproduce the problem in a separate project. Can you tell me an email address where I can send you the source files? Thank you, |
|
Hi, Looking forward to fixing it! Sent from my iPhone
|
|
Email sent. Forgot to mention it here :) |
|
I was wondering if you were able to reproduce the problem with the test project that I sent. |
|
Just looking now, but so far unable to reproduce. Does this happen on devices or the simulator, or both?
|
|
Mostly on device. |
|
Hi, i'm just trying to get it on a phone, but in the simulator we added 1000 contacts to really stretch it. |
|
On simulator I believe the time spent processing the contacts is too short. |
|
Okay, I will try that when our testing team get here. I'd prefer not to create loads of addresses on my personal phone. As actually, the app just crashes on iOS 10, as it tries to access the contacts before it asks for permissions. I'm just going to add it in manually now. UPDATE: The permission for Contacts is not present in iOS 10 (it crashes before it adds the entitlement I think), so I can't manually add it. I'll wait for an iOS 9 device. Will get back to you soon. |
|
Right, i've managed to re-create the hang (it took a lot of trying), however, in this particular case it was not utilising shark at all, but had been blocked by two threads both creating a runloop for the contacts API at the same time. I will continue looking to see if I can't pin it down further.
|
|
Also, what device/iOS version are you using? I'll try and match that this end if I can for a better and more accurate test. I'm currently using an iPhone 6 running iOS 9. 1000 contacts. |
|
I am using the same device - 6 with iOS 9.3.5 |
|
Hi, I'm going to look at this again today. Is there any further information/investigation for this issue? |
|
Thank you for looking into this. I am not 100% sure this is caused by the framework but every time it happens - 2 threads are locked on SharkORM processes. |
|
Hi, Were you able to find anything? At least did you managed to reproduce it on a test device? |
|
I've only been testing for about 1/2 an hour, but so far it has not locked up once. When I did manage to get it to freeze last time it happened to not be doing anything in Shark. Another time, it froze whilst on the line.
Which, really just has to be symptomatic of the co-ordinating thread locking. Knowledge of how iOS/OSx manages it's thread hierarchy is not known to me, but I would have thought there will be a base level runloop (thread) that controls all the others, but I would need to ask someone on SO who may be more knowledgeable. Shark, as a general rule does nothing in any background threads, it merely executes it's code in-line with the developers intentions. We do use performSelectorInBackground: if you're performing async queries, but these are the specific differences. So, although unable so far to identify the culprit, i'm struggling to find any reason to believe it is the ORM. Although I do maintain a completely open mind. I will do some more testing over the next hour or so. Possibly on some older / single core devices. Thanks |
|
Any update for this issue? i also get freeze sometimes at following line
this is called from main thread |
|
Hi, I looked again recently at this, but in the other instance, the code would stop in the middle of nowhere and in places that could not possibly ever pause. Yours however might be different. So, a few questions. Does your hang ever continue, or is a transient pause? Are there other DB operations happening in the background? (they would have to be huge to cause any sort of delay). Is this call normally within any kind of transaction? But I will look at that method now to see if there is anything I can spot. |
|
it pause at this line every time, we can see the thread is wait for some thing from the stack trace. it is not a transient pause. i think there is not any background access, since it occur when our app just launch |
|
|
happen again |
|
Thanks, I think I may have found the bug from the information you provided. I will check it out now and post a fix. By rights it should just crash, but it doesn't. It just hangs instead. |
|
so, it seem two thread access systemEntityRelationships at the same time? |
|
yep, when a static is accessed it seems to block the main thread. I'm just removing the statics and putting them into a class. Those items are a bit of a throwback to some original code when the framework was just c & c++ |
|
Okay, to do this right is going to take some time. ETA on a fix is tomorrow by the looks of it, subject to passing all the tests. |
|
Thank you Adrian. I will try this fix too. |
|
Just done it v2.0.9 should be showing, thanks @Parkyprg for being so patient. It was only when the second report came in that I could even see the pattern. It may not fix it (considering I really struggled to replicate it), but i'd be really surprised if it didn't :) |
|
:) |
|
Is this looking good now? It has caused a performance drop in the ORM, but that will be dealt with separately. |
|
i just get the latest code. look good for now. if any issue found, we will let you know. by the way, i have made some changes on my local branch(some changes about printing sql log base on setting, KVO related bug fix and some additional convenient methods), can i send a pull request? |
|
Yes please do. |
|
i think your change may not fix the problem, the problem is two thread access/modify the systemEntityRelationships list, but your change is wrap the getting logic of the member and add lock in the getting method. it won't avoid the access/modification to the list from two threads after they get the list object with a thread safe wrapper. |
|
The problem is still existing. i run bt all when it occur, get the same result as before, not sure if the problem is access [Class description] in different thread. |
|
i guess the problem is on code [r.sourceClass description] |
|
Those calls to get the systemEntityRelationships create a copy of that dictionary, so that should make it thread-safe (from an iteration point of view), now there might be two things accessing the objects within the dictionary which will be the same, but, as they are being used RO there should never have been a problem accessing them as a single static either. I only wrapped a lock around them to ensure there were not two copies being made at the same time. I strongly believe this to be a problem with apple's implementation of their block dispatcher, which seems to lock threads when creating/copying stack objects. I had other plans for how I was going to deal with all these statics, event registry, backing store & re-implement the caching that would largely do away with large swathes of this code, but I was unsure as to weather it will actually be faster (more performant) in the long run. It would enable good memory savings and some CPU gains on some operations, but retrieving a properties value may well become slower, which would be a more consistent performance concern. As for description, that is simple string manipulation and work on the heap. And there is a heap per thread anyway, so this should never clash. |
|
Anyhow, I will look again at this. Because those changes made the ORM much slower, so if it is not the static objects, then they should go back or be re-implemented differently as this is one scenario where OO is the worst possible solution for this. |
|
I have a suggestion on removing the lock and make the call faster: setup all SRKObject class( i mean setupClass method in SRKObject class) and setup all the required list objects(tableSchema,relationships...) when the open method is called. In this way, the database and the schema information in memory will be setup when open method is called, then the list won't be modified in the app's life cycle. there is only read operation, the lock can be removed. anything is incorrect? |
|
So you could test that by using the setupClasses method, which does nothing but ensure that all the classes are "touched" well in advance of being used in anger which achieves the same result. But none of the incidents indicate any objects being freshly initialised, which would leave them as read only. |
|
i will make the change and test. |
|
Ok - so in my application I am able to reproduce this freeze anytime now. Issuing a 'bt all' command gives me the following:
In Thread #5, I can see WAITING_FOR_ANOTHER_THREAD_TO_FINISH_CALLING, while the main thread is in __ulock_wait status. So here is the deadlock. Is this related to network access somehow? |
|
No, I think @lbwxly is on to something, it looks like the _class_initialize of these classes on different threads at the same time is causing the issue. It does some reading and writing to a few arrays and dictionaries, which we just split out to try and solve it. But what they point to is being potentially mutated. Normally, any kind of access to the same object would just crash. But maybe, your example is suggesting that the _class_initialize is being called from two threads at the same time even though the +initialize method should only ever be called once for a class. |
|
@Parkyprg, you can try to call description method of all sub-classes of SRKObject class when your app launch. i think the freeze will disappear. |
|
Thanks, @lbwxly & @Parkyprg. It's great that it can be quickly replicated. Just a couple of points though. I can't see anything about the description overload that should cause any locking like this. As there should be a new IMP to that method for every call, and a new stack per thread. Also, although the first hang looked like it might be on the call to description, the objc_getProperty call should have left the stack pointer inside the +description call. Plus, I don't think it will be treated as a property, i think that error is in the call: to r.sourceClass. Nestled away was this: Which is also significant (I think) frame #3: 0x00000001904f13bc libobjc.A.dylibWAITING_FOR_ANOTHER_THREAD_TO_FINISH_CALLING_+initialize + 84 |
|
Thank you @lbwxly - after calling |
|
@editfmah i want to send a pull request(submit the change i mentioned yesterday), but github desktop tell me i don't have permission on the project. |
|
@lbwxly I have added you to the project, you should have an invite to the team. Normally, you fork the codebase into your own repository and then we do pull requests from there. For some reason i'm limited to just 5 users on the main repository which is something i'm trying to work out right now. |
|
This issue can be closed. Thank you for your support. |
In one of my applications I am accessing/updating db objects from multiple threads and for some reason it simply freeze.
I am trying to narrow it down, but being multi-threaded, seems to be difficult.
Are there any special steps to make, in order to work with the objects from multiple threads?
The text was updated successfully, but these errors were encountered: