Fix multi thread #22
Merged
Fix multi thread #22
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The blob is mostly a pointer to a char array. We must keep a reference on the `bytes` to be sure that the data is not gc while we are using it. (Assuming the python ZimBlob is not gc either)
If we already have a ZimBlob we don't need to get the data a second time. Reuse the one we have. More that a performance point, this is needed to avoid to loose the reference to the existing blob (and so gc internal data).
The `*_cy_call_fct` will be called by the C++ code in a different thread. To avoid race condition, we need to acquire the GIL before doing python operation. However, those function will also be called from the same thread (in addArticle). To avoid deadlock we need to release the GIL when we enter C++ function.
This ZimArticleWrapper that contains ZimArticle. We create the wrapper "on the fly" when we need it. The wrapper will increment the ref to ZimArticle to be sure that ZimArticle is alive as long as the wrapper is. Even if the ZimArticle is not referenced anymore in the python world.
If the ZimArticle is not referenced in the python world after the call
to add_article[1], the only reference to the zimArticle is the one the
wrapper has.
The wrapper is "stored" in a std::shared_ptr. When libzim ends using it,
it will decrement the reference in the std::shared_ptr, and if it comes
to 0, it will deleted the Wrapper, which in turn, will decrement the
ref to the article, and potentially makes the ZimArticle to be deleted.
However, everything is done in a different thread, so we must protect
the decrement by acquiring the GIL first.
[1] As this :
```
for i in range(N):
article = ZimArticle(...)
zimCreator.add_article(article)
```
None cannot be use to create a ZimBlob.
Contributor
Author
pirate
approved these changes
Apr 23, 2020
jdcaballerov
approved these changes
Apr 23, 2020
Contributor
|
@mgautierfr I think we also need to acquire the GIL at the helper (shared by all calls). I tested with this and works great with 100,000 arts compressed and indexed. I was still getting some SIGSEV before this. |
This was
linked to
issues
Apr 23, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.