Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
concurrent reads #14
Firstly, thank you for creating and maintaining this library!
Secondly, are you interested in adapting this library to support concurrent reads?
For my application, I need to read entries (at random) from a zip file over an extended period of time. The reads might be concurrent, and might be interleaved (read a few bytes from zip entry A, then a few bytes from zip entry B, then A again, etc).
The current API shape makes this use-case impossible without creating multiple
I've hacked the code to accomplish my goals, and thought I'd share it here in case you or any library users are interested in doing something similar.
I'd rather not maintain a separate fork of this library just to suit my use-case if it can be avoided, so if there is interest in incorporating any kind of support for concurrent reads, I'm happy to help out.
If there is no interest, please don't hesitate to close this issue.
I would be very interested in this.
Could you please explain how you tackle the problem of getting two views into one file?
I tackled the "two views into one file" problem by moving the problem onto the consumer of the API.
In my hacky code, the API consumer is allowed to create multiple concurrent
With such an API shape, it would no longer be necessary for the
If the decompressed data
It would also be possible to do more exotic things like described in my original comment. In that case, the API consumer could use multiple source
I feel like I struggled to articulate this, I hope it makes sense?
It makes a lot of sense.
I do not have the time to implement such a thing, and still don't know what the best approach is.
I was thinking of how to solve this problem and came to a similar solution: each handle to the zip file has its own
Edit, to expand:
Not sure how reusing a single source
The essential problem is the shared state of the current file position. We would have to either have that tracked individually per
Rambling a little bit, but what I propose is taking the
The idea is basically you could share a
Upon inspection this is basically a slightly more formal version of what @mark-buer proposes anyway...
I love the idea of having this feature, but do not know if it is very practical.
Especially since I think the user could also do one of the following things:
Passing a reader object to any of the functions would be a no. This may introduce very confusing bugs for users, and is hard to detect at the library side.
One other possibility is to hand out Cursor like objects to a user. In the case of a deflated file this would be easy, by simply seeking to the correct location. The deflate (and bzip2) case would be harder, as we also need to keep track of the decompression context.
@icefoxen: I do want to take a look at your code, but since the formatting is changed I have a hard time to see the changes. Could you undo them, or split them in separate commits?
The goals, as far as I can tell, are:
The hard part really is that you can't necessarily just
Goal 2 might or might not really matter depending on what you're doing; certainly, relaxing it would make life much easier. All of the problems disappear if you just create multiple
(edited to ramble more)
referenced this issue
Jan 30, 2017
I have added an