-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
query.map() and query.map_async() are no longer implemented #210
Comments
Hi, sorry we took some time to answer. There's really no technical reason for not implementing this. We looked around and it seemed that it was not in use. We are open to considering re-adding this at some point in the beta process, or if you would like to try to implement it, we would be happy to review a PR. |
We will think about it, for now we went with this concept. pool = Pool(50)
it = query.iter(keys_only=True)
items = pool.map(SomeClass.apply_some_values, it)
pool.close() |
@chmoder What is the Are/were you using any of the arguments to |
I would be happy to hear any other suggestions you have if there is a better choice than pool. The reason we used map here is to get the entities of keys in the current entity. (relationship type of thing) For example:
|
Can I ask what you're doing in |
The idea of It's a contrived example above, but when getting a |
Hi @chmoder , Like Carlos says above, the main reason we didn't implement this is we didn't think anyone was using it. (EDIT: And also because there was a large amount of supporting infrastructure just for this one feature.) Having learned otherwise, I don't see any reason why we shouldn't just go ahead and implement. In the meantime, I think your work-around is fine, but it achieves parallelism in a fundamentally different way than NDB and is relatively expensive in terms of computing resources. Since these operations are going to be I/O bound rather than CPU bound, it may be that the use of I will, provisionally, add this to our queue. |
We will certainly switch to python-ndb native methods and test them when the PR comes through for this issue. I understand and appreciate your explanation however I am not sure I am familiar enough with NDB's "single threaded parallelism" to write the implementation myself. (EDIT: please feel free to ask if there is something we can do to help.) In the mean time the pool keeps backward functionality while we refactor our projects. Thank you for the excellent work! |
I'm already on the case. Thanks! |
Are the map functions something we want to add in the future? I have a use case I can refactor
or I can also implement
query.map
*. I am guessing there is some low level reason that the choice to not implement was made; a comment about why would help prevent others from trying if so.The text was updated successfully, but these errors were encountered: