Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature request: support bulk ops for MultiMap, and TransactionalMultiMap #9079

Open
dmitrymz opened this issue Oct 10, 2016 · 9 comments
Open

Feature request: support bulk ops for MultiMap, and TransactionalMultiMap #9079

dmitrymz opened this issue Oct 10, 2016 · 9 comments
Assignees
@abdulazizali77
Labels
good first issue Good for newcomers Module: MultiMap Priority: Low Source: Community PR or issue was opened by a community user Team: Core Type: Enhancement
Milestone
Backlog

Comments

@dmitrymz
Copy link

dmitrymz commented Oct 10, 2016 •
edited by mmedenjak

It will be great to add support for bulk operations for MultiMap, and TransactionalMultiMap, same as IMap.getAll(Set<K> keys), and putAll().

Absence of this functionality makes us use IMap with lists which is inefficient and high-maintenance
@mmedenjak mmedenjak added this to the Backlog milestone Jul 11, 2017
@mmedenjak mmedenjak changed the title Feature request: support bulk ops for MultiMap, and TransactionalMultiMap [map] Feature request: support bulk ops for MultiMap, and TransactionalMultiMap Jul 11, 2017
@burakcelebi burakcelebi added the Source: Community PR or issue was opened by a community user label Jan 3, 2018
@burakcelebi burakcelebi mentioned this issue Jan 3, 2018
Closed
@burakcelebi
Copy link
Member

Related: #46

@mmedenjak mmedenjak changed the title [map] Feature request: support bulk ops for MultiMap, and TransactionalMultiMap Feature request: support bulk ops for MultiMap, and TransactionalMultiMap Feb 4, 2020
@mmedenjak mmedenjak added the good first issue Good for newcomers label Feb 4, 2020
@abdulazizali77
Copy link
Contributor

Hi @mmedenjak @dmitrymz can i pick this up?

Just to confirm, i see that in addition to getAll, putAll, IMap has other bulk like interfaces loadAll, removeAll, evictAll; am guessing that load and evict might not be applicable to MultiMap and TransactionalMultiMap, but remove might be?
Additionally i see that MultiMap has a delete interface, should there be a deleteAll interface? So, what should the new bulk interfaces for MultiMap and TransactionalMultiMap be?

Also, maybe getting ahead of myself, but am inclined to have separate PRs for MultiMap and TransactionalMultiMap wdyt?

@mmedenjak
Copy link
Contributor

Hi @abdulazizali77 !

Let's not rush it and let's first add getAll and putAll. In addition to that, can you make those methods async? That will allow users to wait for those methods to complete asynchronously instead of blocking the user thread until the invocations return. So, instead of getAll and putAll, you can add getAllAsync and putAllAsync. An example of a bulk and async method would be MapProxyImpl#putAllAsync and ClientMapProxy#putAllAsync. These methods are not yet public API but they will be soon. You can look at those methods for inspiration - the MultiMap versions won't be the same but will be similar. You'll also need to modify the client protocol repo to support these methods from hazelcast clients.

And yes, it's best if you send separate PRs for MultiMap and TransactionalMultiMap.

@abdulazizali77
Copy link
Contributor

abdulazizali77 commented Feb 11, 2020 •
edited

Thanks @mmedenjak ! ill get cracking on the MultiMap getAllAsync and putAllAsync first. Can you assign me to this issue?

Probably getting ahead of myself again, but Im assuming that for this work, ill probably only make changes to the MultiMap.yaml and TransactionalMultiMap.yaml definitions after both MultiMap and TransactionalMultiMap changes get in? wdyt?

@mmedenjak
Copy link
Contributor

Yes, I think you'll have to add the definitions in the yaml files, re-generate the codecs as explained in the README document in your hazelcast repo branch and then use those new codecs in your hazelcast repo PR to add the client support.
For instance, you'll get a new codec generated like MapFetchKeysCodec and then you'll write a new task like MapFetchKeysMessageTask and then add that task in DefaultMessageTaskFactoryProvider and then invoke the task like this.

Feel free to take a look at some other and simpler methods in ClientMapProxy to see how tasks and codecs interact.

@abdulazizali77
Copy link
Contributor

Thanks @mmedenjak Ill get back to you should i encounter uncertainties!

abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Feb 26, 2020
@abdulazizali77
Initial commit to add MultiMap bulk put feature request (hazelcast#9079)
fa98e9b 
Add and implement two new interfaces putAllAsync and putAll.
Implementation is partition aware and follows Map.putAllInternal.

Add two new operation classes, MultiMapPutAllOperationFactory and PutAllOperation
which also follows the partitioning in Map PutAll operation flow.
Modify MultiMap serializer hook to add new operations.

Add stub for ClientMultiMapProxy.
@abdulazizali77
Copy link
Contributor

Hi @mmedenjak apologies for the very late update, ive put up the initial PR and my thought about the design (and uncertainties) and implementation detailed there. PTAL

abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Feb 27, 2020
@abdulazizali77
Initial commit to add MultiMap bulk put feature request (hazelcast#9079)
2b2dfd2 
Add and implement two new interfaces putAllAsync and putAll.
Implementation is partition aware and follows Map.putAllInternal.

Add two new operation classes, MultiMapPutAllOperationFactory and PutAllOperation
which also follows the partitioning in Map PutAll operation flow.
Modify MultiMap serializer hook to add new operations.

Add stub for ClientMultiMapProxy.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Mar 22, 2020
@abdulazizali77
Initial commit to add MultiMap bulk put feature request (hazelcast#9079)
9abce0e 
Add and implement two new interfaces putAllAsync and putAll.
Implementation is partition aware and follows Map.putAllInternal.

Add two new operation classes, MultiMapPutAllOperationFactory and PutAllOperation
which also follows the partitioning in Map PutAll operation flow.
Modify MultiMap serializer hook to add new operations.

Add stub for ClientMultiMapProxy.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Mar 22, 2020
@abdulazizali77
Modify new MultiMap bulk store interfaces to take in Map, MultiMap a…
941f97d 
…nd Key and Collection (hazelcast#9079)

 Implement ClientMultiMapProxy, corresponding PutAllMessageTask, unit tests and copy in generated codec.
 Address comments and fix MultiMap javadoc.

 Add new Precondition to throw given Exception.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Mar 22, 2020
@abdulazizali77
Modify new MultiMap bulk store interfaces to take in Map, MultiMap an…
d98853f 
…d Key and Collection (hazelcast#9079)

 Implement ClientMultiMapProxy, corresponding PutAllMessageTask, unit tests and copy in generated codec.
 Address comments and fix MultiMap javadoc.

 Add new Precondition to throw given Exception.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Mar 31, 2020
@abdulazizali77
Initial commit to add MultiMap bulk put feature request (hazelcast#9079)
04cedce 
Add and implement two new interfaces putAllAsync and putAll.
Implementation is partition aware and follows Map.putAllInternal.

Add two new operation classes, MultiMapPutAllOperationFactory and PutAllOperation
which also follows the partitioning in Map PutAll operation flow.
Modify MultiMap serializer hook to add new operations.

Add stub for ClientMultiMapProxy.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Mar 31, 2020
@abdulazizali77
Modify new MultiMap bulk store interfaces to take in Map, MultiMap an…
7552bef 
…d Key and Collection (hazelcast#9079)

 Implement ClientMultiMapProxy, corresponding PutAllMessageTask, unit tests and copy in generated codec.
 Address comments and fix MultiMap javadoc.

 Add new Precondition to throw given Exception.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Mar 31, 2020
@abdulazizali77
Modify interface and implementations of MultiMap.putAllAsync and rela…
3e59db4 
…ted classes (hazelcast#9079)

Remove non async putAll interfaces. Modify MultiMap.putAllInternal interface to take in Data, Data. Amend MultiMap.putAllAsync javadoc. Remove streams implementation

Fix PutAllOperation record saving implementation. Fix incorrect MultimapDataSerializerHook reference.

Implement Preconditions.checkTrue to take in class type.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 1, 2020
@abdulazizali77
Initial commit to add MultiMap bulk put feature request (hazelcast#9079)
30e2460 
Add and implement two new interfaces putAllAsync and putAll.
Implementation is partition aware and follows Map.putAllInternal.

Add two new operation classes, MultiMapPutAllOperationFactory and PutAllOperation
which also follows the partitioning in Map PutAll operation flow.
Modify MultiMap serializer hook to add new operations.

Add stub for ClientMultiMapProxy.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 1, 2020
@abdulazizali77
Modify new MultiMap bulk store interfaces to take in Map, MultiMap an…
240ba55 
…d Key and Collection (hazelcast#9079)

 Implement ClientMultiMapProxy, corresponding PutAllMessageTask, unit tests and copy in generated codec.
 Address comments and fix MultiMap javadoc.

 Add new Precondition to throw given Exception.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 1, 2020
@abdulazizali77
Modify interface and implementations of MultiMap.putAllAsync and rela…
d5eab93 
…ted classes (hazelcast#9079)

Remove non async putAll interfaces. Modify MultiMap.putAllInternal interface to take in Data, Data. Amend MultiMap.putAllAsync javadoc. Remove streams implementation

Fix PutAllOperation record saving implementation. Fix incorrect MultimapDataSerializerHook reference.

Implement Preconditions.checkTrue to take in class type.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 2, 2020
@abdulazizali77
Initial commit to add MultiMap bulk put feature request (hazelcast#9079)
b2263c3 
Add and implement two new interfaces putAllAsync and putAll.
Implementation is partition aware and follows Map.putAllInternal.

Add two new operation classes, MultiMapPutAllOperationFactory and PutAllOperation
which also follows the partitioning in Map PutAll operation flow.
Modify MultiMap serializer hook to add new operations.

Add stub for ClientMultiMapProxy.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Fix MultiMapContainerStatisticsTest to join putAllAsync thread (hazel…
bb1b0a8 
…cast#9079)
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Address sleep time, explicit type and comments (hazelcast#9079)
6cf6a8d
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Modify putAllAsync to use DataCollection (hazelcast#9079)
c639f37
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Reduce sleep time in ClientMultiMapTests (hazelcast#9079)
fa5bba5
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Fix data conversion in PutAllOperation and PutAllBackupOperation (haz…
379594f 
…elcast#9079)
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Change ClientMultiMap PutAll entries param from Map<Data, Data> to Ma…
a7bf45a 
…p<Data, Collection<Data>> (hazelcast#9079)

Update generated codecs and tests.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Make ClientMultiMapProxy.putAllInternal future @nonnull and remove ch…
d6f942f 
…eck (hazelcast#9079)
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Make ClientMultiMapProxy.putAllInternal future @nonnull and remove ch…
e6d09aa 
…eck (hazelcast#9079)
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 17, 2020
@abdulazizali77
Update generated protocol binary compatibility for MultiMap.PutAll (h…
dd153bc 
…azelcast#9079)
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 21, 2020
@abdulazizali77
Fix code quality issues (hazelcast#9079)
50e43b8 
rename variables, remove unused variables, use assertEventually idiom, remove sleep in tests
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 21, 2020
@abdulazizali77
New MultiMap bulk put feature request (hazelcast#9079)
83119aa 
Add two new interfaces for MultiMap.putAllAsync and implement for MultiMapProxyImpl and ClientMultiMapProxy.
Add new operation classes, MultiMapPutAllOperationFactory, PutAllOperation and PutAllBackupOperation.
Add corresponding tests for bulk put, statistics and backup.

Add new client protocol codecs MultiMapPutAllCodec and ListDataCodec and new message handler MultiMapPutAllMessageTask. Update generated client protocol tests and resources.
Add new Preconditions.checkTrueUnsupportedOperation.
vbekiaris pushed a commit that referenced this issue Apr 22, 2020
@abdulazizali77 @vbekiaris
New MultiMap bulk put feature request (#9079)
53c0763 
Add two new interfaces for MultiMap.putAllAsync and implement for MultiMapProxyImpl and ClientMultiMapProxy.
Add new operation classes, MultiMapPutAllOperationFactory, PutAllOperation and PutAllBackupOperation.
Add corresponding tests for bulk put, statistics and backup.

Add new client protocol codecs MultiMapPutAllCodec and ListDataCodec and new message handler MultiMapPutAllMessageTask. Update generated client protocol tests and resources.
Add new Preconditions.checkTrueUnsupportedOperation.
@abdulazizali77
Copy link
Contributor

@mmedenjak , should the issue be renamed?

@mmedenjak
Copy link
Contributor

No need, I'll just mention here that we will not be investing time into transactions at the moment. For more information, see here. Users trying to create enhancement issues for transactional map can now still find this issue and see what was the discussion outcome.

abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 28, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
bb3b6ce
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 28, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
63b74be
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 28, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
98e5bae 
Modify interface to return Map<K,V>
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue Apr 28, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
91bf0da 
Add generated protocol codec class and files
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 9, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
877a517 
Modify interface to return Map<K,V>
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 9, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
68776d2 
Add generated codec
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 9, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
83b8f90 
Rename GetAllOperation to GetOperation
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 9, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
583b2d8 
Implement new MultiMapGetAllOperationFactory and MultiMapGetAllOperation. Add new LockWaitNotifyKeySet
Implement getAll and getAllInternal for local Proxy and ClientProxy
Define new getAll tests.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 11, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
000ab1e 
Modify interface to return Map<K,V>
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 11, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
2fe8273 
Rename GetAllOperation to GetOperation
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 11, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
27bf6cb 
Implement new MultiMapGetAllOperationFactory and MultiMapGetAllOperation. Add new LockWaitNotifyKeySet
Implement getAll and getAllInternal for local Proxy and ClientProxy
Define new getAll tests.
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 11, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
42fffa8 
Add getAllAsync interface and implementation.
Add and modify corresponding tests
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 11, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
482962a 
Add generated codec and tests resources
abdulazizali77 added a commit to abdulazizali77/hazelcast that referenced this issue May 11, 2020
@abdulazizali77
WIP getAll implementation (hazelcast#9079)
724d1f5 
Remove getAll sync and corresponding tests
@abdulazizali77 abdulazizali77 mentioned this issue May 12, 2020
Closed
webashutosh pushed a commit to webashutosh/hazelcast that referenced this issue Jun 29, 2020
@abdulazizali77 @webashutosh
New MultiMap bulk put feature request (hazelcast#9079)
e6d3458 
Add two new interfaces for MultiMap.putAllAsync and implement for MultiMapProxyImpl and ClientMultiMapProxy.
Add new operation classes, MultiMapPutAllOperationFactory, PutAllOperation and PutAllBackupOperation.
Add corresponding tests for bulk put, statistics and backup.

Add new client protocol codecs MultiMapPutAllCodec and ListDataCodec and new message handler MultiMapPutAllMessageTask. Update generated client protocol tests and resources.
Add new Preconditions.checkTrueUnsupportedOperation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abdulazizali77 abdulazizali77

Labels
good first issue Good for newcomers Module: MultiMap Priority: Low Source: Community PR or issue was opened by a community user Team: Core Type: Enhancement
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@burakcelebi @hasancelik @dmitrymz @mmedenjak @abdulazizali77