Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running PredicateBuilder Queries when nodes join is causing timeouts and cluster crash. #6242

Closed
kraythen opened this issue Sep 17, 2015 · 2 comments

Comments

Projects
None yet
5 participants
@kraythen
Copy link

commented Sep 17, 2015

We have many queries that use PredicateBuilder queries such as the one below. These predicates run a lot during the normal usage of our application. However, when we have a new node join the cluster while the predicates are joining we get an info log warning and then the whole cluster stops responding and sometimes fails completely. This happens much more when we used submitToKeyOwner() and are running the predicate inside that task. The log and sample code are below.

Sep 17 12:30:52 staging_i-47ba8f82 application-9001.log:  2015-09-17 17:30:52 +0000 - [INFO] - [10.15.1.136]:5701 [staging] [3.5.2] Partition assignments changed while executing query: PredicateBuilder{
Sep 17 12:30:52 staging_i-47ba8f82 application-9001.log:  (userId=587 AND itemId IN (222375,222374,222373,222379,222378,222377,222376,222383,222382,222381,222380,222387,222386,222385,222384,222389,222388))
public static Set<Integer> findCachedOrderIdsByUserItem(Long userId,  Collection<Integer> itemIds) {
  final EntryObject e = new PredicateBuilder().getEntryObject();
  return hazelcast.getMap("orders").keySet(e.get("userId").equal(userId)
      .and(e.get("itemId").in(itemIds.toArray(new Integer[itemtIds.size()]))));
}

We are running version 3.5.2 on AWS.

@Donnerbart Donnerbart self-assigned this Oct 2, 2015

@Donnerbart Donnerbart removed their assignment Oct 14, 2015

@mdogan mdogan added this to the 3.7 milestone Nov 2, 2015

@mdogan mdogan added Team: QuSP Team: Core and removed PENDING labels Nov 2, 2015

@vbekiaris

This comment has been minimized.

Copy link
Contributor

commented Jul 8, 2016

@kraythen thanks for reporting this issue, does it still stand for you? Would you be able to test with latest released version (3.6.4) or 3.7-EA? There have been several changes in handling of migrations and querying since 3.5.2.

@vbekiaris vbekiaris modified the milestones: 3.8, 3.7 Jul 12, 2016

@mdogan

This comment has been minimized.

Copy link
Member

commented Dec 29, 2016

Queries infrastructure changed significantly since 3.5. Please test with the latest version and re-open issue if you encounter the same problem again.

@mdogan mdogan closed this Dec 29, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.