Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TTL callback is invoked with significant delay #8894

Closed
alexterman opened this issue Sep 18, 2016 · 6 comments

Comments

Projects
None yet
2 participants
@alexterman
Copy link

commented Sep 18, 2016

Dear Hazelcast folks,

I'm using Hazelcast 3.7.1 and trying to use TTL + eviction listener.
In general eviction listener called with significant delay.
I configured setProperty("hazelcast.internal.map.expiration.task.period.seconds", "1"); as it was suggested in the #7722 issue.

I inserted 100 elements to the map and set TTL to 5. Here the gist with logs
As I can see from the prints my eviction callback called in batches of around 15 items 7 -8 times and the max delay is stands for 12 seconds.

I was expecting 1 batch of all items.
Should I configure more stuff to get it work?

Thanks,
Alex

@ahmetmircik

This comment has been minimized.

Copy link
Member

commented Sep 18, 2016

Hi @alexterman,

These are the most aggressive settings. You can retry by playing them.

  • Run cleanup round in every second
    System.setProperty("hazelcast.internal.map.expiration.task.period.seconds", "1");
  • Scan all entries of map in every round to find expirables.
    System.setProperty("hazelcast.internal.map.expiration.cleanup.percentage", "100");
  • Set cleanup operation count to partition count. 271 is current default.
    This means send one cleanup operation to each partition in every round.
    Otherwise it will pick a small value which doesn't hurt the system and scans partitions gradually.
    System.setProperty("hazelcast.internal.map.expiration.cleanup.operation.count", "271");
@alexterman

This comment has been minimized.

Copy link
Author

commented Sep 19, 2016

Hi @ahmetmircik ,

Thanks for your reply.
After adding the suggested parameters situation improved right a way.
Seems like all entries evicted in one batch and max delay stands for 2 seconds.
A few things I noticed:

  1. After increasing number of inserted entries to 1 million ~20% of entries were delayed, but still 2 seconds delay (delay is not increasing).
  2. The parameters , you mentioned, works only if configuring them in a System level and not instance level (which means all instances will be forced to use those settings).
  3. I couldn't find documentation describing those parameters in relation to eviction and TTL.

Thanks,
Alex

@ahmetmircik

This comment has been minimized.

Copy link
Member

commented Sep 19, 2016

we will try to address 2 and 3 in 3.7.2
Update: Fixes are available in 3.8 but not in 3.7.2

@ahmetmircik ahmetmircik modified the milestones: 3.7.2, 3.8 Sep 19, 2016

@alexterman

This comment has been minimized.

Copy link
Author

commented Sep 19, 2016

Great, thanks.

@alexterman alexterman closed this Sep 19, 2016

@ahmetmircik

This comment has been minimized.

Copy link
Member

commented Sep 19, 2016

we will close it after improvements done, now keeping it open

@ahmetmircik

This comment has been minimized.

Copy link
Member

commented Sep 22, 2016

2 is fixed via #8907
3 is fixed via #8950

These fixes will be available in 3.8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.