Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Queue.size does not decrease if MapStore is attached #172

Closed
alexey-krylov opened this Issue · 9 comments

4 participants

@alexey-krylov

Currently we have hazelcast.xml like this:

<group>
    <name>dev</name>
    <password>dev-pass</password>
</group>

<network>
    <port auto-increment="true">5701</port>
    <join>
        <multicast enabled="false"/>
        <tcp-ip enabled="true">
            <interface>127.0.0.1</interface>
        </tcp-ip>
    </join>
</network>

<properties>
    <property name="hazelcast.mancenter.enabled">false</property>
    <property name="hazelcast.rest.enabled">false</property>
    <property name="hazelcast.shutdownhook.enabled">false</property>
    <property name="hazelcast.version.check.enabled">false</property>
    <property name="hazelcast.jmx">true</property>
    <property name="hazelcast.logging.type">slf4j</property>
    <property name="hazelcast.version.check.enabled">false</property>
</properties>

<queue name="MessageStatesLog">
    <backing-map-ref>MessageStatesLog.backing-map</backing-map-ref>
</queue>

<map name="MessageStatesLog.backing-map">
    <time-to-live-seconds>5</time-to-live-seconds>
    <map-store enabled="true">
        <class-name>CatchingMapStore</class-name>
        <write-delay-seconds>15</write-delay-seconds>
    </map-store>
</map>

In our case we are putting many objects to MessageStatesLog queue, which is MapStore'd in the database.
JMX-console definitely shows me invalid size of this queue - it is never decreases, only increases. Also i see correct counters for backed map - q:MessageStatesLog -it's size is increases and decreases during load.

This issue is a real problem for usage closely with "max-size-per-jvm" property - if queue size goes to this, then all next 'puts/offers' is never be done.

We need some kind of "black hole" functionality - queue with MapStore, max-size-per-jvm and TTL specified. I expect that all objects that was putted will be sent to MapStore and Queue and backing-map will forget about them.
Maybe this case is not trivial, but i don't see any limitations to this from the documentation.

@alexey-krylov

Hazelcast version is 2.1 (02.05.2012), OS Windows Server 2008 R2

@mdogan
Owner

I could not understand the actual problem here;

Do you mean

  • JMX stats for queue are wrong?

-OR-

  • Items polled are not removed from queue, so queue size consistently increases?
@alexey-krylov

We have a Queue with backing-map. In backing-map we have 5 seconds TTL and MapStore.
I expect that items will be removed from the Queue and backing-map after 5 seconds.

Currently in JMX stats for queue i didn't see that - queue size is only increasing.

@mdogan
Owner

Maps have dedicated cleanup threads periodically cleaning invalid/expired/removed entries. But queues do not have, expired items are removed from queue during poll/take operations. (Queue items are just long keys, actual data are stored in map.)

Do you poll/take from this queue ever?

@alexey-krylov

Maybe it's possible to remove from queue on server-side when item is expiring on the backing-map and we know that this map has an queue 'owner'?
As i said in issue description - i'm trying to create some kind of 'black hole', so there is no component doing poll/take for this 'black hole' queue.

@mdogan
Owner

Sure, some enhancements can be done. But continuously adding items without polling/taking is not a common use case for a queue.

@ppeddi

I don't think this works for Map either. We have a map backed by persistence with TTL. I see that the item is not removed from persistence storage after the TTL. Worse is that when I call get() on the map, hazelcast doesn't have it but it loads from the persistence storage so the effect of TTL became void essentially.

If TTL is suppposed to work with persistnce storage also, then its a bug.

@mdogan
Owner

TTL causes item to be evicted. Evictions don't trigger map-store.
Only explicit remove call make item to be removed from map-store.

@enesakar
Owner

Can you try with version 3? Reopen the issue if needed.

@enesakar enesakar closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.