Near Cache: Race Condition in 3.1.6

[lukasblu]

Hi,

we discovered a race condition which was introduced from 3.1.5 to 3.1.6. The following code reproduces the issue. Simply run it twice (meaning start it twice in parallel) on your development laptop. If you run it with 3.1.5 it works whereas it fails in 3.1.6.

This issue might be related to issue #1829

package com.nm.test.hazelcast;

import com.hazelcast.config.*;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IMap;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class TestPutIfAbsent {

    private static final Logger logger = Logger.getLogger(TestPutIfAbsent.class);

    public static void main(String[] args) {

        // vars
        final int numThreads = 32;
        final int numIterations = 5000;

        // configure logging
        BasicConfigurator.configure();

        // create map
        Config config = new XmlConfigBuilder().build();
        config.setProperty("hazelcast.logging.type", "log4j");
        config.setProperty("hazelcast.initial.min.cluster.size", "2");

        // configure near cache
        MapConfig mapConfig = config.getMapConfig("testMap");
        NearCacheConfig nearCacheConfig = new NearCacheConfig();
        nearCacheConfig.setMaxSize(0);
        nearCacheConfig.setTimeToLiveSeconds(0);
        nearCacheConfig.setMaxIdleSeconds(0);
        nearCacheConfig.setEvictionPolicy("NONE");
        nearCacheConfig.setInvalidateOnChange(true);
        nearCacheConfig.setInMemoryFormat(InMemoryFormat.BINARY);
        mapConfig.setNearCacheConfig(nearCacheConfig);
        HazelcastInstance hcInstance = Hazelcast.newHazelcastInstance(config);
        final IMap<String, String> map = hcInstance.getMap("testMap");

        // create threads
        List<Thread> threads = new ArrayList<Thread>();
        for (int i = 0; i < numThreads; i++) {
            Thread thread = new Thread(new Runnable() {

                @Override
                public void run() {
                    Random random = new Random();

                    // loop over keys
                    for (int k = 0; k < numIterations; k++) {
                        int keyInt = random.nextInt(1024);
                        String key = String.valueOf(keyInt);
                        String value = "merhaba" + keyInt;
                        map.get(key); // TODO this causes the problem
                        map.putIfAbsent(key, value);
                        String value2 = map.get(key);
                        if (value2 == null) {
                            throw new RuntimeException("Value should never be null here.");
                        }
                    }
                    logger.info("Thread '" + Thread.currentThread().getName() + "' done.");
                }

            }, "put-if-absent-test" + i);
            threads.add(thread);
        }

        // start threads
        for (Thread thread : threads) {
            thread.start();
        }

        // join threads
        for (Thread thread : threads) {
            try {
                thread.join();
            } catch (InterruptedException ignored) {
            }
        }

        // stop hazelcast
        hcInstance.getLifecycleService().shutdown();
    }

}

The map.get() before the map.putIfAbsent() combined with a near cache causes the problem.

Thanks for looking into this and best,
Lukas

[noctarius]

Thanks for the report, is there a chance you can test this with 3.2-RC1, too?

[pveentjer]

Thanks for this excellent report. I'm currently working on fixing some other tests. once done I'll have a look at this unless someone else picks it up.

[pveentjer]

With 3.2-RC1 I get the following:

 Exception in thread "put-if-absent-test12" java.lang.RuntimeException: Value should never be null here.
at com.hazelcast.TestPutIfAbsent$1.run(TestPutIfAbsent.java:67)
at java.lang.Thread.run(Thread.java:744)

Exception in thread "put-if-absent-test5" java.lang.RuntimeException: Value should never be null here.
at com.hazelcast.TestPutIfAbsent$1.run(TestPutIfAbsent.java:67)
at java.lang.Thread.run(Thread.java:744)

We'll look into it. Will be fixed for the 3.2 release and we'll see how fast we can fix it for the 3.1 release. are in bugfixing mode.. so we'll polish them away quickly

[lukasblu]

Cool, thanks for your work.

Now, I spent some more time looking into this issue and while doing so, stumbled accross two other places where I think the code accessing the NearCache is not fully correct or could be improved. Since I'm not fully sure if it is a real issue (I only looked at the source code), should I point you to these two places here or open separate issues for them?

Thanks and best,
Lukas

[mdogan]

If it's not directly related to this *putIfAbsent() *race condition, then I
think you should file separate issues..

@mmdogan

On Tue, Feb 25, 2014 at 2:44 PM, Lukas Blunschi notifications@github.comwrote:

Cool, thanks for your work.

Now, I spent some more time looking into this issue and while doing so,
stumbled accross two other places where I think the code accessing the
NearCache is not fully correct or could be improved. Since I'm not fully
sure if it is a real issue (I only looked at the source code), should I
point you to these two places here or open separate issues for them?

Thanks and best,
Lukas

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1861#issuecomment-36068315
.

[lukasblu]

Okidoki, see new issue #1863

[lukasblu]

Well, back to this issue...

map.get(key); // here the NULL_OBJECT is put into the NearCache
map.putIfAbsent(key, value);
String value2 = map.get(key); // when the race condition occurs, then here we still see the NULL_OBJECT
if (value2 == null) {
  throw new RuntimeException("value2 should never be null here.");
}

Assuming my two comments in the code above are correct, it looks like map.putIfAbsent() does not synchronously remove the NULL_OBJECT from the near cache. I tried to find where the invalidation of the near cache is triggered after the putIfAbsent(), unfortunately I was not able to discover it... I hope this still helps to solve the issue.

Best,
Lukas

[lukasblu]

Hi @ahmetmircik

it could well be that my fix for issue #1905 would also fix this issue here.

Cheers,
Lukas

[ahmetmircik]

#1922 also solves this issue. Closing now.