RiakBucket creation *still* not free of race conditions [JIRA: CLIENTS-439]

[Tinche]

We see None being returned from RiakClient.bucket(..) a few dozen times per day on our worker processes. This is probably exacerbated by the fact we're forcing garbage collection in another thread periodically.

This has been tackled in f944a8b by using WeakValueDictionary.setdefault, but until http://bugs.python.org/issue19542 is merged in (status still open), setdefault isn't atomic itself, hence a race condition returning None is possible.

[hazen]

Thanks for reporting this, @Tinche. I was going to say, I thought we already addressed this. Looks like another solution might need to be crafted in the mean time.

[ovaltzer]

Hey @lukebakken \ @javajolt
Just facing the very same issue here. we are using riak client for python (2.7.0).
Does instantiating the client per thread should be good enough to solve this issue for sure ? was it tested ?
We tried that and almost sure that this problem was recreated even though we have a new client for each thread.

@Tinche Have you find your way around this somehow ?

Thanks for the help.
Oded Valtzer

[lukebakken]

Would it be possible to provide code that reproduces this reliably? Thanks.

[ovaltzer]

Hey,
It's a very basic command, while running in python multi threaded envrionment.
The code that fails is this:

def save(self, key, value):
        try:
            ro = self.bucket.new(key)
            ro.data = value
            ro.store()
        except Exception as e:
            raise RiakInterfaceException(
                "Failed storing [key {}] to [bucket={}] riak. error:{}".format(key, self.bucket_name, e))

with RiakException:

error:'NoneType' object has no attribute 'new'

This save function is part of an internal wrapping of a riak bucket object. we create the bucket object self.bucket = riak_client.bucket_type('x').bucket('y') at the moment before we use the save method, so the bucket object is not 'hanging' for a while.
moreover, we create a riak client object for each thread we open.
This code works in most of the times, but sometimes, on different buckets, we get this errors. highly unexpected and only reproduced in multi threaded environment. Is the RiakClient python library use any global unsafe objects?
Do we need to maybe close someway the riak client before the thread terminates ?

Thanks,
Oded

[lukebakken]

running in python multi threaded envrionment

Can you describe this more? Are you using the threading module? Which version of Python are you using?

we create a riak client object for each thread we open

The RiakClient object is thread safe, so only one instance is necessary. You must have a new RiakClient for each operating system process, of course.

Is the RiakClient python library use any global unsafe objects?

It does not.

The fastest way to get a fix is to provide a 100% complete set of code that reproduces this issue, thanks.

[ovaltzer]

We started with a global riak client object and we encountered these problem and found out about this issue above which precisely says that there issues with multi threading.
We are using standard python Thread Pool on top of the tornado event loop.
the version of python we use is 2.7.

Thanks for the quick responses