New cassandra backend

[piotrmaslanka]

As per #2669 this is a Cassandra backend using DataStax python-driver.

For compatibility reasons it's called new_cassandra, and not cassandra. Configuration options also differ, mainly to force users to understand that it's a different thing.

Using old cassandra backend will raise a deprecation warning.

Schema is different from Thrift one. Assuming keyspace exists, column family will be created by default (fixes #1763).

[malinoff]

Please, avoid mutable class-level objects. IMO we don't need all these connection-related parameters on class level.

[thedrow]

Can you please rebase?

[piotrmaslanka]

Rebased all right.

[egon1024]

Just curious if there's any sort of timeline on getting this into Celery? We're looking to use it in a new project that's ramping up right now (Celery with Cassandra results storage, but on Python 3 so pycassa isn't an option).

[thedrow]

I'll speak with @ask about it first but it looks good.

[ask]

How about we keep it as new_cassandra in 3.1, but drop the old one in 3.2?

[ask]

Good work! Tested it and it seems to work. Merged into master

[thedrow]

@ask That would kinda break semantic versioning that I think we should adopt.
Keep it as new_cassandra for now.

[egon1024]

Fantastic! Thanks for merging this folks! Any idea when the next release will be that will contain this?

[thedrow]

@egon1024 Sorry, we don't yet. Unfortunately we have some pretty critical bugs to resolve before we release a new major version.
I'm also working to rewrite our test suite to be more extensive and include end to end tests.

[egon1024]

No worries - keep up the great work, it's very much appreciated. :)

[ask]

@thedrow 3.2 should probably be 4.0 at this point.
What do we do when the new_new_cassandra backend comes?

semantic versioning is good, but adding experimental features to point releases can be useful, but only
if it's completely orthogonal to existing features (like the qpid transport, or new result backends).

[thedrow]

If it's 4.0 then we can rename the backend. Yes.

[davidharrigan]

@ask @piotrmaslanka My team is looking to use Cassandra for our Celery backend, but I'm not sure if our timeline matches for when this feature becomes stable, so I am looking to patch up our local mirror of Celery with this feature. What are the steps that I should take about doing that? I also don't see any code diff in this PR that would enable new Cassandra backend.

[piotrmaslanka]

@davidharrigan I guess celery/backends/__init__.py and celery/backends/new_cassandra.py are the only files to add/modify in this case. celery/master is fine to fork from, just please patch against #2901 or wait until this is integrated. I will be running this backend in production from 5th Nov (it passed stress tests on staging), so if there are any bugs I'll get to fix them ASAP.

[davidharrigan]

Awesome, thanks for the quick response :D

[piotrmaslanka]

@davidharrigan There was a small revolution in the master - old cassandra is no more. Please be aware of that, and repatch against #2901. I rebased it for simpler inclusion.