Remove Django Dependency

[dcramer]

This ticket describes removing the Django dependency on the Sentry client. The server itself could potentially be rewritten to use generic libraries, for for now it's not important. The cilent ideally should work as a WSGI middleware and be able to dispatch errors via HTTP POST to the server (which would be the standard).

[adamn]

What would be the ORM? I personally am fine having SQLAlchemy be a dependency but it seems like you'll have to choose SQLAlchemy or Django as a dependency.

[dcramer]

I honestly wouldn't mind Django as a dependency if it wasn't so damn large. Batteries included was once nice, and now is just bloat.

[adamn]

By large you mean size - meaning that it takes a while to download from PyPI on a new machine? If so, then it makes sense to just leave it depending on Django for normal people and big shops should just host their own PyPI on their own network so that size doesn't matter much.

[dcramer]

It's still kind of up in the air. There's a few things I'm looking at with Sentry 2:

• ZODB as a default storage mechanism
• Built-ins for Redis, Mongo, and other popular capable key/value stores
• Potentially just simple backends using psycopg2 and mysqldb (as we dont need full-on ORMs since the schema is now very simple).

[adamn]

I would say that the idea of a pluggable backend is great (although we're happy with MySQL on this front). I wouldn't do psycopg2 or mysqldb directly though - it's much better to implement a manager on the models with raw sql that runs efficiently on pg or mysql

[SEJeff]

Zope is a dead weight. Please do not go down that path.

[matclayton]

Any thoughts on adding Cassandra as a backend, could be a good fit, with high speed writes and being sort of k/v store based. I've not had time to look at the sentry datamodel to see what the fit is like. Also Mongodb and capped collections might be interesting as well, if you log a lot and dont want to keep large amounts of historic data.

[dcramer]

I think Cassandra and MongoDB would be great targets for the initial backends (or at least, post-release candidate supported backends)

[dcramer]

The default implementation is now going to be Redis. I'm not sure (as of release) if we'll ship with more than that, but it should be pretty easy to extend.

We still need to consider how the Django integration will work, but it will likely be something like sentry.contrib.django, which literally just plugs in an exception handler and a request parser.

[adamn]

You're talking about the default backend for the sentry messages, right?

It seems like an app at Sentry's level should not be using any direct implementations for a datastore backend, but rather an abstraction such as: the Django ORM (current), SQL Alchemy, django-nonrel, or lower level with django-dbindexer or djangotoolbox.

[dcramer]

@adamn I think SQL Alchemy would be a great approach, but Sentry will have to have its own general ORM (though not a SQL ORM) to actually map down to those levels. There will still be some odd behavior if people use a custom SQL backend in Django that isnt supported by SQL Alchemy, but that's not a big deal.

As an example, Sentry has unique operations such as "add message to group set", "increase count of message", etc. which can translate to most engines, but translate differently. For example, in Redis, we can use zsets and neat toys like that, but in SQL we rely on indexes rather than generating our own.

[SEJeff]

Wait at this point are you moving the sentry server away from Django? Just curious what that solves if you did that. It makes perfect sense why you'd want to remove the Django dep from the sentry client, but I can't see it for the sentry server.

[matclayton]

Random thought which came up the other week when we were moving datacenters. One of the issues we have with sentry is the need to maintain an additional deployment. We were bouncing around the idea of running sentry on appengine. This would have the advantage of allowing new users to get up and running quickly, and for 90% of users would probably be fine with the free quota. Just wondered if you'd given any thought to go down this route, or if the data model is too alien from what you are planning with redis.

[dcramer]

@matclayton I'm not entirely sure how well Big Table would work, but with the new data model it's designed to be pretty darn flexible. I'd like to think it's probably doable (depending on what dependancies we require)

@SEJeff the main problem with Django is that the dependency is huge. It's a very large download for a lot of components which frankly we just don't need.

[adamn]

@dcramer Maybe it makes the most sense to make a new project that works as an abstract backend for Sentry and that project would plug into redis (or MongoDB, etc..). That way, non-Sentry projects can use things like zsets in a more abstract way.

In other words, I'm proposing a new Python package that acts as a NoSQL abstraction layer with backend support for Redis, MongoDB, SimpleDB, etc..

[dcramer]

Sentry 2 no longer requires Django, but still works as drop-in.