Currently not webscale

[avelardi]

Without a NoSQL backend, I'm not seeing a clear or easy way to make this project webscale. I see the MySQL connector exposed, but that doesn't offer proper sharding or performance increases when scaled. At the rate this project is gathering momentum, I fear we will hit the theoretical limits of the backend. Redis/MongoDB is the clear solution to all of these issues. A quick comparison will show webscale technology is the clear way to go with built in sharding modules and double hashed memory storage:

Better performance can be achieved by running Redis in front of it so we can then store all values in the queue. On top of that, we can have varnish cache additional content for even better scalability in the end.

I'm gonna start working on a clean transition version so we can migrate to a solution better suited for the cloud.

Note: In addition to moving to Varnish/Redis/Mongo (the defacto webscale stack in the cloud to date) we'll also enable sha-1 encryption keys during memory transport to ensure all data is securely stored in ram persistently, but this will need to be another issue.