RavenDB Support

[devmondo]

Hi,
this is awesome library, thanks for it.

is there a possibility to use ravendb as back end storage instead of Redis ?

thanks in advanced.

[devmondo]

thanks very much for the detailed reply :)

the problem is as you said, Redis has no official support on windows, will require more pluming in Production.

we have been using RavenDB for 2 years now, though initially it had many quirks, now it is very stable product plus in version 3.0 which will be coming 2014 it will have Owin host support, and Mono, Java Support, so it will be multi platform.

also RavenDB support is now implemented by many other third party products like NServicebus.

also RavenDB has great management studio, uses Linq, Full text search, and works on HTTP, and built in Caching, which i think will be huge for your library and you will have many future scaling ideas.

i think if you create some kind of Repository pattern then it will be easy to implement what ever Backend DB System, SQL server, RavenDB, Redis etc...

what made me interested in your library is that on the description page it is mentioned that it can be used for Mass Email sends, and this is exactly the problem i am facing now and trying to resolve so when i saw that i got so excited as your product seems so easy to use and has awesome great set of features for a version 1.0 and i am sure it will be improved.

thanks again and i am really excited to use this. :)

[odinserj]

I've read something about RavenDB. Its following features are great and will help to build support of RavenDB without violation of HangFire principles:

• Changes API will help to build queues without polling semantics (that sucks).
• Expiration bundle will help to keep the system clean of old tasks.
• Pessimistic locking will help with distributed servers.

[devmondo]

awesome man,

one more good thing, is that RavenDB have Embedded mode, so when you want to use it in small projects, then you don't have to set it up on IIS, it is all smooth, even you can run the management Studio in embedded mode.

regarding using Expiration Bundle, true it is great, but you have to be aware about one Fact when using the Bundles, which is, you have to set it up from the start, in other words, when you create the Database and before you start Saving Data. as some bundles will require some modifications that will lead to some undesired effects if not installed from the start.

which leads me to the idea that, if we want to use Hangfire and make it has it is own Database in Raven, then this is fine as you can set that up in advanced, but if we want to save HangFire Data into the Application existing Database then we will have the problem above, as not everyone will install the expiration bundle in advanced.

cant really wait for this to support RavenDB, and if you need some help, their Google forum is so active and you will always get answers.

all the best mate :)

[hahmed]

Thanks for this library, it looks really good.

I want to use some sort of backend system too, for processing my jobs, my server is windows based and my application is asp.net/mvc. Not sure how complicated it is to set this up for me...

[odinserj]

(I've cleaned this topic a bit to remove obsolete messages)

Storage abstraction is done. SQL Server and Redis have a lot of differences and I hope that abstraction is fine. Storage implementation is based on the following topics:

1. Basic types. Jobs, job parameters, states & queues. It is the simplest thing to implement, but operations on these types should be as fast as possible.
2. Data types. Counters, Sets, Values, Hashes, Lists – they are emulate Redis data types and increase extensibility. Some states, filters, state handlers use them.
3. Distributed locking. Redis does not support ACID transactions. To avoid data inconsistency, each job is being locked during the transition of its state. It is possible to run multiple HangFire Servers at the same time on different machines, so these locks should be distributed.
4. Atomic writes. Redis does not support ACID transactions. But when job is locked and no one could change its state in the middle, we could use Redis transactions (that are write-only). This guarantees that the state will be applied or not applied at all.
5. Connection pooling. Connection lifetime made as smaller as possible. They are created very frequently. That is why some form of connection pooling should be used.
6. Monitoring API. The most of HangFire storage operations are fine grained. However, when you are trying to use fine grained API for statistics pages, you begin to issue an enormous amount of queries. So, it is better to use coarse grained APIs for statistics, where one query can do the most of the work.
7. Cleanup expired things. To don't force users to maintain HangFire, it uses expiration dates. All expired entities should be removed.

To provide implementation for your storage, you should add the implementation for the following types:

• JobStorage class.
• IStorageConnection interface.
• IWriteOnlyTransaction interface.
• IMonitoringApi interface.

You can see an example of the storage implementation here. I have no plans to provide implementation for other storages, it takes a lot of time (but much lower than to implement background job system 😃).

[devmondo]

thank you very much for the updates

[odinserj]

@hahmed, @devmondo, how r u? What are you using for background processing? I think HF model is stable enough now and going to grow the community. And want to start from tutorials. Could you tell me the difficulties you encountered with HF? What questions do you have? Thanks in advance!

[devmondo]

hi man,

we use Quartz here and some .NET Async, but it is far from being Ideal, we have tried NService bus, to huge to expensive.

hang fire looks like and awesome alternative but we need Docs and Tuts, without it we are pretty much left with only reading source code and tests, and personally i cant wait for that to happen.

one side note, where did you get your Avatar from, i love it it is hilarious :)

[hahmed]

@odinserj I am cool thanks.

I wanted to use hangfire but I ended up using a custom and very simple background processing system - mainly due to time constraints to be honest with you and understanding what I did was far easier and tightly integrated with my workflow.

Some background work is coming my week next week, I can pretty much use this library as is, I did have difficulty getting this to work, I wanted to try the redis version and the sql version and see which feels best to me. We will see how easy this is, I will see where I can pitch in and help with docs and stuff.

To be honest with you, I have never used redis before and just need to understand that a bit better too.

Like for example - redis is all in memory, what happens when you restart the machine, do you lose all the jobs in the queue? (Shows my lack of understanding I know)

By the way, do you use servicestack in the backend for redis? Is there licensing a bit less open source now? This was something that I had a question on way back...

[odinserj]

First of all, thank you! You made me change my mind about SQL Server support.

I made a new version of http://hangfire.io. It contains the philosophy of HangFire together with features and advanced features description. These pages are not so detailed as documentation, however, they can shed light on the main functions.

While I'm working on introduction to HF, I can tell you in short about some of the finer points of HangFire processing (please, read the pages above, they clarify many points):

1. Redis is about 4x-10x times faster (all operations) than SQL Server on my MacBook Pro. But requires knowledge about its maintenance and Linux machine, because Windows ports are not production-ready.
2. Job creation is not free, but it is optimized as much as possible (even in SQL Server), try to test on your servers.
3. Job performance is not free either, and it rests in the storage (I've profiled it heavy using dotTrace, SqlProfiler and sql performance dashboard). You should test it yourself. And you can scale the processing across multiple servers. Storage sharding is feasible, but I don't want to do it right now.
4. Redis-based worker receive a job practically without any delay (blocking command is being used), SQL Server uses polling now (see Use SQL Server Service Broker when possible to get new jobs #52), you can choose the polling interval when constructing the storage instance through options.
5. Your jobs can be retried even if you applied an [Retry(0)] to them – in rare cases of thread abort, if ShutdownTimeout (30s by default) was expired on application shutdown (see reasons). You should consider to make your background methods idempotent.
6. You should be prepared for stale data because it can be changed after queueing and before performing.
7. For dependency management and if you familiar with IoC containers, please see HangFire.Ninject or HangFire.Autofac. Or use static methods instead.
8. Application pool becomes idle when there were no requests for 20m by default. Jobs can be unprocessed (but they will be processed anyway after restart). You can tweak this parameter.
9. Logging and monitoring require some code yet. But I can tell you what to do.

Regarding to Redis, you should read its documentation, it is really great. Here is the topic about persistence models. Shortly, with RDB enabled you can restart your machine. With AOF enabled, you can reset your machine. When your HDD or SSD (or what do you use) became broken, you need to do much more things (as in SQL Server).

Please, tell me what item to detail, and I'll do it. And I plugged disquis on documentation site, so please ask any questions. More questions – more information for me. And this information will lead to documentation 😄

PS. @devmondo, I bought my avatar on http://istockphoto.com 😃

[odinserj]

Regarding to Redis. Yes, HangFire.Redis uses ServiceStack.Redis package. The author says the following:

so all 3.x releases will remain as BSD now and in the future. The AGPL/FOSSEX only applies to v4+ which you will be able to use for free for OSS projects using one of the listed OSS licenses (e.g. BSD).

So, you can use 3.9.x in proprietary projects. And 4.0 is dual-licensed. AGPL is very restricted, proprietary use is prohibited. But you can buy commercial license.

HangFire.Redis is linked to 3.9.x version, and I have a plan to choose either Booksleeve or StackExchange.Redis (other assemblies do not depend on ServiceStack.* anymore). And there is one known issue, #53.

[devmondo]

hey man, thank you very much, the new page looks a lot better really and more simplified.

based on my reading the way i see it, is we can use redis for now and then change to any storage later when it is needed, correct ?

sorry for the lame question, but to use redis for now, what we have to do?

so if this work out for me, will it let me replace NserviceBus ?

thanks again

[odinserj]

Why do you think to use Redis? You can now use SqlServer, it is enough until you need really great performance. Then you can switch to Redis.
I can't tell you about replacing nservicebus with hf. If you are using nsb only as a protocol between your web app and your workers, then you can replace it. How do you use nsb?

[devmondo]

thanks for reply man,

regarding redis, why not ? does it not offer better performance based on what you mentioned above ?
if it is a simple case of installing Redis and add some connection string in web.config then why not ?

in terms of NService Bus, we use it for the usual stuff, send mails, Payment gateways, we don't use the fancy stuff there, as i think it is for really large enterprise stuff, correct ?

[odinserj]

Ok, why not :) But there is no simple case of installing Redis server on Windows, you should run it on Linux system, either on a physical or virtual machine (I'll clarify the documentation about this case, thank you).

To install the Redis version of HangFire, you can simply do:

1. Install-Package HangFire
2. Install-Package HangFire.Redis
3. Uncomment the line that starts with JobStorage.Current = new RedisStorage in App_Start\HangFireConfig.cs (and comment SqlServerStorage creation line).

Regarding the use of NServiceBus, it seems to me that you are using it just to start the fire-and-forget tasks to perform them outside of the request processing pipeline, in the background. In this narrow case, you can use HF instead. Enterprise Service Bus pattern can greatly decouple multi-service communication, but it is too complex just for background jobs. On the other hand NServiceBus is a mature product with good documentation, community and support.

To be honest with you, I should warn you about the current version of HangFire. It is not 1.0 yet, it is not production-ready for everyone use, and there could be some unknown issues. However it provides much simpler development and debugging (and is fully transparent to you, check integrated web monitoring), because it is much simpler itself. We are using it on production, however you should decide to use it or not by yourself.

P.S. I suggest you to close this off-topic issue and move to the HangFire room at JabbR.

[devmondo]

thanks a lot for the detailed info man, i really appreciate it, will close this and move to Jabber as you suggested

[ghuntley]

+1 for RavenDB support.

[OV2012]

++1 for RavenDB support

[laedit]

👍 for RavenDB support

[AlexRetzlaff]

👍 for RavenDB support

Any takers for a open source project (I would contribute 1 oDesk .Net dev to kick it off)

[mjohnson0580]

+1 for RavenDB support

[bar10dr]

+1 for RavenDB support

The thing with Raven is you could do it all using nuget installs, it would take 10 seconds to set up a working prototype instead of having to set up databases and all that mess.

[vip32]

+1 for RavenDB support

[stevo-knievo]

+1 for RavenDB 4 support :)