-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dagobah's features compared to chronos or azkaban? #23
Comments
Hey there! Unfortunately, I'm not familiar with chronos or azkaban, but by a quick glance at their product pages they seem to be good examples of why other scheduling projects weren't great fits for me. Namely:
Now, to your questions:
|
This is the exact reason why I liked dagobah. Its simple and is flexible enough to be used for other usecases. I am also looking for a cron replacement. Azkaban is meant for hadoop job automation but it can also act as a cron replacement (with retries, DAG etc). Chronos is awful and completely an overkill.
You got it right. Dagobah currently runs on one machine. But I am trying to add fabric to Dagobah so that it can execute commands on a remote machine. Celery is good too, but the management overhead is less if I just use fabric and execute remote commands.
Good to know, I will try to expose this data in the web app. |
I have added remote task execution here: https://github.com/utkarsh2012/dagobah/blob/master/dagobah/core/core.py#L634 it works nicely with the existing stuff (UI needs some work like updating remote machine endpoint, needs tests). It uses paramiko and spawns processes for every remote request. What do you think? |
Hey @utkarsh2012, this looks awesome! I will review your branch when I get some time and get back to you. |
Looks like Travis CI build is broken, missed to add dependency for paramiko in setup.py and requirements.txt. Also the UI might need some work, I hacked up the solution in a day to see how will it work. I might submit more fixed if I find bugs. |
Did this ever get included? I can't seem to find remote options in the web UI |
Just a minor correction, Azkaban can be used for anything, not just Hadoop based Map-Reduce. It can be used only as a superior cron (dependencies, transparency, partial workflows, etc...) only. You don't need hadoop at all. |
dagobah is exactly what I was looking for, its much simpler (than chronos or azkaban) but also not similar to them.
So some questions/suggestions:
I am playing around with Dagobag and I am trying to work on 1 and 2 now, not sure about the timeline though :)
The text was updated successfully, but these errors were encountered: