Best practice, if any, for workflows? #482
Replies: 3 comments 8 replies
-
|
Hey there! Thank you for your interest in Cronicle. Workflows are indeed coming! It's one of the major features being introduced in Cronicle 2.0 (which is being renamed to Orchestra). It is due for release this year (2022). I have to be honest, this is the first I've heard of rundeck, so I need to do some research and see what it can do, so I understand exactly what you mean by "usability and flexibility". Orchestra Workflows will be an ordered "sequence" of jobs, which is implemented in the UI as a flexible list, with drag-drop reordering, and options you can set for each job. Also, jobs can easily pass data and files to the next job in the list, and control the flow (skip, jump around). Are these the kinds of features you are looking for? I see that rundeck has a text-based workflow editor with an actual interpreted language for decision making and tree branching:
I have to say, this is much different than the workflow implementation in Orchestra. I don't want my users to have to learn a new language to use the product. While this offers great flexibility, it is also a huge leap in complexity, and will be a barrier for new users. Orchestra's workflow implementation is much simpler -- the UI is basically a reorderable list of jobs to run. However, the flexibility is there under the covers. Each job can control the flow by sending commands back to the workflow controller. An individual job can:
You can also insert delays between jobs, and add decisions in the UI based on job outcomes. Each job can also trigger actions like notifications and web hooks. Oh, and each job in a workflow can itself be another workflow (workflowception). You can also choose to run workflow jobs in parallel instead of in sequence, but you lose out on the chaining features in that case. Parallel mode is mainly for "multiplexing" (running a single job on multiple servers in parallel). Please let me know if these features will suffice for your needs, or if I need to implement something else that rundeck provides. Thank you! -- Joe |
Beta Was this translation helpful? Give feedback.
-
|
Update: I dug further into some rundeck videos, and I found a more traditional workflow sequence editor here:
I guess the interpreted language editor is optional, and they also provide a job list. The latter is more akin to what Orchestra will provide, although I must say my UI will be much prettier 😉 |
Beta Was this translation helpful? Give feedback.
-
|
I am very excited about the new orchestra... 🤩 |
Beta Was this translation helpful? Give feedback.
-
|
Hooo boy, I've never jumped into the Rundeck DSL. I primarily use shell commands within a workflow, as shown in your second comment. Passing data from step to step is also a nice thing to have, but not anything I require (yet). It sounds like the 2.0 release will absolutely be what I'm looking for, at least functionally speaking. What's pictured in your second comment are the individual tasks in a workflow, and, if I'm understanding you correctly, Cronicle exclusively schedules and runs those constituent tasks until the 2.0 release where they will be able to be launched as a bundled sequence of tasks (perhaps configurable in a DAG, like OpsWise?). From what I currently understand, tasks are written against plugins. For my purposes, I almost exclusively use local shell commands, albeit they are "local" to the destination server. Rundeck facilitates this as running on a "node" without needing anything actually installed (other than an SSH key) on the destination server. A node is simply a configuration string for a destination server to execute things on using SSH. I'm sure a Cronicle plugin could do effectively the same thing -- like a "Remote Shell Script" instead of "Shell Script". Otherwise it could just, within the shell script, issue "ssh user@targetserver command" or some such. Not ideal, but workable. Otherwise, ultimately, it looks like Cronicle can have an agent on the destination server as a slave, and the scheduled task could simply target that particular agent. This is honestly the correct way to do it, with the master server handling all the actual scheduling, but it's super nice not to have to install things on other servers. The UI and job structuring remains a concern, albeit this is purely from my point of ignorance on basically all things Cronicle. I went as far as sitting down and playing with the category-based arbitrary concurrency and queuing and that's about it. One of the major, major boons of Rundeck is the approachability of the UI for a standard user. Right now, my jobs are sorted out in a Context->Source->JobName style tree/folder view. I find this extremely approachable for non-tech savvy users needing to utilize the automation system to launch jobs. An example of this tree sorting would be CarType->CarCountry->CarMake->CarModel, so we'd have sedan, coupe, truck at the root level. A user would log in and see just these three things. They need to do something with coupes, so they expand coupe. Now they see American, Asian, European. They need to do something with an American car, so they expand American, and now they see Ford, Chevy, Dodge, etc.. you get the idea. From a support/maintenance standpoint, I can be told "hey, something's wrong with the Ford Focus job" and I'd be able to quickly and easily navigate to that job. "Categories" in Cronicle are a start to this, but I would like to see more powerful sorting of jobs to reduce visual noise to a user. My system currently runs about 2,000 job executions per 24 hours across around 160 jobs that organize about 400 tasks -- these are very tiny numbers, for reference. As it stands in Cronicle, I'd need to nebulously chain these 400 tasks into their 160 jobs, but if a user wanted to fire off one of those 160 jobs? They'd need to (know how to) find the "first" job in that chain and kick that off. No go. I want to show them these digested categories -- further, I want to use these digested categories so I can maintain jobs that much easier! That's how Rundeck does the UI. Airflow is a half step ahead of where Cronicle currently is with its tagging/chiclet display for jobs. I would be interested in any sort of available alpha/beta of the 2.0 release should it be available. I'd love to do a proof of concept in our stack to see how it meshes with our needs. |
Beta Was this translation helpful? Give feedback.
-
|
Oh and as far as opswise, it's a stupid expensive enterprise scheduling application. Extremely heavy, but hilariously powerful. Building a job was done in a DAG style view, complete with arbitrary resource management, incredible logging, and APIs to leverage. It was very much not a self-service app though, like Rundeck poises itself to be. I feel it is prudent to explain what I mean by "self service" -- my first foray into Rundeck was a company's prodops team spinning up an instance to allow other departments to do server management within specific parameters. You'd log in and see a job to spin up an ephemeral environment, for example. Launch it, it'd ask some parameters like your initials, how long you need the environment to be active, what configurations and applications it should adopt from a multi-select input field, etc.. All these parameters are coded by the administrator of Rundeck in this case, so you can railroad your users into very specific usage cases and take a nice, hands-off approach to what would otherwise be a ticket request needing to be read, interpreted, and fulfilled by a human running the very same build script on a server somewhere. In terms of my uses, I have users allowed to log in to my rundeck instance and all they see are the reporting-specific jobs (which is any job with "Report" in the tree path, to circle back to that tree view I keep talking about). They can fire a job and optionally change the recipient email list for that specific instance of the job. No more calling me at 7AM to rerun a report! :) |
Beta Was this translation helpful? Give feedback.
-
Oh I know, I get it. Rendering a tree view UI in the browser is not the problem. I've coded many over the years. It's a matter of changing the app design to accommodate this. Not only the job list page, but also search, filtering, bookmarking, creating the hierarchy, and presenting all that in a clean way with good UI / UX. Also, regarding the job list UI, Rundeck merely lists the job title on each row. Orchestra's event view is a full-width table with many columns, showing all kinds of things. I've already put many months into the design and implementation, with the assumption that categories are "flat". It's just going to be a lot of work to change everything around to support nested categorization / tree view like this. I will add it to the TODO list and start working on it, but I can't promise this for the initial release. It's a large base change to an app that is more than 3/4 complete at this point.
Thank you very much for the offer! I am currently coding Orchestra offline in stealth mode, but after release the open source version will be up on a public GitHub repo like Cronicle is. I should point out that a user by the name of @mikeTWC1984 has forked Cronicle, and created his own version with lots of new features, including workflows. You might want to check that out, if you don't want to wait for Orchestra: https://github.com/cronicle-edge/cronicle-edge |
Beta Was this translation helpful? Give feedback.
-
|
I totally get what you mean about self-service usage, and creating a "form" for users to fill out to run a job. That is a feature I haven't considered yet, so I will add it to the TODO list as well. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Right on. I very much look forward to the Orchestra release -- where should I be watching for that? I'll also give cronicle-edge a look-see, thanks for the pointer. |
Beta Was this translation helpful? Give feedback.
-
Thanks again! I'll post a new discussion topic here, augment the Cronicle README with a notice at the top, and also tweet from the CronicleApp Twitter account, as well as my own, when Orchestra drops. I guess I should start a new Twitter account for Orchestra, too! I'll work on that. Oh, and Orchestra's official website will be: https://orchestra.dev/ (currently under construction). |
Beta Was this translation helpful? Give feedback.
-
I've been hunting for a good automation system for a while now. Cronicle seems flexible in the right ways, but the missing key here is the concept of workflows.
Cronicle seems to be built around scheduling specific tasks, and sure a task can call a second task, which can call a third, and so on, but this kind of setup is buried and nightmarish to support in a production setting where there are hundreds of jobs each with a half dozen steps.
Basically, I'm looking for the usability and flexibility of something like rundeck, but with the arbitrary job/task level concurrency and queuing of Cronicle. Is what I'm looking for beyond the scope of Cronicle?
Beta Was this translation helpful? Give feedback.
All reactions