build: replace kit with task

[agilgur5]

Motivation

• Task is a very popular (~9k stars), well-maintained (has releases at least once a month in the past 6 months), well-documented, Go-based task runner with a nearly identical spec to kit, as can be seen in the diff

  • basically can do virtually all of the same things and a good number of other things, with lots of docs and active maintenance

• Comparatively, kit was only created in the past year by Alex, is not well-known (~40 stars, and anecdotally, I'd never heard of it before), and seems to be sporadically maintained

  • It also has some unfixed bugs like Artifacts not set in Status.Nodes | ArtifactGC prevents Workflow deletion #11879 (comment) and has had other bugs like kit not downloading #10768 etc
    • Discussed some of these over Slack as well

To be explicit, I'm not big on Task or anything (Earthly is my current go-to, as mentioned on Slack etc) -- it was just the most similar and most popular alternative I knew of

Modifications

• intentionally made this PR have as few changes as possible to reduce the diff and make it look very familiar

  • used one-line arrays instead of multi-line to be similar to kit's make-like "stringy types"
  • used singular cmd syntax instead of the usual plural cmds syntax

• then followed Task's styleguide:

  • name as Taskfile.yaml: https://taskfile.dev/styleguide/#use-taskfileyml-and-not-taskfileyml
  • new lines between tasks: https://taskfile.dev/styleguide/#add-spaces-between-tasks
  • consistent ordering of keywords: https://taskfile.dev/styleguide/#use-the-correct-order-of-keywords
    • strangely it doesn't mention keywords within tasks, but I did cmd -> dir -> deps -> sources -> env consistently
      • similar to how I ordered the GH Actions keywords too in ci: better naming for various steps/jobs #11752

• replace all installation and usage of kit with task

  • following the installation guide: https://taskfile.dev/installation/
    • use npm for the Makefile install as it's compatible on different OSes and matches some of our existing tools (like mdspell, markdownlint, etc)
      • NOTE: this does still need adding to the Nix dir
  • replace kit in Makefile, pre-build.sh, and ci-build.yaml with task

Verification

This is a draft, I actually haven't tested it yet

[agilgur5]

I don't think any of these mutexes are actually strictly necessary, two downloads and two builds can run in parallel, they just may bottleneck (or they may not, I don't believe they do on my laptop), so it might be a concurrency optimization to run them serially instead.

Task does have the run key to ensure something runs only once. And it does have a serial syntax, but it's not as flexible as a string for a mutex lock (not entirely the same semantics as a mutex does not specify ordering)

[agilgur5]

the schema is a pretty nice addition too 😉

it's got integrations with VSCode and other editors as well if that's your preference

[agilgur5]

So far task also seems much more efficient than kit. It's running pretty silently on my laptop in the background whereas previously make start UI=true would make all my fans turn on quite loudly.

Screenshot from Docker Desktop (where you can also see how much resources I allocated):

[agilgur5]

All of the E2E tests seem to be failing on port issues (like the ones I had with devcontainer CLI above). CI doesn't run in a devcontainer though, so those should be a bit different

[terrytangyuan]

cc @isubasinghe @alexec

[isubasinghe]

Needs updating in the Nix file, happy to do that myself if you want?
Technically the Nix setup doesn't need task though because it already has something similar.

[agilgur5]

@isubasinghe could you elaborate on this? You mentioned devenv before, but I didn't know that had task runner support (again, my Nix knowledge is pretty dated)?

We discussed it at the Contributor meeting too if it could be used as a replacement on all environments, but no one knew the answer so we were looking to you for more info.
There was also this warning in the Nix docs that we weren't sure exactly what it applied to

[agilgur5]

Checked it myself and it looks like devenv up is a replacement for goreman, but neither are full task runners, they just run processes concurrently (no deps, sources, env, etc).

task and kit are make replacements rather than goreman replacements.

[isubasinghe]

Yeah it does have a task runner, infact Nix can replace the Makefile and task/goreman/etc.
https://github.com/argoproj/argo-workflows/blob/master/dev/nix/flake.nix#L393-L403 is how the processes to run are declared.

There is a manual step of setting up the k8s cluster needed at the moment but even this can be fully integrated into Nix.

That controller should have a version attached to it, I did not set this version. So production builds using Nix are not really possible yet. This is still very fixable though, more than happy to talk about it if this is something we want to do.

[agilgur5]

Yeah it does have a task runner, infact Nix can replace the Makefile and task/goreman/etc.
https://github.com/argoproj/argo-workflows/blob/master/dev/nix/flake.nix#L393-L403 is how the processes to run are declared.

Yea I've seen that -- that's equivalent to goreman but that doesn't have deps, sources, env etc -- features of task, kit, and make. Unless that feature exists somewhere too and just isn't being used?

[isubasinghe]

Generally looks good to me, not approving or disapproving yet, will wait till tests pass.

[terrytangyuan]

Per our discussions at today's contributors meeting, one better option is to revert back to use goreman which is widely adopted and used by Argo CD.

[isubasinghe]

Per our discussions at today's contributors meeting, one better option is to revert back to use goreman which is widely adopted and used by Argo CD.

@terrytangyuan
This might be the easiest way to get rid of kit, a careful revert should do the trick.

There will almost certainly be conflicts.

[agilgur5]

Per our discussions at today's contributors meeting, one better option is to revert back to use goreman which is widely adopted and used by Argo CD.

I don't think there was consensus on "better", just that Workflows, CD, and Akuity already use(d) it, so it would be more familiar within Argoproj.

We had consensus on removing kit, but there were a few options on the table, such as task, goreman, Nix devenv (see above), and just sticking to make (which can also run concurrent tasks and already has task dependencies etc).

@terrytangyuan This might be the easiest way to get rid of kit, a careful revert should do the trick.

There will almost certainly be conflicts.

If that's what we want to do, I can tackle that

[agilgur5]

this was removed in the goreman -> kit migration (this line), added it back now since task does not handle port forwarding

[agilgur5]

this fixed the Controller E2E tests. some Server tests are failing because apparently the Server didn't start at all. apparently task was waiting for deps to finish (which totally makes sense), but starting a servers will never finish. so the cmds need to be backgrounded properly. then everything works ok.

I think kit's ports keyword (mentioned above) was just watching for ports to be opened before proceeding to the next task. nifty feature for servers.

[agilgur5]

I have good news and bad news.

Good news:

• I have figured out how to get the tests to pass (needs some more modifications though)

Bad news:

• task is not a replacement for goreman. goreman is a port of foreman, which addresses the whole foreground/background process issue.
  • task has rejected this feature in the past unfortunately: Run Tasks in Background go-task/task#762
  • task is a replacement for make
    • make was still used in tandem with goreman per build: Use goreman for starting locally. #3074 and Add a "start foreground" option to Makefile #2965, bc make does not solve the foreground/background process issue
      • i.e. it is out-of-scope for both make and task
• kit is a replacement for both make and goreman
• notably, goreman is not a replacement for kit, task, or make. it does one piece of what kit does and does not handle any of the rest.

[alexec]

Let’s correct for the record:

• Kit is maintained, but it’s basically feature complete. A lack of activity is a good thing, look at ‘jq’.
• That said, the last release was 3 weeks ago.
• Kit has no open bugs.
• Kit is probably not the cause of the fan issue you mention. I’ve heard no similar reports from other users.

@juliev0 do you need to open a bug?

Why does it need to be changed? I don’t see any discussion of the problem being solved here.

That’s said, simpler toolchains are cheaper to maintain. If Nix can do the job , can we try that?

[juliev0]

Let’s correct for the record:

• Kit is maintained, but it’s basically feature complete. A lack of activity is a good thing, look at ‘jq’.
• That said, the last release was 3 weeks ago.
• Kit has no open bugs.
• Kit is probably not the cause of the fan issue you mention. I’ve heard no similar reports from other users.

@juliev0 do you need to open a bug?

Why does it need to be changed? I don’t see any discussion of the problem being solved here.

That’s said, simpler toolchains are cheaper to maintain. If Nix can do the job , can we try that?

Sure, I’ll open a bug. Thanks

Update: added here

[juliev0]

@alexec I'm not sure the trade offs of the various tools (need to understand what @agilgur5 lists above). I think one issue if we end up keeping it, is that it needs more documentation on how users should use it in the "running locally" page. I think people are generally confused about where to find their log files, they don't know anything about the tasks.yaml file, etc. Also, need to make sure all of the core functionality works. Apologies for not writing the bug as an issue earlier.

[alexec]

It looks like we've had some tool creep in workflows (CD may have had the same) and this has resulted in too many tools.

Kit prints logs to the console if you run CI=1 kit up, that might make it easier to debug by default.

[agilgur5]

• Kit is maintained, but it’s basically feature complete. A lack of activity is a good thing, look at ‘jq’.

To be fair, I didn't say it wasn't. I merely compared its activity to task (which is no doubt more active). Discussed more in kitproj/kit#36 (comment).
Also I'm not sure that jq is a fair comparison, given that it's much smaller in scope. kit's README also compares to jq as a single binary, but jq is ~270kb while kit is ~8mb. But let's keep that in the other issue.

Kit is probably not the cause of the fan issue you mention. I’ve heard no similar reports from other users.

All I know is that when I swapped it out, my fans made a lot less noise and my memory usage was considerably lower 😅 .
I did not trace it carefully, but it seemed like there might be a memory leak in the auto-restart feature or something as I noticed the memory just kept increasing when I made changes to the code and not going back down. It could be a memory leak in one of the tasks themselves too though; again, I did not trace it, that was merely an observation and not one of the initial motivators.

Why does it need to be changed? I don’t see any discussion of the problem being solved here.

I listed out related discussions in issues and Slack in my opening comment, but Julie has succinctly summarized them above; a handful of bugs and a lack of familiarity and documentation with kit.

That’s said, simpler toolchains are cheaper to maintain. If Nix can do the job , can we try that?

There's a separate discussion on Nix above. It can fulfill what goreman did, but it does not do all that kit does.

@alexec while you are here, would you mind elaborating on the reasons for using kit inside of Workflows? It replaced goreman, but kit is more generally a make replacement (with process management as an additional feature), yet make was not fully replaced and still makes up the bulk of the scripting. If make were wholly replaced, I'd understand that (I am certainly not the biggest make fan), but as only a partial replacement I was a bit confused. goreman is similarly a very tiny-scoped tool that does pretty much one thing

[alexec]

All I know is that when I swapped it out,

I've raised a bug. but more diagnostic info would be useful. Your screen shot indicates that a container is using a lot of memory, but does not say what that container is? Do you have a ps output that shows that it is kit?

handful of bugs

I count 2 bugs?

lack of familiarity

This is a reasonable argument. This would make tools like Nix and Goreman more attractive.

kit is more generally a make replacement for Make

Kit is not a replacement for me Make. It is a replacement for foreman.

[agilgur5]

I've raised a bug. but more diagnostic info would be useful. Your screen shot indicates that a container is using a lot of memory, but does not say what that container is? Do you have a ps output that shows that it is kit?

It's the Workflows devcontainer. I can put more details in the relevant bug, but as I mentioned before, it was just an observation, so I hadn't looked into it in detail to check for a root cause.

I count 2 bugs?

That is enough for a handful, to be fair 😝.
There are some others like Codespaces running indefinitely on kit pre-up: #11423. I even got a notification from GH that it used up 24 hours of compute before I even had access to a terminal (it apparently ran all night even after I closed the tab and turned off my computer).
I just looked into that yesterday and think it can be solved by switching the devcontainer's onCreateCommand to a postCreateCommand, but something weird is going on with how Codespaces is interacting with kit specifically at build-time. I saw it restart a few processes repeatedly too, not sure why.
I wouldn't be surprised if there were other bugs too.

EDIT: filed kitproj/kit#44 regarding Codespaces. I literally only tried it myself yesterday

This is a reasonable argument. This would make tools like Nix and Goreman more attractive.

That's the primary reason, to be fair 👍
It just pops up occasionally from a bug or contributor issue. I wrote this PR as I was literally just comparing task's syntax and wanted to see how easy it would be to transition. It was pretty easy, though task doesn't do process management like foreman / goreman unfortunately 😕

Kit is not a replacement for me Make. It is a replacement for foreman.

🤔 It does a lot more than foreman though? As it's a general task runner (like task and make) rather than a simple process manager

[agilgur5]

🤔 It does a lot more than foreman though? As it's a general task runner (like task and make) rather than a simple process manager

talked a bit with Alex offline over CNCF Slack. we didn't get into the full meat of this yet but I think Alex specifically meant make as a "build tool" rather make as a "task runner".
but we mostly use it as a "task runner" in Workflows and that is the primary comparison to task as well (and task compares itself to make very explicitly).
(Earthly is more of a "build tool", for comparison)
Nix is more of a "dependency management tool", so also a bit orthogonal 😅

basically a lot of tools with partial overlap

[terrytangyuan]

Should we live with it if kit already has all the features we need and there are no known stressing bugs?

[terrytangyuan]

Should we close this?

[agilgur5]

Thanks for following up on this Terry.

and there are no known stressing bugs?

I do still have a few months-old open issues on Kit, so it is not quite all clear skies. In particular, kitproj/kit#48 is one I hit into with some frequency (and can take a few minutes to realize). The workaround is simple (quit kit and run yarn install outside of kit, then restart kit), but not a great experience and could be especially confusing for new contributors.

Since task still does not handle background processes and contributors preferred kit's DX to goreman's DX, I was thinking what we should do is reduce kit to a background process runner and move the rest of the tasks out to limit the scope and bugs like the above.

I had initially thought to do a partial move to task to complete that, but at that point I was wondering if we should just entirely replace Make with another tool (task or otherwise). I thought that some contributors may not want to move away from Make, especially since we've been using it for so long and have familiarity with the current Makefile, so I was looking to do the partial replace of kit by moving foreground tasks into Make. #12583 is one effort toward that -- I still need to fix some race conditions with that, but the other potential con is that there's no fancy log handling for parallel tasks in Make.

I think reducing the scope of kit to background processes would be good, but the question of what tool to move to is still up in the air. Does anyone have thoughts on using Make or replacing it?

Should we close this?

Otherwise I think we can close this since task is not a full kit replacement and goreman is not desired to replace the other portion of kit. Especially if we don't want to replace Make with task, then I think there's no reason to keep this open, other than some discussion of the above.