Profiling

[elevin72]

It would be a very cool/useful feature if there was a way to profile (both in memory usage, and in time) a task. This includes profiling all children tasks of a given task. What is the relationship between a task that is run, and the underlying process that it calls. Obviously I am just speaking about Linux for now

[trulede]

Depending on what you are doing, the log output (-v flag IIRC) is fairly good for identifying slower tasks. AND you can always instrument the commands/processes with some additional commands.

Golang has some profiling capability, you would be able to use that without too much effort.
https://go.dev/doc/diagnostics

Eventually, most (all) Tasks result in a shell command being run. You can instrument them yourself with additional commands, or perhaps try a system profiling tool which might pickup the kind of detail you want. "top" is quite capable in that regard. "pidstat" might be even more appropriate. Redis is a very good way to aggregate runtime instrumentation (e.g. task start, end and any other output); spin up a redis container, add redis-cli commands to instrument processes/commands, run Task ... then run a script over the collected data to get some profiling results.

[pd93]

This is an interesting idea, but I'm not convinced how in-scope this is for the main task binary. I feel like the vast majority of users would not use this feature. As mentioned above, other external tools are also capable of doing similar things.

We can leave this open to gather votes/comments, but I think it's unlikely to happen any time soon.

[neriaad]

I would find such a feature very useful since in my organization we have some massive tasks and profiling them recursively to their subtasks can be very difficult to do manually.

[trulede]

There might be something in the new log/slog library (https://go.dev/blog/slog). There it is possible to write structured logs (i.e. JSON + custom fields). A few fields from the Task object would be sufficient to build a graph of the task execution. That would be useful in a number of cases.

[seanamos]

I'm less interested in the memory side, but very interested in getting a summary at the end of execution of task timings.

[trulede]

Related #2384 #2453

[trulede]

I will close this as a duplicate of 2384. With advanced logging using slog, and the feedback collected from the many issues on the topic, you should get the coarse profiling for task execution.