deep review and redesign of API for work queue functionality

[pabigot]

As requested by @andrewboie here and rephrasing this comment:

As demonstrated in #23132 although k_delayed_work_cancel() is documented to return -EALREADY if the work item has been completed, it actually returns it if the work item has already been submitted.

I'm skeptical of the whole k_work implementation. I'm seeing at least nine distinct states for a work item that is not cancelled:

• never submitted
• delayed on timeout
• timeout expired, not yet added to queue (in z_clock_announce())
• marked pending but not on queue (by k_submit_to_queue)
• on queue
• removed from queue (by z_work_q_main)
• cleared pending (by z_work_q_main)
• invoking handler (by z_work_q_main)
• complete

Most of these can't be inferred by inspecting the state.

Somebody should take the time to go through the whole k_work API and implementation and revise it so the documentation accurately describes the behavior and the functionality is useful. By "useful" I mean that after an attempt to cancel you should always be able to tell whether the item was:

• submitted but not yet committed to processing (delayed or on queue): can be cancelled, 0
• processing (including committed to processing, not yet in handler): cannot be cancelled, now -EINVAL preferably -EINPROGRESS
• complete (which may be indistinguishable from unsubmitted): too late to cancel, now -EALREADY preferably -EINVAL

This capability should be present whether the item is a standard work item, a delayed work item, or a pollable work item.

[pabigot]

@andrewboie @andyross I plan to start on this later this week. If it's something you'd like done in-house at Intel instead please reply here.