feat(host,provers,tasks): handle local job cancellation

[petarvujovic98]

No description provided.

[smtmfft]

Can you make some tests for the request & cancel process? I think we need to cover more as the v2 will be online soon. to make our future life easier, we need to do it. however, I don't know if there is a good way to test server API, maybe not easy to write simple unit test for it??
anyway, I came up with few cases, and I believe there are more than these, let's try to cover them as more as possible.

1. request->mock crash->request(should return the right error status)
2. request->cancel->request(returns cancel by user or restarts the task??)
3. request repeatly or cancel repeatly

[petarvujovic98]

@smtmfft I agree, we need to have testing for this. This, however is not a simple unit test, rather a end-to-end test where we go through a whole request flow with all the components spun up.

I've already been playing around with this, but I need to add it to CI and polish it further to be good.
I'll do it in a separate PR which will be addressing testing (so it will be a bit bigger).

BTW

2. request->cancel->request(returns cancel by user or restarts the task??)

This restarts the task, but we should have a test enforcing this behavior

[smtmfft]

@smtmfft I agree, we need to have testing for this. This, however is not a simple unit test, rather a end-to-end test where we go through a whole request flow with all the components spun up.

great! I think you can take current ontake upgrade to improve it.