New task execution framework

[Xuanwo]

This pr will introduce a new task execution framework which could be exported to another project, and serves both qsctl and qscamel.

---

Situation

We do need concurrency.

it's 9102 for now, we have more cores and faster networks. We need to make our best to improve user experience and speed is one of the most importent parts. So we need concurrency here.

We do need tests.

qsctl is the tool with the most users for qingstor object storage. And problem happened will make users angry, even affecting their business and data. So we need to test our tools for every single operation.

We do need happiness.

developers are also human beings(it's obviously before we got AI coding for us). So developer experience is as important as user experience. We need to make fun in developing qsctl. Less copy-and-paste, less interface type assertion by hand, less panic in runtime, less possibility to fix bug on holiday.

Goal

• Concurrency
• Strong type
• Fetchable results
• Orderly execution
• Free arrangement
• Easy to test

Concept

Type

Type will be used to do two things:

• Store and transmit data between tasks.
• Enforce task requirement in compiling time.

Every type will be like following:

type Key struct {
	valid bool
	v     string
}

func (o *Key) GetKey() string {
	if !o.valid {
		panic("Key value is not valid")
	}
	return o.v
}

func (o *Key) SetKey(v string) {
	o.v = v
	o.valid = true
}

type KeyGetter interface {
	GetKey() string
}

type KeySetter interface {
	SetKey(string)
}

Firstly, we got a struct with typed value(not interface) and valid marker. Then we have GetXXX and SetXXX functions with typed value as arguements. After these, we can have XXXGetter and XXXSetter to describe abilities that task have.

Task

Every task implements navvy.Task interface which is a simple func(){}. And task struct will be look like this:

// CopyFileTask will copy a file to storage.
type CopyFileTask struct {
	copyFileTaskRequirement

	// Runtime value
	types.Todo
	types.TotalSize
}

// Run implement navvy.Task
func (t *CopyFileTask) Run() {
	utils.SubmitNextTask(t)
}

// initCopyFileTask will create a CopyFileTask and fetch inherited data from CopyTask.
func NewCopyFileTask(task types.Todoist) navvy.Task {
	t := &CopyFileTask{
		copyFileTaskRequirement: task.(copyFileTaskRequirement),
	}
	t.new()
	return t
}

// copyLargeFileTaskRequirement is the requirement for execute CopyLargeFileTask.
type copyLargeFileTaskRequirement interface {
	navvy.Task
	types.PoolGetter

	// Inherited value
	types.KeyGetter
	types.PathGetter
	types.StorageGetter
	types.TotalSizeGetter
}

// newCopyFileTask will create a new copy file task.
func (t *CopyFileTask) new() {
	// init code here.
}

Types will be embed, so that we can use t.GetPath() instead of t.Path.GetPath().

Task have two kind of value:

• inherited_value: inherited_value will be inherited from parent task.
• runtime_value: runtime_value will be created in self new or run funcs.

And there are two kind of tasks:

• required: required task do not have runtime value, it only use values provided by parent task via it's requirement, such as FileMD5SumTask
• dependent: dependent task is a new task with it's self task queue and runtime value, such as CopyFileTask

We need to write run for required task, and new for dependent task. In other opinion, required task will do the real job, dependent task will be a compose of different required tasks.

Todo && Todoist

Todo is a slice of TodoFunc, and Todoist is the interface with following functions:

type Todoist interface {
	AddTODOs(...TodoFunc)
	NextTODO() TodoFunc
}

Every task will have a Todo to manager it's job queue and submit self to the pool.

Implemention

Type

Type will be generated via types.json:

{
  "BucketName": "string",
  "BytesPool": "*sync.Pool"
}

Type will be shared across task pacakge.

Task

Task will be generated via tasks.json too:

"CopyFile": {
   "type": "dependent",
   "path": "copy_file",
   "depend": "Copy",
   "description": "copy a file to storage",
   "inherited_value": [
     "Key",
     "Path",
     "Storage"
   ],
   "runtime_value": [
     "TotalSize"
   ]
}

Review

Let's check whether we achieve our goals.

1. Every value in task is strong typed
2. We can get values in tasks.
3. Task will be executed orderly and free to do task arrangement, see CopyLargeFile as an example.
4. Every task could be test without depending each other with mockXXXTask which satisfied XXXTaskRequirement

Roadmap

It's obviously that this framework is far from finished, but I think it's ready to be merged and we can bulid qsctl without waiting for the changes of this framework.

We will do following things for this framework:

• Retry
• Error handling
• Split into another project to serve qscamel

---

That's all, thanks for reviewing.