Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New task execution framework #107

merged 29 commits into from Sep 23, 2019


Copy link

Xuanwo commented Aug 20, 2019

This pr will introduce a new task execution framework which could be exported to another project, and serves both qsctl and qscamel.


We do need concurrency.

it's 9102 for now, we have more cores and faster networks. We need to make our best to improve user experience and speed is one of the most importent parts. So we need concurrency here.

We do need tests.

qsctl is the tool with the most users for qingstor object storage. And problem happened will make users angry, even affecting their business and data. So we need to test our tools for every single operation.

We do need happiness.

developers are also human beings(it's obviously before we got AI coding for us). So developer experience is as important as user experience. We need to make fun in developing qsctl. Less copy-and-paste, less interface type assertion by hand, less panic in runtime, less possibility to fix bug on holiday.


  • Concurrency
  • Strong type
  • Fetchable results
  • Orderly execution
  • Free arrangement
  • Easy to test



Type will be used to do two things:

  • Store and transmit data between tasks.
  • Enforce task requirement in compiling time.

Every type will be like following:

type Key struct {
	valid bool
	v     string

func (o *Key) GetKey() string {
	if !o.valid {
		panic("Key value is not valid")
	return o.v

func (o *Key) SetKey(v string) {
	o.v = v
	o.valid = true

type KeyGetter interface {
	GetKey() string

type KeySetter interface {

Firstly, we got a struct with typed value(not interface) and valid marker. Then we have GetXXX and SetXXX functions with typed value as arguements. After these, we can have XXXGetter and XXXSetter to describe abilities that task have.


Every task implements navvy.Task interface which is a simple func(){}. And task struct will be look like this:

// CopyFileTask will copy a file to storage.
type CopyFileTask struct {

	// Runtime value

// Run implement navvy.Task
func (t *CopyFileTask) Run() {

// initCopyFileTask will create a CopyFileTask and fetch inherited data from CopyTask.
func NewCopyFileTask(task types.Todoist) navvy.Task {
	t := &CopyFileTask{
		copyFileTaskRequirement: task.(copyFileTaskRequirement),
	return t

// copyLargeFileTaskRequirement is the requirement for execute CopyLargeFileTask.
type copyLargeFileTaskRequirement interface {

	// Inherited value

// newCopyFileTask will create a new copy file task.
func (t *CopyFileTask) new() {
	// init code here.

Types will be embed, so that we can use t.GetPath() instead of t.Path.GetPath().

Task have two kind of value:

  • inherited_value: inherited_value will be inherited from parent task.
  • runtime_value: runtime_value will be created in self new or run funcs.

And there are two kind of tasks:

  • required: required task do not have runtime value, it only use values provided by parent task via it's requirement, such as FileMD5SumTask
  • dependent: dependent task is a new task with it's self task queue and runtime value, such as CopyFileTask

We need to write run for required task, and new for dependent task. In other opinion, required task will do the real job, dependent task will be a compose of different required tasks.

Todo && Todoist

Todo is a slice of TodoFunc, and Todoist is the interface with following functions:

type Todoist interface {
	NextTODO() TodoFunc

Every task will have a Todo to manager it's job queue and submit self to the pool.



Type will be generated via types.json:

  "BucketName": "string",
  "BytesPool": "*sync.Pool"

Type will be shared across task pacakge.


Task will be generated via tasks.json too:

"CopyFile": {
   "type": "dependent",
   "path": "copy_file",
   "depend": "Copy",
   "description": "copy a file to storage",
   "inherited_value": [
   "runtime_value": [


Let's check whether we achieve our goals.

  1. Every value in task is strong typed
  2. We can get values in tasks.
  3. Task will be executed orderly and free to do task arrangement, see CopyLargeFile as an example.
  4. Every task could be test without depending each other with mockXXXTask which satisfied XXXTaskRequirement


It's obviously that this framework is far from finished, but I think it's ready to be merged and we can bulid qsctl without waiting for the changes of this framework.

We will do following things for this framework:

  • Retry
  • Error handling
  • Split into another project to serve qscamel

That's all, thanks for reviewing.

Xuanwo added 24 commits Aug 15, 2019
action/ls.go Outdated Show resolved Hide resolved
action/bucket.go Outdated Show resolved Hide resolved
Xuanwo added 2 commits Sep 23, 2019
@Xuanwo Xuanwo changed the title Refactor cmd New task execution framework Sep 23, 2019
Xuanwo added 2 commits Sep 23, 2019
@Xuanwo Xuanwo merged commit d162780 into say-goodbye-to-python Sep 23, 2019
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
continuous-integration/travis-ci/push The Travis CI build passed
@Xuanwo Xuanwo deleted the refactor-cmd branch Sep 23, 2019
Xuanwo added a commit that referenced this pull request Oct 25, 2019
* misc: Upgrade navvy

* Refactor relation between task and cmd

* task: Add support for upload local file

* task/types: Add check before get value

* task/copy: Implement copy from stram/file to object

* task/common: Move mulptipart and object operation to common

* task: Split copy into copy_file and copy_stream

* task/utils: Remove not used funcs

* task: Rename FilePath to Path

* task: Generate task definitions

* task: Auto generate inherited value code

* task: Add copy root task

* task: Add ParseTowArgs and SetupStorage

* task: Add test for file task

* task/types: Add more clear info for get invalid value

* task/common: Add md5sum test

* task/common: Add test for multipart

* travis: Use go 1.13 instead

* task/copy_file: Add test for copy file tasks

* task/copy_stream: Add test for copy stream task

* utils: Add stream input support

* action: Remove not used tests

* task/utils: Move PraseInput into task

* task/tools: Format input json

* task: Add requirnment for dependent task

* task/tools: Refactor template

* task/tools: Do not add name in json file

* Fix conflict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
2 participants
You can’t perform that action at this time.