Roadmap v0.2 - Background

[huachaohuang]

Tasks:

• Design document
• Background traits
• A local background implementation
• A remote background implementation
  • Background client
  • Background service
• Integrate the background service with microunit

[huachaohuang]

Background is still a dark corner for v0.2. I will try to solve it after we get other things done.

[tisonkun]

@huachaohuang we should first decide what's delivered in v0.2. Basically, I can imagine that this module is used for compaction. But full-featured compaction cannot be finished in such an early version IMO.

@sticnarf & @you06 may be interested in manipulating background tasks according to their experiences from yatp. But the functionality/requirement is still unclear.

[huachaohuang]

We only need to implement a simple compaction strategy in the hash engine for v0.2. But the hard part is not about compaction. It's about the abstraction to run background jobs. Depending on the environment, a background job may mean a user function, an external command, or a docker image. So it is not very clear what abstraction Kernel should provide to Engine to cover all these use cases.

[huachaohuang]

I think I am wrong about Background. It is not a part of Kernel, it is a part of Engine. Background is also a user of Kernel. For example, a compaction job running in Background needs to access Kernel to update metadata.

[huachaohuang]

Every engine defines its own set of background jobs. Background simply helps to run those jobs.

[tisonkun]

@huachaohuang we may review the definition of Kernel now or later. It seems that so far the kernel goes into a situation where the previous manifest stands.

[huachaohuang]

I am also considering renaming Background to Jobpool or Taskpool or something, Background doesn't seem like a good name.
@tisonkun I think the Kernel definition is fine so far, it handles all the stateful operations. Specifically, Kernel manages the underlying Journal, Storage, and Manifest. Although some code doesn't play nicely for now. But I will fix that too.

[huachaohuang]

I am considering dropping this in v0.2 too. Engual 0.2 simply showcases how an engine can be used. Users don't need to know about these background jobs anyway.

[tisonkun]

Agree.