Allow lily to extract specific models as a subset of a task.

[kuchaguangjie]

[kuchaguangjie]

Another option maybe to provide a flexible task, which may include any table you want. Or, maybe that doesn't fit into how tipset is handled currently, I'm not sure.

[kuchaguangjie]

Regarding the concern of schema migration, mentioned here: https://filecoinproject.slack.com/archives/C0174P5M11T/p1635261449040800?thread_ts=1635213514.039000&cid=C0174P5M11T

If we keep all the tables, but only left the ones not used empty (aka. don't insert data into it with lily job), will this problem remain?

[kuchaguangjie]

Regarding the missing data, we can use csv or lily walk, if we find we need include another table, and its history. Though that feels more complexity than getting all the data.

[frrist]

Hey @kuchaguangjie, yes this is doable -- and apologies for such a late response, this is a great idea. It will require a non-trivial amount of refracting in order to land. We will work to add this feature to a coming release.

How would you like this to behave from the cli? Would passing the models for extraction instead of the task names suffice? Example:

$ lily walk --tasks=miner_locked_funds,miner_sector_events,miner_sector_deals,miner_infos --from=100 --to=200

[kuchaguangjie]

@frrist Thanks, about the cli command, as long as it can specify in table level it's fine I guess.
You may split task into smaller tasks, or add another option to include / exclude table(s) as mentioned in initial post, or you may have even better ideas, while still keep the refactoring relatively easier.

[placer14]

@kuchaguangjie #871 🚀 ✨

[frrist]

via #869