Add backtest and backforward task #1568
Conversation
* add BackForwardTask; * adjust prompt_template.yaml which default config failed to backtest; * run workflow in loop * add update method to prompt_template.py
github-actions
bot
added
the
waiting for triage
| @@ -193,6 +193,13 @@ SummarizeTask_user : |- | |||
| Here is my information: '{{information}}' | |||
| My intention is: {{user_prompt}}. Please provide me with a summary and recommendation based on my intention and the information I have provided. There are some figures which absolute path are: {{figure_path}}, You must display these images in markdown using the appropriate image format. | |||
|
|
|||
| BackForwardTask_system : |- | |||
There was a problem hiding this comment.
Will it be clearer if we use the following format?
BackForwardTask:
system:
user:
qlib/finco/task.py
Outdated
|
|
||
| with R.start(experiment_name="finCo", resume=True): |
There was a problem hiding this comment.
We don't have to use start.
We can just get the recorder and update it.
qlib/finco/task.py
Outdated
| self._context_manager.set_context("model", model) | ||
| self._context_manager.set_context("dataset", dataset) | ||
|
|
||
| return [RecorderTask()] |
There was a problem hiding this comment.
Why not return a CMDTask with qrun XXX.yaml?
qlib/finco/task.py
Outdated
| return [BackTestTask()] | ||
|
|
||
|
|
||
| class BackTestTask(Task): |
There was a problem hiding this comment.
I think backtest is done in qrun in CMDTask.
What we need to do is to analysis it further.
qlib/finco/task.py
Outdated
| @@ -648,3 +702,38 @@ def get_figure_path(self): | |||
| def save_markdown(self, content: str): | |||
| with open(Path(self.workspace).joinpath(self.__DEFAULT_REPORT_NAME), "w") as f: | |||
| f.write(content) | |||
|
|
|||
|
|
|||
| class BackForwardTask(Task): | |||
There was a problem hiding this comment.
When we fine-tune the prompt, we don't have to define it as an instance of Task.
The prompt fine-tuning is supposed to be a learning workflow at an upper level. So, they do not necessarily share the same concepts.
qlib/finco/workflow.py
Outdated
| @@ -15,6 +15,7 @@ class WorkflowContextManager: | |||
|
|
|||
| def __init__(self) -> None: | |||
| self.context = {} | |||
| self.epoch = 0 | |||
There was a problem hiding this comment.
If there is a learning workflow outside, WorkflowContextManager is not necessarily aware of the learning process.
| task_list = res + task_list | ||
|
|
||
| # todo: add early stop condition | ||
| for i in range(10): |
There was a problem hiding this comment.
I don't think the learning process should be coupled with Workflow Manager.
There was a problem hiding this comment.
Added learnManager
* add LearnManager class & use LearnManager to update system prompt; * use qrun to replace recorder for training and backtesting;
qlib/finco/task.py
Outdated
| self._context_manager.set_context("dataset", dataset) | ||
| self._context_manager.set_context("recorder", recorder) | ||
| command = f"qrun {workflow_path}" | ||
| self._output = subprocess.check_output(command, shell=True) |
There was a problem hiding this comment.
Please set cwd to make sure the mlflow results are generated in the workspace.
Otherwise you can use CMDTask
There was a problem hiding this comment.
added cwd to check_output.
qlib/finco/task.py
Outdated
| dataset=self._context_manager.get_context("dataset"), | ||
| recorder=self._context_manager.get_context("recorder"), | ||
| workspace=workspace, | ||
| model=model, |
There was a problem hiding this comment.
I don't think SignalRecord is one of Analysor.
Please refer to SignalAnalyzer.__name__: SignalRecord,
Thus model and dataset is not necessary paramters
There was a problem hiding this comment.
Analyser is independent of Recorder now
qlib/finco/task.py
Outdated
| model=self._context_manager.get_context("model"), | ||
| dataset=self._context_manager.get_context("dataset"), | ||
| recorder=self._context_manager.get_context("recorder"), | ||
| workspace=workspace, |
There was a problem hiding this comment.
Why is workspace necessary if we already passed the recorder?
There was a problem hiding this comment.
renamed workspace to output_dir
* analyser independent of recorder; * rename analyser's workspace attribution; * analyser load variable by recorder.
Fivele-Li
added
enhancement
Description
Motivation and Context
How Has This Been Tested?
pytest qlib/tests/test_all_pipeline.pyunder upper directory ofqlib.Screenshots of Test Results (if appropriate):
Types of changes