Skip to content

[SYSTEMDS-2972] Dependency Task execution for transform encode#1351

Closed
ilovemesomeramen wants to merge 14 commits intoapache:masterfrom
ilovemesomeramen:dag_exec
Closed

[SYSTEMDS-2972] Dependency Task execution for transform encode#1351
ilovemesomeramen wants to merge 14 commits intoapache:masterfrom
ilovemesomeramen:dag_exec

Conversation

@ilovemesomeramen
Copy link
Copy Markdown
Contributor

This PR adds a Framework for dependent Task execution consisting of the DependencyTask and DependencyThreadPool as well as the implementation of build and apply in said Framework.

It is now possible for different encoders to provide tasks which best fit the given data characteristics by overriding the getApplyTasks and getBuildTasks functions. These functions can return a DAG of Tasks which will be integrated into the DAG of the whole Matrix/Frame. Subsequently this DAG is then executed in the DependencyThreadPool. This approach enables the possibility of trying different execution orders without the need to make major changes to the code.

At the moment only the Dummycoders implement a different apply method compared to the default. This will be changed in future PRs.

Currently there are two approaches implemented staged and non staged.
During staged execution the building and applying steps take place one after another. Which means the building will be done for the whole Matrix/Frame and then the apply takes place.
Non staged execution on the other hand constructs one large DAG for building and applying.

@phaniarnab
Copy link
Copy Markdown
Contributor

Thanks for the patch, Lukas. I fixed a few typos and added SuppressWarnings to resolve the warning for unused methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants