Skip to content
This repository has been archived by the owner on Feb 21, 2024. It is now read-only.

DSL Compiler Redesign #14

Closed
walterddr opened this issue Jun 10, 2019 · 0 comments
Closed

DSL Compiler Redesign #14

walterddr opened this issue Jun 10, 2019 · 0 comments

Comments

@walterddr
Copy link
Owner

walterddr commented Jun 10, 2019

See:
https://docs.google.com/document/d/1B0O9FbfwC_XDjoB2mL8Dtmv8VeRxhkGX7090dZGbhJ4/edit#

New compiler needs to be flexible enough to handle different DSLs, but should be clean in terms of how to compiler vertex and job graph

  1. vertex should be compiled via incoming edge and component definition only (Rule-base Compilation Pattern #17)
  2. any additional transformation needed is done via downstream vertex (e.g. keyby, repartition, etc)
  3. support reference -> this can either be done in DSL redesign #13 or here.
    [BONUS]. components stored intermediately during compilation in each vertex (e.g. the DataStream coming all the way from source until current) is stored in a more generic manner

Optionally,
Code-Restructure: create a flux-level compilation graph generator in flux-core and remove it from the flink-flux compilation core level (Under discussion whether it should fit in #14 or here)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant