[BEAM-115] WIP: JSON Schema definition of pipeline #662
Conversation
| "additionalProperties": { "$ref": "#/definitions/Value" } | ||
| }, | ||
|
|
||
| // A flat list of windowing strategies. These may be referenced repeatedly. |
There was a problem hiding this comment.
Don't you mean a mapping as well?
|
@aaltay ping for Python feedback. Its quite a rough WIP. |
| // A mapping containing all user-definable functions in the pipeline. | ||
| // The keys must be unique to this pipeline definition, but otherwise are | ||
| // arbitrary. | ||
| "userFns": { |
There was a problem hiding this comment.
Are these object ids used anywhere aside from cross referencing things in the pipeline description?
If that is the case, we could use an array and use array index to reference them, It will be a more compact pipeline, and remove some arbitrary ids. This will be less error prone at pipeline generation time (in regards to uniqueness requirement.)
There was a problem hiding this comment.
They are just for cross-referencing. I wanted to leave the mechanism of coming up with the IDs unspecified.
|
Closing for a bit. |
|
Refer to this link for build results (access rights to CI server needed): |
| <dependencies> | ||
| <dependency> | ||
| <groupId>com.github.fge</groupId> | ||
| <artifactId>json-schema-validator</artifactId> |
There was a problem hiding this comment.
Consider Python libraries for use in smoke testing.
|
This is superseded by #1946 which ports the schema to proto and updates it. |
…apache#662) Co-authored-by: Mariatta Wijaya <Mariatta@users.noreply.github.com>
This is a json-schema sketch of the concrete schema from the Pipeline Runner API proposal document. Because our serialization tech discussion seemed to favor JSON on the front end and Proto on the backend, I made this quick port. The original Avro IDL definition is also on a branch with a test.
Notes & Caveats:
"description"fields.Feel free to leave comments here about the schema or meta issues of e.g. where the schema should live and what libraries we might want to use.