Restore portable schema changes, with simple LogicalType support #9604
Conversation
…Translation (apache#8853)"" This reverts commit dbcb14c.
…anup"" This reverts commit c9da964.
|
Run Dataflow ValidatesRunner |
|
Run JavaPortabilityApi PreCommit |
|
R: @reuvenlax could you take a look at this? It restores my changes to the schema proto, but with a modification to represent all logical types with a "javasdk" urn and serialized java object in the payload (for now). I think the precommit failure is unrelated, Hannah tells me that #9610 should resolve it. I ran the Dataflow VR tests and also merged in #9446 locally to make sure the |
| @@ -66,8 +50,8 @@ | |||
| return builder.build(); | |||
| } | |||
|
|
|||
| private static RunnerApi.Schema.Field toProto(Field field, int fieldId, int position) { | |||
| return RunnerApi.Schema.Field.newBuilder() | |||
| private static SchemaApi.Field toProto(Field field, int fieldId, int position) { | |||
There was a problem hiding this comment.
Rename to fieldToProto (to be consistent with fieldFromProto)?
| RunnerApi.Schema.FieldType.Builder builder = | ||
| RunnerApi.Schema.FieldType.newBuilder() | ||
| .setTypeName(TYPE_NAME_MAPPING.get(fieldType.getTypeName())); | ||
| private static SchemaApi.FieldType toProto(FieldType fieldType) { |
There was a problem hiding this comment.
Similarly, fieldTypeToProto
|
|
||
| private static final String URN_BEAM_LOGICAL_DATETIME = "beam:fieldtype:datetime"; | ||
| private static final String URN_BEAM_LOGICAL_DECIMAL = "beam:fieldtype:decimal"; | ||
| private static final String URN_BEAM_LOGICAL_JAVASDK = "beam:fieldtype:javasdk"; |
There was a problem hiding this comment.
Should we rename this URN to beam:schema:logical_type:javasdk:v1 (similar above)? It sounds clearer to me, and I saw other URNs have version numbers
|
Run JavaPortabilityApi PreCommit |
|
Run Java PreCommit |
|
Run Python PreCommit |
|
Run Dataflow ValidatesRunner |
|
Run SQL PostCommit |
|
Thank you @Hannah-Jiang! looks like syncing past #9610 fixed the precommit issue :) |
|
Run Dataflow Runner Nexmark Tests |
|
LGTM. Will merge when all tests pass. |
|
I'll write a patch for @robinyqiu's naming suggestions as well |
|
Run Dataflow Runner Nexmark Tests |
|
Run Dataflow ValidatesRunner |
|
Run SQL PostCommit |
|
Run Python PreCommit |
|
BEAM-8286 seems to be the cause of the Python PreCommit failure |
|
Whoops fat fingered the "close and comment" |
|
Merged past #9620 which should fix the python precommit |
|
run Dataflow ValidatesRunner |
|
run Flink ValidatesRunner |
|
run SQL PostCommit |
|
run Portable Dataflow ValidatesRunner |
Un-reverts changes to the portable schema representation, with a couple of changes (22f3852):
beam:fieldtype:javasdkURN. Long-term I would like to move types to a registry and discourage this URN (BEAM-7855), but for now it replicates the current implementation.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.