Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avro Schema Definition flags are not exposed on Pulsar Functions #6765

Closed
psilos opened this issue Apr 18, 2020 · 0 comments · Fixed by #6868
Closed

Avro Schema Definition flags are not exposed on Pulsar Functions #6765

psilos opened this issue Apr 18, 2020 · 0 comments · Fixed by #6868
Labels
type/bug The PR fixed a bug or issue reported a bug
Milestone
2.6.0

Comments

@psilos
Copy link

psilos commented Apr 18, 2020

Describe the bug
A few stories have been added that allow Avro Schema Compatibility on Pulsar to conform with the default Avro behaviour, namely:

Unfortunately these flags are not exposed into Pulsar Function configuration, therefore we cannot AUTO_CONSUME or AUTO_PRODUCE to topics with stricter Avro Schema Validation

Expected behavior

  • The same flags to be exposed under Functions Configuration
  • Upon setting those flags, a function should be able to consume from a Topic with strict(default) Avro Schema Validation
  • Upon setting those flags, a function should be able to produce from a Topic with strict(default) Avro Schema Validation
@psilos psilos added the type/bug The PR fixed a bug or issue reported a bug label Apr 18, 2020
@sijie sijie mentioned this issue Apr 18, 2020
Closed
@codelipenghui codelipenghui added this to the 2.6.0 milestone Apr 19, 2020
@315157973 315157973 mentioned this issue May 4, 2020
Merged
codelipenghui pushed a commit that referenced this issue Jun 9, 2020
@315157973
[issue #6765] Expose definition flags to function (#6868)
2aff473 
1）
Change the value of the CLI tool parameter "--custom-schema-input" from
"TopicName-> schemaType" 
 to
"topicName-> {" schemaType ":" type"," jsr310ConversionEnabled ": true," alwaysAllowNull ": true}"

2）
Modify Function.proto, add properties "jsr310ConversionEnabled", "alwaysAllowNull"。So that we can receive the above 2 parameters

3）
Modify the "FunctionConfigUtils#convert" method ， put the two parameters in "CustomSchemaInputs" into ConsumerSpec

4）
In “JavaInstanceRunnable#setupInput” method, put the above 2 parameters into "ConsumerConfig" and pass it to "PulsarSource", let it set the parameters into "schema" when creating the consumer of Source。
So that，The “function” can get these 2 parameters from the message of 
”currentRecord“
cdbartholomew pushed a commit to kafkaesque-io/pulsar that referenced this issue Jul 24, 2020
@315157973 @cdbartholomew
[issue apache#6765] Expose definition flags to function (apache#6868)
f71f4d9 
1）
Change the value of the CLI tool parameter "--custom-schema-input" from
"TopicName-> schemaType" 
 to
"topicName-> {" schemaType ":" type"," jsr310ConversionEnabled ": true," alwaysAllowNull ": true}"

2）
Modify Function.proto, add properties "jsr310ConversionEnabled", "alwaysAllowNull"。So that we can receive the above 2 parameters

3）
Modify the "FunctionConfigUtils#convert" method ， put the two parameters in "CustomSchemaInputs" into ConsumerSpec

4）
In “JavaInstanceRunnable#setupInput” method, put the above 2 parameters into "ConsumerConfig" and pass it to "PulsarSource", let it set the parameters into "schema" when creating the consumer of Source。
So that，The “function” can get these 2 parameters from the message of 
”currentRecord“
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@315157973 @huangdx0726
[issue apache#6765] Expose definition flags to function (apache#6868)
62b1ddb 
1）
Change the value of the CLI tool parameter "--custom-schema-input" from
"TopicName-> schemaType" 
 to
"topicName-> {" schemaType ":" type"," jsr310ConversionEnabled ": true," alwaysAllowNull ": true}"

2）
Modify Function.proto, add properties "jsr310ConversionEnabled", "alwaysAllowNull"。So that we can receive the above 2 parameters

3）
Modify the "FunctionConfigUtils#convert" method ， put the two parameters in "CustomSchemaInputs" into ConsumerSpec

4）
In “JavaInstanceRunnable#setupInput” method, put the above 2 parameters into "ConsumerConfig" and pass it to "PulsarSource", let it set the parameters into "schema" when creating the consumer of Source。
So that，The “function” can get these 2 parameters from the message of 
”currentRecord“
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
2.6.0
Development

Successfully merging a pull request may close this issue.

[issue #6765] Expose definition flags to function 315157973/pulsar
3 participants
@psilos @jiazhai @codelipenghui