Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pulsar function Should support Generic output Message #9892

Closed
pointearth opened this issue Mar 12, 2021 · 3 comments
Closed

Pulsar function Should support Generic output Message #9892

pointearth opened this issue Mar 12, 2021 · 3 comments
Labels
lifecycle/stale type/feature The PR added a new feature or issue requested a new feature

Comments

@pointearth
Copy link
Contributor

pointearth commented Mar 12, 2021

Is your feature request related to a problem? Please describe.
In the pulsar function, I don't want to define a particular class as O in
public interface Function<I, O> { O process(I var1, Context var2) throws Exception; }

because I have a POJO class, but I always want to push a wrap class to wrap the POJO class as output. because the only different thing between this wrap class and the POJO class is wrap class always adds a JSON field to any of POJO class.

We can prodcue a Generic message with pulsar.client.api:
RecordSchemaBuilder recordSchemaBuilder = SchemaBuilder.record(USER_CONST.SCHEMA_NAME); recordSchemaBuilder.field(USER_CONST.AGE).type(SchemaType.INT32); recordSchemaBuilder.field(USER_CONST.NAME).type(SchemaType.STRING); recordSchemaBuilder.field(USER_CONST.JSON_FOR_EVENT).type(SchemaType.STRING); SchemaInfo schemaInfo = recordSchemaBuilder.build(SchemaType.AVRO); GenericSchema userSchema = Schema.generic(schemaInfo); Producer<GenericRecord> producer = pulsarClient.newProducer( userSchema) .topic(TOPIC_CONST.TOPIC_FUN_GENERIC) .create(); User user = new User(); user.age = 27; user.name = "Simon Generic"; Gson gson = new Gson(); String jsonForEvent = gson.toJson(user); GenericRecordBuilder genericRecordBuilder = userSchema.newRecordBuilder(); GenericRecord genericRecord = genericRecordBuilder .set(USER_CONST.AGE, user.age) .set(USER_CONST.NAME,user.name) .set(USER_CONST.JSON_FOR_EVENT, jsonForEvent) .build();

but, I am trying many ways to produce dynamic object into a pulsar, just like:
public class FunWrapGeneric implements Function<String, GenericRecord> { @Override public GenericRecord process(String input, Context context) throws Exception {

but I always got failed. Can we support this?

Describe the solution you'd like

  1. Add GenericRecord support for pulsar function
    public class FunWrapGeneric implements Function<String, GenericRecord>

  2. or Add a generic Class
    public class FunWrapGeneric implements Function<String, RecordWrap<T>> { @Override public RecordWrap<T> process(String input, Context context) throws Exception {
    and interface RecordWrap
    support add or reduce fields from T.

  3. or Add an override of pulsar function want a Generic class as a parameter of pulsar function
    FunWrapGeneric implements Function<String,Schema>

cc @sijie @codelipenghui

@pointearth pointearth added the type/feature The PR added a new feature or issue requested a new feature label Mar 12, 2021
@eolivelli
Copy link
Contributor

This is related to:
#9590
and to
#9481

I really would like to see this feature land soon.
I believe that the work at #9590 could be extended in order to add an integration test about this specific case

@codelipenghui
Copy link
Contributor

The issue had no activity for 30 days, mark with Stale label.

@tisonkun
Copy link
Member

Move feature "request" to the discussion forum.

@apache apache locked and limited conversation to collaborators Dec 11, 2022
@tisonkun tisonkun converted this issue into discussion #18875 Dec 11, 2022

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
lifecycle/stale type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

No branches or pull requests

4 participants