-
Notifications
You must be signed in to change notification settings - Fork 956
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for schema prefixes. #123
Comments
I'm having this issue to. Did you find a workaround? |
I'm also having this issue. It would be great to give possibility to provide target namespace... |
Any workaround to fix the issue? is it the right approach to add the namespace to the generated schema and re-register? |
@sarathkumar85 would be great if you could some more details on how you are using the Schema generated by Connect. I would assume it's a downstream application(possibly streams) where you are trying to generate the SpecificRecord class based on the schema registered by connect? |
You could use a SMT to add namespace to the connect schema. Below is a sample SMT that helps add the namespace https://gist.github.com/vikas-tikoo-zefr/152100accce437adb258e5c6af031866 |
I have this issue. I have a table named "merchant". The schema kafka-jbdc-connect generateed is: |
@terry19850829 The comment above has the solution you can implement. |
I have seen the solution. Thanks. |
@jcustenborder @mageshn What is the process you used to get the SMT from @vikas-tikoo-zefr into your kafkaconnect definition? I am running into the same issue, and am unsure if this is still the best SMT to use as @terry19850829 and if so how the jar is created and added to the kafkaconnect classpath to solve the namespace issue. Thanks in advance. |
@kalbmj the SMT should still work. To use, add the following configs to your connector definition -
As far as deployment is concerned, you'll need to make a jar off it, and copy it onto your plugins path directory. |
Thanks for the response @vikas-tikoo-zefr . I am new to custom SMTs, so sorry for all of the questions: 1. what is needed alongside the java file in order to create the jar? or should this be brought into an unbuilt kafka-connect-jdbc jar and built there? 2. is there an example you know of anywhere for building the SMT jars (or pom examples)? Thanks. |
This can be a jar on its own. For dependencies |
Thanks @vikas-tikoo-zefr - I've used the base of this project https://github.com/cvasilak/kafka-connect-transforms to change the class/package to be specific to org.example.kafka.connect.transforms/Namespacefy, and then added the built jar to my classpath, but am still seeing the following on validation {"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nInvalid value org.example.kafka.connect.transforms.Namespacefy for configuration transforms.AddNamespace.type: Class org.example.kafka.connect.transforms.Namespacefy could not be found.\nInvalid value null for configuration transforms.AddNamespace.type: Not a Transformation\nYou can also find the above list of errors at the endpoint Config has the following: Thanks in advance for any insight. |
Looks like we're up and running with the suggestions from above @vikas-tikoo-zefr . Thanks again. |
I run into this issue as well (and was able to use the workaround, once I found this issue). It would be great if this basic use case can be supported by default in JDBC Source Connector. |
Thanks @vikas-tikoo-zefr , you saved me as well! :) |
@vikas-tikoo-zefr what is the location where the jar should be placed? Is it CONNECT_PLUGIN_PATH? |
@aoboturov yup. Thats been our setup as well. |
Closing this since the workaround has solved the problem for multiple people. |
Note : The real workaround here is to use the native SetSchemaMetadata SMT. The generated Avro Schema will use "your.package.name" as the namespace, and "ClassName" as the name. Example :
|
@zthulj I'm importing hundreds of tables. Isn't there a way to add a common namespace without specifying the tablename/classname? With the JDBC source connector that should be a pretty common case. |
I agree with @helpermethod in which there should be an out of the box way to just insert the namespace. |
@helpermethod @codependent The work around mention in this thread solves this, although it took me some time to get it right. I packaged the custom plugin mention here in a fat jar and added it to a kafka connect plugin directory, and then used the transform mention here with a minor modification:
|
Supported natively as @grzi mentioned. |
As @helpermethod has mentioned, the workaround only works when importing a single table with a single connector, it is not a viable workaround when importing multiple tables with a single connector, and creating hundreds of connectors is not practical. |
The schemas that are currently generated are based on the table name. When using the AvroConverter from the Confluent Schema Registry, the resulting avro schema does not have a namespace. This causes the avro-maven-plugin to attempt to place the generated source in default java package. The following exception is raised.
The text was updated successfully, but these errors were encountered: