Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python Schema] Support setting namespace for python schema #12175

Merged
merged 4 commits into from
Sep 25, 2021

Conversation

gaoran10
Copy link
Contributor

@gaoran10 gaoran10 commented Sep 24, 2021

Motivation

Currently, the python schema didn't support setting namespace in the schema definition.

This PR is used to support setting namespace in the python schema definition.

I have two ways to set the namespace for Record, one is to add a new class method for Record to set the namespace, one is to add a new field _avro_namespace for class Record. I select the second way.

for example, add a special field _avro_namespace for the custom Record to add the namespace.

class User(Record):
    _avro_namespace = 'xxx.xxx.xxx'
    name = String()
    age = Integer()

The schema definition is like this.

{
  'name': 'User', 'namespace': 'xxx.xxx.xxx', 'type': 'record', 'fields': [
    {'name': 'age', 'type': ['null', 'int']}, 
    {'name': 'name', 'type': ['null', 'string']}
  ]
}

Modifications

Modify the method schema_info of the class Record to add the namespace field when generating the schema definition.

Verifying this change

Add test to verify the schema definition.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

@codelipenghui codelipenghui added this to the 2.9.0 milestone Sep 24, 2021
'type': 'record',
'fields': []
}
if schema['namespace'] is None:
del schema['namespace']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding and then deleting, we could just add it only if it's not None

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

@@ -57,6 +57,8 @@ def _get_fields(cls, dct):

class Record(with_metaclass(RecordMeta, object)):

_namespace = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering given that this is very specific to Avro, if we should name it as _avro_namespace to make it clear.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other thing is that we should add this to the Python schema docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, I'll the name _avro_namespace.

@codelipenghui codelipenghui added the doc-required Your PR changes impact docs and you will update later. label Sep 25, 2021
@merlimat merlimat merged commit f0d8fb0 into apache:master Sep 25, 2021
codelipenghui pushed a commit that referenced this pull request Sep 25, 2021
* support set namespace for python schema

* fix

* fix

* fix comment

(cherry picked from commit f0d8fb0)
@codelipenghui codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Sep 25, 2021
@gaoran10 gaoran10 deleted the gaoran/python-schema-add-namespace branch September 26, 2021 03:04
@Anonymitaet Anonymitaet added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. and removed doc-required Your PR changes impact docs and you will update later. labels Sep 26, 2021
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
…2175)

* support set namespace for python schema

* fix

* fix

* fix comment
momo-jun added a commit to momo-jun/pulsar that referenced this pull request Aug 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-picked/branch-2.8 Archived: 2.8 is end of life doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. release/2.8.2 release/2.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants