Skip to content

Support defining and validating schema on Python client#3391

Merged
merlimat merged 6 commits intoapache:masterfrom
merlimat:python-schema
Jan 30, 2019
Merged

Support defining and validating schema on Python client#3391
merlimat merged 6 commits intoapache:masterfrom
merlimat:python-schema

Conversation

@merlimat
Copy link
Copy Markdown
Contributor

Motivation

Allow applications to declare the schema for messages in python client library.

The schema is used to perform validation against the topic schema and also
for serialization/deserialization

Added support for Bytes, String, JSON and Avro definitions.

The schema of a message can be easily defined as:

import pulsar.schema

class Example(pulsar.schema.Record):
    a = Integer()
    b = String()
    c = Long(default=5)

With that, one can create a producer/consumer. Eg:

producer = client.create_producer(
                    topic='my-topic',
                    schema=pulsar.schema.AvroSchema(Example)

producer.send(Example(a=1, b='hello'))

Type is also validated to ensure a producer will only send data of the same type.

TODOs (in separate PRs):

  • Add tests for schema compatibility across Python and Java
  • Documentation and examples
  • AUTO_CONSUME schema type

@merlimat merlimat added type/feature The PR added a new feature or issue requested a new feature component/python labels Jan 20, 2019
@merlimat merlimat added this to the 2.3.0 milestone Jan 20, 2019
@merlimat merlimat self-assigned this Jan 20, 2019
@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

2 similar comments
@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

@sijie
Copy link
Copy Markdown
Member

sijie commented Jan 24, 2019

run integration tests

@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

3 similar comments
@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

@merlimat
Copy link
Copy Markdown
Contributor Author

run integration tests

@sijie
Copy link
Copy Markdown
Member

sijie commented Jan 28, 2019

run integration tests

@merlimat
Copy link
Copy Markdown
Contributor Author

run java8 tests

@merlimat merlimat merged commit 86a9f43 into apache:master Jan 30, 2019
@merlimat merlimat deleted the python-schema branch February 2, 2019 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/feature The PR added a new feature or issue requested a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants