Prototype review PR #1
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- reads schemas from a base_uri and schema_uri in event - validation (and defaults augmentation) with ajv - produces to Kafka
- EventBus takes validate and produce functions, no need for it to know how to deal with WMF specific configurations or schema repos. - Events are produced to error topic. TODO: handle kafka errors, more tests
…est.yaml This allows us to fully mock the Eventbus instance used when testing, while still piping events through the same functions used for prod by creating them in eventbus-init-utils.js
to be compatible with node 6
Pchelolo
reviewed
Oct 22, 2018
Pchelolo
reviewed
Oct 22, 2018
This adds some extra default behavior if events don't have stream or topic names in them. This makes the explanation and usage of a default eventbus install simpler; The 'only' extra information clients need to include in their events is the schema URI. If stream_field is not configued, Extracted schema URIs will be sanitized and used as the Kafka topic.
Pchelolo
reviewed
Nov 14, 2018
This allows for users of Eventbus (like the service template route) to pass in extra context for validate() and produce() implementaitons to use. This is used by wikimedia-eventbus to instantiate two different Kafka producers, a guarunteed and a 'hasty' producer, to use for both reliable and fire and forget ?hasty=true cases.
default-eventbus changes: - remove topic_prefix support - remove partition_field support - remove key_field support - remove HastyKafkaProducer support - remove id_field support wikimedia-eventbus changes: - remove schema_uri -> stream name default mapping; all events must have stream_name - topic_prefix, id_field and HastyKafkaProducer support moved into wikimedia-eventbus
This removes the restriction that schema URIs match exactly $id fields.
This allows us to use the title rather to ensure that an event of a particular schema is allowed in the stream, rather than relying on the schema_uri. The title will be more consistent and canonical than schema_uris, which might change due to location moving, etc. Also refactory wikimedia-eventbus a bit to simplify.
we can consider implementing this if we have a use case
The default Ajv instance knows about the Draft 7 JSONSchema meta schema, but does not on its own include meta schemas for other JSONSchema drafts. Schemas should point to their meta schema in the $schema field. JSONSChema meta schemas are self validating, so they have their $id fields and $schema fields set to the same URL. If Ajv does not yet have a meta schema referneced by that URL, this will cause infinite recursion as our loadSchema implementation will look up remote schemas by $id. This patch adds a option for EventValidator to match $id schema URLs as being meta schemas, instead of data (event) schemas. If a schema's URL matches, it will be added as a meta schema to Ajv, breaking the infinite recursion. --- This patch also includes a change to the default-eventgate schema_uri_field option. If this is given as an array, the event will be searched for the field names in the array when extracting a schema uri. Both of thse changes allow wikimedia to support draft 4 events with schema URIs at meta.schema_uri, as well as new draft 7 ones we will make with schema URIs at $schema.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.