Skip to content

add schema registry support for protobuf#10775

Closed
bananaaggle wants to merge 17 commits intoapache:masterfrom
bananaaggle:add_schema_registry_support_for_protobuf
Closed

add schema registry support for protobuf#10775
bananaaggle wants to merge 17 commits intoapache:masterfrom
bananaaggle:add_schema_registry_support_for_protobuf

Conversation

@bananaaggle
Copy link
Contributor

@bananaaggle bananaaggle commented Jan 18, 2021

Confluent's Schema Registry supports for deserializing protobuf now. So I add this feature to druid-protobuf-extension module. This module only work when user provide a .desc file in old version. Now it can get schema from schema registry dynamically.
This submission adds the io.confluent:kafka-protobuf-serializer:6.0.1 dependency to the druid-protobuf-extension module.
Old spec is:

"parser": {
      "type": "protobuf",
      "descriptor": "file:///tmp/metrics.desc",
      "protoMessageType": "Metrics"
      ......
}

New spec is:

"parser": {
      "type": "protobuf",
      "protoBytesDecoder": {
           "type": "schema_registry",
          "url": "http://XXXXXX:8081",
          "capacity": 10,
        }
      ......
}

and

"parser": {
      "type": "protobuf",
      "protoBytesDecoder": {
          "type": "file",
          "descriptor": "file:///tmp/metrics.desc",
          "protoMessageType": "Metrics"
        }
      ......
}

Description

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Key changed/added classes in this PR
  • MyFoo
  • OurBar
  • TheirBaz

yuanyi and others added 8 commits January 18, 2021 14:12
* Retain order of AND, OR filter children.

If we retain the order, it enables short-circuiting. People can put a
more selective filter earlier in the list and lower the chance that
later filters will need to be evaluated.

Short-circuiting was working before apache#9608, which switched to unordered
sets to solve a different problem. This patch tries to solve that
problem a different way.

This patch moves filter simplification logic from "optimize" to
"toFilter", because that allows the code to be shared with Filters.and
and Filters.or. The simplification has become more complicated and so
it's useful to share it.

This patch also removes code from CalciteCnfHelper that is no longer
necessary because Filters.and and Filters.or are now doing the work.

* Fixes for inspections.

* Fix tests.

* Back to a Set.
…tialize before starting up, even though set awaitInitializationOnStart true (apache#10779)

* enhance the logic of Start up DruidSchema immediately if there are no segments.

* add UT to test DruidSchema init

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
@bananaaggle
Copy link
Contributor Author

bananaaggle commented Jan 25, 2021

MVN package failed caused by lack of io.confluent:kafka-protobuf-serializer:6.0.1. Is Druid CI use a independency MVN repo instead of public MVN repo? I see same problem in this pull request:#3529. Can I add a new dependency for this extension? And is there someone can help me?

himadrisingh and others added 9 commits January 25, 2021 18:44
* AWS Web Identity Support

required for AWS IRSA

* Update kinesis-ingestion.md

* disabling coverage tests

apache#10541 (comment)

* exclude coverage

* Update licenses.yaml
* Bump http-proxy from 1.18.0 to 1.18.1 in /web-console (#7)

Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](http-party/node-http-proxy@1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump elliptic from 6.5.1 to 6.5.3 in /web-console (#6)

Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.1 to 6.5.3.
- [Release notes](https://github.com/indutny/elliptic/releases)
- [Commits](indutny/elliptic@v6.5.1...v6.5.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dot-prop from 4.2.0 to 4.2.1 in /web-console (#5)

Bumps [dot-prop](https://github.com/sindresorhus/dot-prop) from 4.2.0 to 4.2.1.
- [Release notes](https://github.com/sindresorhus/dot-prop/releases)
- [Commits](sindresorhus/dot-prop@v4.2.0...v4.2.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump bl from 1.2.2 to 1.2.3 in /website (#4)

Bumps [bl](https://github.com/rvagg/bl) from 1.2.2 to 1.2.3.
- [Release notes](https://github.com/rvagg/bl/releases)
- [Commits](rvagg/bl@v1.2.2...v1.2.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump prismjs from 1.20.0 to 1.23.0 in /website (#3)

Bumps [prismjs](https://github.com/PrismJS/prism) from 1.20.0 to 1.23.0.
- [Release notes](https://github.com/PrismJS/prism/releases)
- [Changelog](https://github.com/PrismJS/prism/blob/master/CHANGELOG.md)
- [Commits](PrismJS/prism@v1.20.0...v1.23.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Run integration tests in a second stage

* maybe run the integration tests

* better

* drop sudo
…k/Hydrant Object overhead (apache#10740)

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* fix checkstyle

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* fix test

* fix test

* add log

* Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

* address comments

* fix checkstyle

* fix checkstyle

* add config to skip overhead memory calculation

* add test for the skipBytesInMemoryOverheadCheck config

* add docs

* fix checkstyle

* fix checkstyle

* fix spelling

* address comments

* fix travis

* address comments
@bananaaggle bananaaggle deleted the add_schema_registry_support_for_protobuf branch January 28, 2021 02:26
@bananaaggle bananaaggle restored the add_schema_registry_support_for_protobuf branch January 28, 2021 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants