Create wrappers for Java based models #137
Comments
Merged
agrski
added a commit
that referenced
this issue
Dec 2, 2022
* Remove Kafka Protobuf Serdes dependency as unused * Add retry library for pipeline subscriptions, etc. * Add retry policy for pipeline subscription attempts * Formatting * Refactor Kafka admin client to instance variable * Make pipeline creation method more idiomatic Avoid mutable state. Improve formatting. Rename variables for clarity and to avoid shadowing. * Rename method for clarity * Refactor Kafka admin-client interactions to new class This avoids leaking Kafka-specific knowledge into what is essentially a gRPC-wrapper class. It also avoids scope creep and increasing file size for the pipeline subscriber class. * Formatting * Check pipeline steps are valid before creating Kafka topics (if not already extant) Interactions with Kafka are out-of-process, hence slower and less reliable. We may also decide there is nothing to do, if the pipeline is not valid. * Refactor Kafka topic creation into a single pipeline of transformations * Await creation of Kafka topics in parallel * Refactor Kafka result waits to private method * Formatting * Refactor Joiner serde definitions to companion object These are stateless - essentially type definitions - so there is no need to redefine them for evey instance. * Use when statement instead of if-else for clarity * Formatting * Replace call to deprecated Kafka API The existing 'of' call has been deprecated. As we already allow a significant join window, this does not add a wall-clock grace period. * Refactor pipeline header filtering to extension method on KStreams * Refactor inference v2 type conversion to extension method for chainer * Replace inference v2 conversion logic in joiner with new extension method * Add CLI options for Kafka partition count & replication factor These will be used when creating topics that do not yet exist. * Rename CLI var for consistency & group CLI vars * Add CLI var for use of clean Kafka state on startup * Add local properties for new Kafka CLI vars * Formatting * Refactor inference v2 (un)marshalling to KStream extension methods * Remove type constraint on KStream key type in extension method * Remove TODO as only intend to support inference v2 internally Internally means within the context of a pipeline. * Remove unused import * Add tail-recursive version of joiner logic * Replace recursive joiner builder with tail rec equivalent * Make bootstrapping param to method nullable This simplifies the API to only provide pending joins when they exist, making the caller's task easier to read and understand. * Rename private method for clarity * Fix bug using wrong var in joiner topology construction * Remove redundant non-null filtering in joiner * Rename data class to disambiguate from upcoming new class * Add CLI var for KStream join window duration * Refactor domain params for KStreams into data class Rather than passing an increasingly large number of configuration options, it is cleaner to pass a single data class which wraps up that set of options/config. * Rename var for clarity & consistency * Refactor Kafka (Streams) type aliases & serdes to separate file These definitions were scattered around and duplicated in a few places. * Formatting & removal of unused imports * Remove done TODOs * Rename method & vars for clarity The method name was noun-like rather than verb-like, which is counterintuitive. The variable names were cryptic. * Formatting for consistency * Fix broken tests by adding Kafka domain params instance * Formatting * Remove stale comment * Formatting + fix incorrect type alias * Remove redundant parameter from source projection subtype * Formatting * Add comparison logic for joiners in transformer tests * Fix joiner cases in transformer tests + rename param for clarity * Rename param for clarity * remake operator to get resource yaml changes Co-authored-by: Clive Cox <cc@seldon.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To cover
The text was updated successfully, but these errors were encountered: