[Feature] Better processors #1378
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <haris@meroxa.io>
Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com> --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com>
* refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <haris@meroxa.io>
Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com> --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com>
…uit into feature/better-processors
hariso
approved these changes
Feb 12, 2024
* update processor-sdk, use config parameter from conduit-commons * add unified diff lib * add utilities for running processor examples and collecting specifications * update diff readme * fix markdown linter error * fix liter errors * update processor-sdk * fix race condition in test
* display raw data in processor examples as string * ensure examples are formatted the same way in processors.json
* add processors field.set & field.subset.exclude * delete extra test * fix tests * add field.rename processor * add exclusion validation to field.rename processor * refactor + fix error return in Process method * add field.convert processor * add filter processor * only structured data * address reviews part1 * use paramgen, update Configure method, update tests * evaluate value for processor setField * use a slice of referenceResolvers for processors with multiple fields params * do not allow .Position to be set in setField processor * update setField test, add non existent field test * use the new Rename() method from referenceResolver * address reviews * address reviews2 * fix renameField bug + add New() method to each processor * linter fix * add processor examples * processor json * Update pkg/plugin/processor/builtin/convertField.go Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * address reviews * setFeild example * setFeild example * make generate * fix paramgen regex * generate * fix regex * add paramgen to tools --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
* updated docs, regenerate * config * first pass * downgrade lib * processing * example * fix config parsing * regenerate * fix config * regenerate, tests * linter, move to opencdc * add test certs * config refactor, tests * more tests * linter * Regenerate * fix linter * git diff * git diff * regenerate * avro examples * run schema registry in examples on port 54321 * fix linter error * rename processor * fix schema registry integration test * don't run avro examples in integration tests --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
* add json.decode processor * make generate * processors * address reviews * make generate * add support for decoding JSON types other than map * make generate * address review comments --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
* update proto API definitions * fix proto linter warnings * deprecate ListPlugins * implement plugin list endpoint * processor plugin orchestrator * move plugin endpoints under connector and processor service
lovromazgon
changed the title
Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * avro.decode example, remove base64 decoding --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
* encode * base64 decode * organize imports
* encode json processor * delete forgotten test * address reviews * linter fix * add processor examples * don't encode nil, output string --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
* refactor builtin processors, generate specs into separate files * move processors into folder impl * go mod tidy * add example summaries * add example summaries * fix name of avro.decode processor * add example summaries * regenerate specs * update default registry * update default registry: fix tests, linter, regenerate * dep upgrades * refactorings * move json encode processor to impl/json * rename field.subset.exclude to field.exclude * rename constructor for unwrap.debezium processor * regenerate specs * dep downgrades --------- Co-authored-by: Haris Osmanagic <haris@meroxa.io>
* init * simplify * fix evaluation * add support for batches in runnable processor * batch test in runnable processor * optimize case when all records are passed through in a processor * make sure remaining records are copied over * fix len check * make sure processor doesn't return more records --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
…uit into feature/better-processors
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR refactors the internal structure and creates types needed to work with standalone and builtin processors (registries, service), it ties everything into the existing code (processor service, pipeline nodes, orchestrator etc.).
This PR includes work from multiple sub-PRs, check them out if you are interested in the discussions we had:
Builtin processor PRs:
Closes #1285, closes #1286, closes #1287, closes #1289, closes #1290, closes #1292
Quick checks: