Skip to content

Blueprint v200 py plugins#4521

Merged
hezyin merged 20 commits intoapache:mainfrom
merico-ai:blueprint_v200_py_plugins
Feb 28, 2023
Merged

Blueprint v200 py plugins#4521
hezyin merged 20 commits intoapache:mainfrom
merico-ai:blueprint_v200_py_plugins

Conversation

@CamilleTeruel
Copy link
Contributor

⚠️ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

  • I have read through the Contributing Documentation.
  • I have added relevant tests.
  • I have added relevant documentation.
  • I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.

Summary

What does this PR do?

Does this close any open issues?

Closes xx

Screenshots

Include any relevant screenshots here.

Other Information

Any other information that is important to this PR.

@CamilleTeruel CamilleTeruel force-pushed the blueprint_v200_py_plugins branch 2 times, most recently from e0e7209 to 7400bcd Compare February 28, 2023 16:01
Camille Teruel and others added 20 commits February 28, 2023 17:17
A stream must declare its domain types ("CICD", "TICKET", "CROSS", etc), not the domain model it produces.
…tV200

A remoteDatasourcePlugin struct as a PluginMeta must be castable to DataSourcePluginBlueprintV200.
Make Plugin inherit ABC to ensure all abstract methods are implemented by subclasses.
Not all plugins need transformation rules, so we make it optional.
Even if connections and transformation rules are sent from go to python,
they are also models backed by a tool table.
Now that we receive a Connection with its id, we don't need separate connection_id in Context.
…mand

Use the genuine tool scopes as input and genuine domain scopes as output to the make-pipeline command.
For tool scope inputs, python plugins are oblivious of the BlueprintScope concept, the go side taking care
of fetching and passing the real tool scopes.
For the domain scope outputs, it requires to send back the domain scope as a json object together with a type name to allow reconstruction on go side.
Because of limited go reflective abilities, the reconstruction function hardcodes the existing domain scope types.
Python plugins now support a two level scope hierarchy: scopes and scope groups.
When the group_id query parameter is absent, the default implementation calls
`remote_scope_groups` that is redefined in each plugin. If the group_id is given,
the default implementation call `remote_scopes` that return the scopes that are in a given group id.
@CamilleTeruel CamilleTeruel force-pushed the blueprint_v200_py_plugins branch 2 times, most recently from 4cd3eb6 to a8d2ddf Compare February 28, 2023 16:22
@CamilleTeruel CamilleTeruel marked this pull request as ready for review February 28, 2023 17:12
Copy link
Contributor

@hezyin hezyin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hezyin hezyin merged commit ce1031a into apache:main Feb 28, 2023
@keon94 keon94 deleted the blueprint_v200_py_plugins branch February 28, 2023 19:04
@keon94
Copy link
Contributor

keon94 commented Feb 28, 2023

Relates to #4450

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants