Fix #2894 - Profiler Processor & Metrics #2900
Conversation
| config_dict: dict, | ||
| metadata_config_dict: dict, | ||
| ctx: WorkflowContext, | ||
| **kwargs |
There was a problem hiding this comment.
Allows us to pass more parameters to the processor, such as the session in the ORM Processor. It should be transparent for the existent Processors
| self.session.add_all(data) | ||
| self.session.commit() | ||
|
|
||
| hist = Metrics.HISTOGRAM(TestHist.num, bins=5) |
There was a problem hiding this comment.
We can now pass kwargs when creating metrics to be more flexible in internal functionalities. Still needs to be updated in the Processor
| return issubclass(_type.__class__, Numeric) | ||
|
|
||
|
|
||
| def is_quantifiable(_type) -> bool: |
There was a problem hiding this comment.
helps handle what types we support for each metric
Sonar is complaining about the join condition EDIT: ok fixed this, just needed a fresher mind 😵💫 |
|
[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed! |
| :param _from: From where do we load the sink class. Ingestion by default. | ||
| """ | ||
| processor_class = get_class( | ||
| "metadata.{}.processor.{}.{}Processor".format( |
There was a problem hiding this comment.
We need to make this more flexible to load the class without depending on the packaging convention.
Not important right now but can be annoying for users to write their own sources, processors
There was a problem hiding this comment.
That's a good point. Let's tackle this together with #2848. I'll add a comment there
Describe your changes :
This PR fixes #2894 and partially #2878
There are two main changes in this PR:
workflowlogic. We have also updated the Status to properly log the results.QueryMetric. We have also developed intermediate functions (CONCAT) that can be used when computing other metrics.Type of change :
Frontend Preview (Screenshots) :
For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.
Checklist:
Reviewers