Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add support for doris datasource #14087

Merged
merged 10 commits into from
Nov 28, 2023
Merged

feat: add support for doris datasource #14087

merged 10 commits into from
Nov 28, 2023

Conversation

chyueyi
Copy link
Contributor

@chyueyi chyueyi commented Nov 23, 2023 •
edited

Describe your changes:

feat : add support for doris datasource

Apache doris is a very popular mpp database, we use doris as our datawarehouse,so I worked on doris datasource ingestion.

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@github-actions github-actions bot added UI UI specific issues Ingestion backend labels Nov 23, 2023
@github-actions GitHub Actions
Copy link

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@harshach harshach added the safe to test Add this label to run secure Github workflows on PRs label Nov 23, 2023
@harshach
Copy link
Collaborator

hi @chyueyi thanks for your contribution.
@pmbrull for review

@github-actions GitHub Actions
Copy link

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 23, 2023 •
edited

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 48%
 48.55% (23681/48780) 30.26% (8865/29297) 30.11% (2683/8911)

TeddyCr
TeddyCr reviewed Nov 24, 2023
View reviewed changes
ingestion/src/metadata/ingestion/source/database/doris/metadata.py

table_columns = []
primary_columns = []
# row schema: Field, Type, Collation, Null, Key, Default, Extra, Privileges, Comment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a pedantic model to have a more explicit schema validation and access for the row?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry,I don't know how to do that. Could you please give me a example @TeddyCr

@TeddyCr
Copy link
Contributor

TeddyCr commented Nov 24, 2023

Thanks @chyueyi for the PR looks great. I left a comment. Could you also add the python dependency in the pyproject.toml for pydoris package.

@github-actions GitHub Actions
Copy link

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@chyueyi
Copy link
Contributor Author

chyueyi commented Nov 27, 2023

Thanks @chyueyi for the PR looks great. I left a comment. Could you also add the python dependency in the pyproject.toml for pydoris package.

added it👌

@harshach
Copy link
Collaborator

@TeddyCr is it good to merge?

@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 28, 2023

[open-metadata-ui] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 28, 2023

[open-metadata-ingestion] SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

0.0% 0.0% Coverage
4.3% 4.3% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@TeddyCr TeddyCr enabled auto-merge (squash) November 28, 2023 08:02
@Sachin-chaurasiya
Copy link
Member

Sachin-chaurasiya commented Nov 28, 2023 •
edited

Hello @chyueyi , thanks for the PR, can you add the doc for the source here?

Also, please attach the screenshot or recording for the UI.

Feel free to let us know if you need any help.

@TeddyCr TeddyCr merged commit b6b337e into open-metadata:main Nov 28, 2023
18 of 19 checks passed
@TeddyCr
Copy link
Contributor

TeddyCr commented Nov 28, 2023

Hello @chyueyi , thanks for the PR, can you add the doc for the source here?

Also, please attach the screenshot or recording for the UI.

Feel free to let us know if you need any help.

@chyueyi regarding this comment could you open it in a new PR?

@chyueyi
Copy link
Contributor Author

chyueyi commented Nov 28, 2023

Hello @chyueyi , thanks for the PR, can you add the doc for the source here?
Also, please attach the screenshot or recording for the UI.
Feel free to let us know if you need any help.

@chyueyi regarding this comment could you open it in a new PR?

OK

@chyueyi chyueyi deleted the issue-12344 branch November 28, 2023 09:23
@sureshms
Copy link
Contributor

@TeddyCr and @chyueyi, can some of these issues be closed? https://github.com/open-metadata/OpenMetadata/issues?q=is%3Aissue+is%3Aopen+doris

MrVinegar pushed a commit to MrVinegar/OpenMetadata that referenced this pull request Dec 15, 2023
@chyueyi @harshach
feat: add support for doris datasource (open-metadata#14087)
750941f 
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
@pmbrull pmbrull mentioned this pull request Dec 18, 2023
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chirag-madlani chirag-madlani chirag-madlani approved these changes

@TeddyCr TeddyCr TeddyCr approved these changes

@harshach harshach Awaiting requested review from harshach harshach is a code owner

@karanh37 karanh37 Awaiting requested review from karanh37 karanh37 is a code owner

Assignees

@pmbrull pmbrull

Labels
backend Ingestion safe to test Add this label to run secure Github workflows on PRs UI UI specific issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@chyueyi @harshach @TeddyCr @Sachin-chaurasiya @sureshms @chirag-madlani @pmbrull