Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Catalog] Doris Catalog #5175

Merged
merged 23 commits into from
Dec 9, 2023
Merged

Conversation

gnehil
Copy link
Contributor

@gnehil gnehil commented Jul 28, 2023

Purpose of this pull request

config example:

sink {
  Doris {
    source_table_name = "jdbc"
    catalog {
      factory = "Doris"
    }
    fenodes = "127.0.0.1:8030"
    query-port = "9030"
    username = "root"
    password = ""
    table.identifier = "test_db.test_tbl"
    savemode = "KEEP_SCHEMA_AND_DATA"
  }
}

create table action is only triggered when savemode = "KEEP_SCHEMA_AND_DATA"

Check list

@gnehil gnehil changed the title [feature] Doris Catalog [Feature][Catalog] Doris Catalog Jul 28, 2023
@gnehil gnehil marked this pull request as ready for review July 31, 2023 02:06
@Hisoka-X Hisoka-X added this to the 2.3.4 milestone Aug 23, 2023
@Hisoka-X
Copy link
Member

Hisoka-X commented Aug 23, 2023

create {
    distribution-columns = "id"
    distribution-buckect = "10"
    properties {
        replication_allocation = "tag.location.default: 1"
    }
}

I think we need to basically unify the definition methods of different connectors for table create. Can you implement Doris's automatic table create function definition according to the table create template provided by StarRocks https://seatunnel.apache.org/docs/2.3.3/connector-v2/sink/StarRocks#save_mode_create_template ? Please.

@Hisoka-X
Copy link
Member

Could you add some e2e for this new feature? Please.

@gnehil
Copy link
Contributor Author

gnehil commented Sep 14, 2023

Could you add some e2e for this new feature? Please.

done

@EricJoy2048
Copy link
Member

Please fix code style by run sh tools/spotless/pre-commit.sh

@gnehil
Copy link
Contributor Author

gnehil commented Sep 15, 2023

Please fix code style by run sh tools/spotless/pre-commit.sh

done

@EricJoy2048 EricJoy2048 reopened this Oct 12, 2023
@Hisoka-X
Copy link
Member

Could you update your fork repository? The CI didn't start because your dev branch no up to date.
image

@gnehil
Copy link
Contributor Author

gnehil commented Oct 12, 2023

Could you update your fork repository? The CI didn't start because your dev branch no up to date. image

It's done, please review again

@Hisoka-X
Copy link
Member

@Hisoka-X
Copy link
Member

The CI seem like failed. Could you investigate it?
image

@EricJoy2048
Copy link
Member

Is there any latest progress on this PR?

@CalvinKirs CalvinKirs merged commit 1d3e335 into apache:dev Dec 9, 2023
4 checks passed
Carl-Zhou-CN pushed a commit to Carl-Zhou-CN/incubator-seatunnel that referenced this pull request Dec 12, 2023
* doris catalog

* fix create table

* add license header

* add license header

* code style

* do not exec create table when save mode is null

* do not exec create table when save mode is not equals to KEEP_SCHEMA_AND_DATA

* update create table

* add doris catalog e2e test

* add license header

* add license header and fix style

* static string sql template

* spotless check

* resolve api change

* disable ut

* merge test

* code style

* test jdbc driver

* add be alive check

* mod container

* fix container

* support drop table and fix create template

* resolve conflicts

---------

Co-authored-by: gnehil <adamlee489@gamil.com>
Co-authored-by: gnehil <gnehil489@github>
alextinng pushed a commit to alextinng/seatunnel that referenced this pull request Dec 19, 2023
* doris catalog

* fix create table

* add license header

* add license header

* code style

* do not exec create table when save mode is null

* do not exec create table when save mode is not equals to KEEP_SCHEMA_AND_DATA

* update create table

* add doris catalog e2e test

* add license header

* add license header and fix style

* static string sql template

* spotless check

* resolve api change

* disable ut

* merge test

* code style

* test jdbc driver

* add be alive check

* mod container

* fix container

* support drop table and fix create template

* resolve conflicts

---------

Co-authored-by: gnehil <adamlee489@gamil.com>
Co-authored-by: gnehil <gnehil489@github>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants