[Connector]Add hbase source connector #6348
Conversation
|
Create hbase table and inset 10000 rows: |
...l-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/resources/hbase-to-assert.conf
Show resolved
Hide resolved
...src/main/java/org/apache/seatunnel/connectors/seatunnel/hbase/source/HbaseSourceFactory.java
Show resolved
Hide resolved
|
Excuse me, is there any support about Kerberos authentication in hbase source connector? |
|
I have modified the code according to the suggestion, thanks for @hailin0 @Hisoka-X review. I have communicated with @lihjChina on wechat. At present, he has used this pr in the production environment of the company. After this pr is merged, next he will contribute to add Kerberos authentication in hbase source connector. |
|
cc @TyrantLucifer as well. |
There was a problem hiding this comment.
Overall LGTM, Could you please offer some screeshots to verify it worked? Maybe require three engines(flink/spark/zeta).
There are also relevant screenshots of the data that need to be verified for correct data reading.
Due to hbase e2e did not be enabled in CI/CD, so there has no way to prove that this connector is usable.
| Connection connection = ConnectionFactory.createConnection(hbaseConfiguration); | ||
| return connection; | ||
| } catch (IOException e) { | ||
| throw new RuntimeException(e); |
| return Double.valueOf(Bytes.toString(cell)); | ||
| case BYTES: | ||
| return cell; | ||
| case DATE: |
There was a problem hiding this comment.
date/time/timestamp as String? It maybe has some problem. We can add parameters to setting the format of these types as text file connector.
There was a problem hiding this comment.
date/time/timestamp as String? It maybe has some problem. We can add parameters to setting the format of these types as text file connector.
+1
There was a problem hiding this comment.
Thanks for advice. I have already modified it.
|
Use my hbase cluster for testing： For testing different engines, generate 10000 rows and written to hbase. spark： zeta: |
|
Why not use hbase docker? |
This requires adding mapping information. |
| |--------------------|--------|----------|---------------| | ||
| | zookeeper_quorum | string | yes | - | | ||
| | table | string | yes | - | | ||
| | query_columns | list | yes | - | |
There was a problem hiding this comment.
From this option we can implement schema projection ? If true, You need update here
[x] [schema projection](../../concept/connector-v2-features.md)
and the HbaseSource need implement SupportColumnProjection interface.
HbaseSource need implement SupportParallelism too, because it support parallelism .
|
I found that the |
|
Please fix ci |
Purpose of this pull request
#3018
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note.