Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#3292] fix(spark-connector): passing Gravitino catalog properties to spark connector #3358

Conversation

jerryshao
Copy link
Contributor

What changes were proposed in this pull request?

  1. passing Gravitino catalog properties with spark.bypass. prefix to spark connector
  2. refactor the catalog properties transform logic.

Why are the changes needed?

Fix: #3292

Does this PR introduce any user-facing change?

yes, add document

How was this patch tested?

  1. add UT
  2. test in local envriment

… spark connector (#3270)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment
@FANNG1
Copy link
Contributor

FANNG1 commented May 13, 2024

ci failure not related, re trigger it.

> Could not resolve all files for configuration ':meta:compileProtoPath'.
   > Could not download protobuf-java-3.24.4.jar (com.google.protobuf:protobuf-java:3.24.4)
      > Could not get resource 'https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.24.4/protobuf-java-3.24.4.jar'.
         > Could not GET 'https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.24.4/protobuf-java-3.24.4.jar'.
            > Connect to repo.maven.apache.org:443 [repo.maven.apache.org/146.75.92.215, repo.maven.apache.org/2a04:4e42:87:0:0:0:0:215] failed: Network is unreachable (connect failed)

@jerryshao jerryshao closed this May 13, 2024
@jerryshao jerryshao reopened this May 13, 2024
@jerryshao
Copy link
Contributor Author

@FANNG1 can you please identify the problems here?

@FANNG1
Copy link
Contributor

FANNG1 commented May 13, 2024

@FANNG1 can you please identify the problems here?

download Jdbc driver failed, re trigger it.

TrinoConnectorIT > initializationError FAILED
    java.net.ConnectException: Connection timed out
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Net.java:579)
        at java.base/sun.nio.ch.Net.connect(Net.java:568)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
        at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:174)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
        at com.datastrato.gravitino.integration.test.util.JdbcDriverDownloader.downloadJdbcDriver(JdbcDriverDownloader.java:33)
        at com.datastrato.gravitino.integration.test.trino.TrinoConnectorIT.startDockerContainer(TrinoConnectorIT.java:119)

@FANNG1
Copy link
Contributor

FANNG1 commented May 13, 2024

@jerryshao is it ok to merge it ? blocks #3366 now

@FANNG1 FANNG1 merged commit 7a6d045 into branch-0.5 May 13, 2024
32 of 44 checks passed
@jerryshao jerryshao deleted the cherry-pick-branch-0.5-013021d82734293a9fa27dfc1c8720ff912268d5 branch May 14, 2024 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants