Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-44681] Fix issues when writing Go application code using Spark Connect Go client library #14

Closed
wants to merge 2 commits into from

Conversation

hiboyang
Copy link
Contributor

@hiboyang hiboyang commented Aug 4, 2023

What changes were proposed in this pull request?

When trying to write Go application code using Spark Connect Go client and put the Go application code in users' own repo (e.g. https://github.com/user-foo/my-go-application), the Go application code cannot resolve Spark Connect Go client from https://github.com/apache/spark-connect-go correctly, due to two issues:

  1. The name github.com/apache/spark-connect-go/v_3_4 cannot resolve correctly to find the module, it complains not finding go.mod file under github.com/apache/spark-connect-go/v_3_4. After change the name to github.com/apache/spark-connect-go/v34, it is good.

  2. The Go application code needs generated Go protobuf code as well. Thus need to commit the generated Go protobuf code into github.com/apache/spark-connect-go repo.

Why are the changes needed?

See above.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Tested by run Go application code to use Spark Connect Go client.

@hiboyang
Copy link
Contributor Author

hiboyang commented Aug 4, 2023

@HyukjinKwon @grundprinzip @zhengruifeng I find some issues when I write a standalone Spark Connect application using this spark-connect-go repo as a library. The PR is to fix these issues.

@hiboyang
Copy link
Contributor Author

hiboyang commented Aug 9, 2023

This PR is mostly generated code like internal/generated/base.pb.go, internal/generated/base_grpc.pb.go, etc. We need to put those files into this repo so other Go application could reference this repo as a Go library properly.

Please ignore those generated code. Except those code, the PR is actually small :)

@@ -13,7 +13,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

module github.com/apache/spark-connect-go/v_3_4
module github.com/apache/spark-connect-go/v34
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's interesting that this hasn't caused issues before really. But I'm fine with the changes.

Copy link
Contributor Author

@hiboyang hiboyang Aug 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing! Yeah, previously we have code inside this repo referencing this same library , thus v_3_4 does not cause problem. When I wrote another Go application outside of this repo, referencing this repo as a library, then got issue.

@zhengruifeng
Copy link

merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants