Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add google cloud storage support to java_export #463

Merged
merged 2 commits into from
Dec 4, 2020
Merged

Add google cloud storage support to java_export #463

merged 2 commits into from
Dec 4, 2020

Conversation

zacharya19
Copy link
Contributor

Closes #458 (this issue is for publishing only, I would probably open another one to add GCS support to maven_install, hopefully I'll get to it).
@shs96c LMK what you think.

@jin jin requested a review from shs96c October 5, 2020 02:00
shs96c
shs96c requested changes Oct 8, 2020
View reviewed changes
Copy link
Collaborator

@shs96c shs96c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it's in good shape. Thank you for writing it!

@jin can weigh in on the request to avoid asking people to add extra steps to their workspaces. Is there any way we can import them into the repo itself?

It'd be nice to have some tests, but I guess it's hard to add those for publishing.

If credentials are needed, how are they detected and specified?

README.md Outdated
@@ -952,6 +960,10 @@ bazel run --stamp \
//user_project:exported_lib.publish`
```

It's also possible to publish to a Google Cloud Storage bucket (make sure to not add a trailing slash):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: could we just remove a trailing slash if we see one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought about it, but I didn't want to mess with the users args, in case s/he really want the extra slash from some reason.
Now that I think about maybe it doesn't really make sense, will add it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

defs.bzl Outdated
"""
Adds the maven dependencies that are needed to run java_export rule.
"""
maven_install(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to use a jvm_import to pull in a jar we add to the repo? Other than the vast number of dependencies?

The current way of using rules_jvm_external doesn't need anyone to go through the cumbersome "import the ruleset, load the rule set deps, run the deps setup" dance. That's why the current deployer doesn't use anything outside the Java 8 JRE.

Copy link
Contributor Author

@zacharya19 zacharya19 Oct 8, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm wrong, but AFAIK bazel doesn't support transitive deps, so the only real way to avoid forcing the user from adding those deps is to save the jars in our repo, which isn't a solution IMHO.
Am I wrong about it? Even with jvm_import it will include target from different repo, which the user will need to add.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's just a single (or few) jar dependency, we could vendor it.

How about doing it this way instead of requiring a load statement? https://github.com/bazelbuild/rules_jvm_external#exporting-and-consuming-artifacts-from-external-repositories

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GCS lib has 23 deps (so 24 total jars), we can vendor them but I'm not sure it's the right approach.
LMK what you think about it, if you prefer it to be vendored I'll change it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shs96c and I had a discussion offline about this, and I think that it's would be a cleaner approach to go with your implementation, with an internal maven_install repo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@zacharya19
Copy link
Contributor Author

zacharya19 commented Oct 8, 2020 •
edited
Loading

If credentials are needed, how are they detected and specified?

Right now I'm using the application default, since it's GCS best practice.
I can add support for providing credentials file, but I think it's better to use application default.

jin
jin approved these changes Oct 13, 2020
View reviewed changes
Copy link
Member

@jin jin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please also get approval from @shs96c.

defs.bzl Outdated Show resolved Hide resolved
defs.bzl Outdated
@@ -154,6 +154,22 @@ def _parse_artifact_str(artifact_str):
else:
return parse.parse_maven_coordinate(artifact_str)

JAVA_EXPORT_REPO_NAME = "internal_java_exports_maven"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefix with _INTERNAL to make it clear that this is not meant for consumption by end users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed to INTERNAL_JAVA_EXPORT_REPO_NAME since vars starts with underscore are internal and can't be imported in the build file.

README.md Outdated Show resolved Hide resolved
@zacharya19 zacharya19 force-pushed the z/JavaExportGCS branch 2 times, most recently from 7bc2bf5 to a601d56 Compare October 14, 2020 08:52
@zacharya19 zacharya19 requested review from jin and shs96c October 14, 2020 08:52
@shs96c
Copy link
Collaborator

shs96c commented Oct 27, 2020

Sorry for the slow feedback on this. After discussing with @jin, I think the best thing to do to avoid vendoring in all the jars we need is to move rules_jvm_external to a two-step initialisation process (load the deps, run the setup). I'll prep a PR for that for you to build on.

@shs96c
Copy link
Collaborator

shs96c commented Nov 13, 2020

After a far too long, I've created #481 to help make it easier for us to pull in the bits that we need to rules_jvm_external

@shs96c
Copy link
Collaborator

shs96c commented Nov 19, 2020

I've landed #481 so it would be helpful if you could rebase on top of that, and use those features to pull in any dependencies, please.

@zacharya19
Copy link
Contributor Author

zacharya19 commented Nov 27, 2020 •
edited
Loading

@shs96c Done, LMK what you think, it's a much smaller change now, most of it is the maven_pin file.

shs96c
shs96c approved these changes Dec 4, 2020
View reviewed changes
Copy link
Collaborator

@shs96c shs96c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for you patience while we got the infrastructure in to support this, and thank you again for the patch. LGTM!

@shs96c shs96c merged commit 1d6fa32 into bazelbuild:master Dec 4, 2020
@jin jin mentioned this pull request Jan 6, 2021
Closed
4 tasks
@rmrod rmrod mentioned this pull request Apr 1, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shs96c shs96c shs96c approved these changes

@c-parsons c-parsons Awaiting requested review from c-parsons

@jin jin Awaiting requested review from jin jin is a code owner

Assignees
No one assigned
Labels
cla: yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Java export - publish to private google cloud storage bucket
4 participants
@zacharya19 @shs96c @jin @googlebot