Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-15844. Tag S3GuardTool entry points as LimitedPrivate/Evolving #1041

Closed

Conversation

steveloughran
Copy link
Contributor

@steveloughran steveloughran commented Jul 1, 2019

Marking S3GuardTool as LimitedPrivate("management-tools")/Evolving to reassure people using it as an API that we intend to retain it as an API for creating/destroying/maintaining S3Guard tables

Tested S3 Ireland (as part of a HADOOP-16384 test run)

Change-Id: Ic58bbd917da901a1bd2322ae0f41ca00e60dcc06

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 30 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1073 trunk passed
+1 compile 37 trunk passed
+1 checkstyle 25 trunk passed
+1 mvnsite 36 trunk passed
+1 shadedclient 710 branch has no errors when building and testing our client artifacts.
+1 javadoc 22 trunk passed
0 spotbugs 55 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 54 trunk passed
_ Patch Compile Tests _
+1 mvninstall 30 the patch passed
+1 compile 28 the patch passed
+1 javac 28 the patch passed
+1 checkstyle 16 the patch passed
+1 mvnsite 33 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 718 patch has no errors when building and testing our client artifacts.
+1 javadoc 21 the patch passed
+1 findbugs 61 the patch passed
_ Other Tests _
+1 unit 287 hadoop-aws in the patch passed.
+1 asflicense 29 The patch does not generate ASF License warnings.
3294
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1041/1/artifact/out/Dockerfile
GITHUB PR #1041
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux d4cf191421b8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1e727cf
Default Java 1.8.0_212
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1041/1/testReport/
Max. process+thread count 447 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1041/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@mackrorysd
Copy link
Contributor

What's the difference you see between LimitedPrivate("management tools") and Public? When things are LimitedPrivate("HBase") there's a specific community that we know to coordinate with. With "management tools" it's arbitrarily open, and I don't know what the compatibility guarantees we need to hold to are. I know with Hadoop 3's compatibility we technically violated compatibility guidelines with the original metrics framework being removed before it had been deprecated for a full release, because metrics were also seen as a similar "administrative" or "auxiliary" interface and it didn't need to really be "public". If we're doing that again, I think we can still set expectations more clearly.

@steveloughran
Copy link
Contributor Author

We've had some LimitedPrivate("management tools") elsewhere, while DistCP copyListing is "LimitedPrivate("Distcp support tools")"

it's effectively saying "its a lot less stable than other things" , but there is some expectation that it may be used externally

There's a limited number of people who do management tools -they know our email addresses. What I don't want to do is say this is only for one specific project as that's a bit dictatorial too.

\cite{all those apps which ended up using stuff exclusive for mapreduce just so they could work}

Maybe: add something about management tools in the compatibility specs: they may be public but unstable, and you get to track the code

@steveloughran steveloughran added the fs/s3 changes related to hadoop-aws; submitter must declare test endpoint label Jul 10, 2019
@steveloughran
Copy link
Contributor Author

Tracked down where the management tools ref was used already org.apache.hadoop.conf.ReconfigurationTaskStatus

Discussion here was to tell hadoop developers that this was used in places, and avoided us saying this was hard coded to any specific product

@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@apache apache deleted a comment from hadoop-yetus Sep 6, 2019
@steveloughran
Copy link
Contributor Author

@mackrorysd do you want to revisit this? It's to tell us developers working on the class that this is sometimes used directly by specific tools and they need to take care when changing things

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
…ent-tools")/evolving

Change-Id: Ic58bbd917da901a1bd2322ae0f41ca00e60dcc06
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 58s trunk passed
+1 💚 compile 0m 32s trunk passed
+1 💚 checkstyle 0m 24s trunk passed
+1 💚 mvnsite 0m 35s trunk passed
+1 💚 shadedclient 17m 13s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 26s trunk passed
+0 🆗 spotbugs 1m 3s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 1m 1s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 32s the patch passed
+1 💚 compile 0m 27s the patch passed
+1 💚 javac 0m 27s the patch passed
+1 💚 checkstyle 0m 18s the patch passed
+1 💚 mvnsite 0m 32s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 21s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 22s the patch passed
+1 💚 findbugs 1m 3s the patch passed
_ Other Tests _
+1 💚 unit 1m 20s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 28s The patch does not generate ASF License warnings.
65m 1s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1041/12/artifact/out/Dockerfile
GITHUB PR #1041
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 36fb71a0adc7 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 9fe4c37
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1041/12/testReport/
Max. process+thread count 422 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1041/12/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Jun 3, 2020
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 23m 51s trunk passed
+1 💚 compile 0m 45s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 compile 0m 37s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 checkstyle 0m 28s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 shadedclient 17m 59s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 35s hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚 javadoc 0m 29s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗 spotbugs 1m 8s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 1m 6s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 39s the patch passed
+1 💚 compile 0m 35s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javac 0m 35s the patch passed
+1 💚 compile 0m 27s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 javac 0m 27s the patch passed
+1 💚 checkstyle 0m 20s the patch passed
+1 💚 mvnsite 0m 36s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 59s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 27s hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚 javadoc 0m 23s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 findbugs 1m 7s the patch passed
_ Other Tests _
+1 💚 unit 1m 26s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 29s The patch does not generate ASF License warnings.
71m 35s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1041/3/artifact/out/Dockerfile
GITHUB PR #1041
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f42fb64cee84 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e756fe3
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1041/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1041/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1041/3/testReport/
Max. process+thread count 413 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1041/3/console
versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor Author

did this with the DirMarker patch. Closing

@steveloughran steveloughran deleted the s3/HADOOP-15844-S3GuardTool branch October 15, 2021 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fs/s3 changes related to hadoop-aws; submitter must declare test endpoint
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants