Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-27562 Publish SBOM artifacts #4953

Merged
merged 2 commits into from
Jan 20, 2023
Merged

HBASE-27562 Publish SBOM artifacts #4953

merged 2 commits into from
Jan 20, 2023

Conversation

dongjoon-hyun
Copy link
Member

This PR aims to publish SBOM artifacts along with the other Apache projects.

Here is an article to give some context.

Software Bill of Materials (SBOM) are additional artifacts containing the aggregate of all direct and transitive dependencies of a project. The US Government (based on NIST recommendations) currently accepts only the three most popular SBOM standards as valid, namely: CycloneDX, Software Identification (SWID) tag, Software Package Data Exchange® (SPDX).

We can use one of the Maven plugin, CycloneDX maven plugin, a lightweight software bill of materials (SBOM) standard designed for use in application security contexts and supply chain component analysis.

https://maven.apache.org/plugins/index.html#misc

The expected results

$ mvn install -DskipTests
...

$ ls -al ~/.m2/repository/org/apache/hbase/hbase-common/3.0.0-alpha-4-SNAPSHOT
total 5064
drwxr-xr-x  11 dongjoon  staff     352 Jan  9 18:28 .
drwxr-xr-x   4 dongjoon  staff     128 Jan  9 18:28 ..
-rw-r--r--   1 dongjoon  staff     482 Jan  9 18:28 _remote.repositories
-rw-r--r--   1 dongjoon  staff  159174 Jan  9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-cyclonedx.json
-rw-r--r--   1 dongjoon  staff  139170 Jan  9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-cyclonedx.xml
-rw-r--r--   1 dongjoon  staff  684842 Jan  9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-sources.jar
-rw-r--r--   1 dongjoon  staff  267751 Jan  9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-test-sources.jar
-rw-r--r--   1 dongjoon  staff  443154 Jan  9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-tests.jar
-rw-r--r--   1 dongjoon  staff  871542 Jan  9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT.jar
-rw-r--r--   1 dongjoon  staff    6620 Jan  9 18:27 hbase-common-3.0.0-alpha-4-SNAPSHOT.pom
-rw-r--r--   1 dongjoon  staff    1811 Jan  9 18:28 maven-metadata-local.xml

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jan 10, 2023

Could you review this, @ndimiduk , @huaxiangsun ?

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 5m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 5s master passed
+1 💚 compile 5m 55s master passed
+1 💚 spotless 0m 59s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 5m 20s the patch passed
+1 💚 compile 5m 53s the patch passed
+1 💚 javac 5m 53s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 19m 57s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌ spotless 0m 13s patch has 22 errors when running spotless:check, run spotless:apply to fix.
_ Other Tests _
+1 💚 asflicense 0m 20s The patch does not generate ASF License warnings.
59m 47s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4953
Optional Tests dupname asflicense javac hadoopcheck spotless xml compile
uname Linux d4c7d0da981e 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 3f1087f
Default Java Eclipse Adoptium-11.0.17+8
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 81 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Copy link
Contributor

Apache9 commented Jan 10, 2023

Please run mvn spotless:apply to fix the format error?

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 23s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 56s master passed
+1 💚 compile 2m 20s master passed
+1 💚 shadedjars 4m 34s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 39s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 57s the patch passed
+1 💚 compile 2m 14s the patch passed
+1 💚 javac 2m 14s the patch passed
+1 💚 shadedjars 7m 4s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 17s the patch passed
_ Other Tests _
-1 ❌ unit 322m 19s root in the patch failed.
358m 43s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4953
Optional Tests javac javadoc unit shadedjars compile
uname Linux 3087f985030f 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 3f1087f
Default Java Eclipse Adoptium-11.0.17+8
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/testReport/
Max. process+thread count 2434 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 5m 14s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 37s master passed
+1 💚 compile 2m 3s master passed
+1 💚 shadedjars 4m 54s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 1s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 16s the patch passed
+1 💚 compile 2m 12s the patch passed
+1 💚 javac 2m 12s the patch passed
+1 💚 shadedjars 5m 27s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 38s the patch passed
_ Other Tests _
+1 💚 unit 388m 9s root in the patch passed.
426m 2s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4953
Optional Tests javac javadoc unit shadedjars compile
uname Linux 9df1f3b4c312 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 3f1087f
Default Java Temurin-1.8.0_352-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/testReport/
Max. process+thread count 4761 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@dongjoon-hyun
Copy link
Member Author

Thank you, @Apache9 . I did mvn spotless:apply and applied it now.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 54s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 19s master passed
+1 💚 compile 6m 7s master passed
+1 💚 spotless 0m 50s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 5m 49s the patch passed
+1 💚 compile 6m 12s the patch passed
+1 💚 javac 6m 12s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 19m 58s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚 spotless 1m 5s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 19s The patch does not generate ASF License warnings.
57m 19s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4953
Optional Tests dupname asflicense javac hadoopcheck spotless xml compile
uname Linux 0fc75f598802 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 4add525
Default Java Eclipse Adoptium-11.0.17+8
Max. process+thread count 82 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk
Copy link
Member

FYI @dongjoon-hyun most of HBase's dependencies are masked behind the hbase-thirdparty libraries. For full utility, I suspect that we'll need that project to also publish sbom.

@ndimiduk
Copy link
Member

Okay this is a problem.

[WARNING] The following plugins are not marked as thread-safe in Apache HBase:
[WARNING]   org.cyclonedx:cyclonedx-maven-plugin:2.7.3
[WARNING] 
[WARNING] Enable debug to see precisely which goals are not marked as thread-safe.

Is there a newer version of this plugin that supports multi-threaded builds?

@ndimiduk
Copy link
Member

ndimiduk commented Jan 10, 2023

Looks like not yet, CycloneDX/cyclonedx-maven-plugin#77

@ndimiduk
Copy link
Member

Have you explored the Apache release process for the resulting artifacts. Does the release manager need to do anything special with the attached artifacts in order to publish them properly?

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 19s master passed
+1 💚 compile 2m 20s master passed
+1 💚 shadedjars 4m 30s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 12s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 1s the patch passed
+1 💚 compile 2m 14s the patch passed
+1 💚 javac 2m 14s the patch passed
+1 💚 shadedjars 5m 6s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 24s the patch passed
_ Other Tests _
+1 💚 unit 311m 10s root in the patch passed.
343m 50s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4953
Optional Tests javac javadoc unit shadedjars compile
uname Linux c44657c7ea78 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 4add525
Default Java Eclipse Adoptium-11.0.17+8
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/testReport/
Max. process+thread count 4866 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jan 10, 2023

Thank you for review, @ndimiduk . This activity focuses on publishing SBOMs of the Maven Central jars.
So, the SBOMs are also published into Maven Central as an immutable files and signed by the ASF signer in the same way with jars. FYI, here is Apache ORC 1.8.2 RC1 vote artifact which is I'm leading now, @ndimiduk .

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 14s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 32s master passed
+1 💚 compile 2m 16s master passed
+1 💚 shadedjars 4m 49s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 20s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 21s the patch passed
+1 💚 compile 2m 5s the patch passed
+1 💚 javac 2m 5s the patch passed
+1 💚 shadedjars 5m 32s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 40s the patch passed
_ Other Tests _
+1 💚 unit 397m 42s root in the patch passed.
431m 39s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4953
Optional Tests javac javadoc unit shadedjars compile
uname Linux b9a038f125a0 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 4add525
Default Java Temurin-1.8.0_352-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/testReport/
Max. process+thread count 4788 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Copy link
Contributor

Apache9 commented Jan 18, 2023

Have you explored the Apache release process for the resulting artifacts. Does the release manager need to do anything special with the attached artifacts in order to publish them properly?

Let me test locally.

@Apache9
Copy link
Contributor

Apache9 commented Jan 18, 2023

This is what I've gotten in the local staging dir by running command

mvn clean deploy -DskipTests -Dcheckstyle.skip=true -DaltStagingDirectory=/home/zhangduo/sbom-staged -P apache-release,release -DskipRemoteStaging

zhangduo@zhangduo-VirtualBox:~/sbom-staged/deferred/org/apache/hbase/hbase-client/3.0.0-alpha-4-SNAPSHOT$ ll -h hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.*
-rw-rw-r-- 1 zhangduo zhangduo 229K  1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.json
-rw-rw-r-- 1 zhangduo zhangduo  833  1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.json.asc
-rw-rw-r-- 1 zhangduo zhangduo 196K  1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.xml
-rw-rw-r-- 1 zhangduo zhangduo  833  1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.xml.asc

Seems fine, we will publish the sbom files along with other files, no more works needed.

And as @ndimiduk have already pointed out, for hbase-thirdparty there is a problem that, we shade and relocate other libraries so we will miss the information when others depend on hbase-thirdparty. But anyway, I think this can be improved later.

@dongjoon-hyun
Copy link
Member Author

Thank you so much, @Apache9!

@Apache9
Copy link
Contributor

Apache9 commented Jan 19, 2023

Going to merge this later unless objections.

Thanks.

@Apache9 Apache9 merged commit 5feb06b into apache:master Jan 20, 2023
Apache9 pushed a commit that referenced this pull request Jan 20, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 5feb06b)
Apache9 pushed a commit that referenced this pull request Jan 20, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 5feb06b)
Apache9 pushed a commit that referenced this pull request Jan 20, 2023
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 5feb06b)
@dongjoon-hyun
Copy link
Member Author

Thank you all!

vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 5feb06b)
(cherry picked from commit 67a30f3)
Change-Id: I164dea1e48ad1ec524f867547d454d4555984549
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants