BIGTOP-4035: Fix ranger deb packaging and puppet script

[JiaLiangC]

Description of PR

Fix ranger deb packaging and puppet script

How was this patch tested?

manual test

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
• Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

[sekikn]

@JiaLiangC Thank you for the update, and sorry that my previous comment was insufficient. For Rocky 8, we still have to keep the package name as "python36", otherwise we get the following error. To put it simply, we have to use "python36" only for Rocky 8 and "python3" for other distros including CentOS 7.

$ ./docker-hadoop.sh -d -dcp -C config_rockylinux-8.yaml -F docker-compose-cgroupv2.yml -G -L -k hdfs,hbase,ranger -k ranger -c 1

...

Notice: /Stage[main]/Ranger::Admin/Package[ranger-admin]/ensure: created
Error: Could not update: Execution of '/usr/bin/dnf -d 0 -e 1 -y list python3' returned 1: Error: No matching Packages to list
Error: /Stage[main]/Ranger::Admin/Package[python3]/ensure: change from 'purged' to 'latest' failed: Could not update: Execution of '/usr/bin/dnf -d 0 -e 1 -y list python3' returned 1: Error: No matching Packages to list
Notice: /Stage[main]/Ranger::Admin/Exec[initdb]: Dependency Package[python3] has failures: true

Other updates are supposed to work as expected, so I'll merge it once the issue above is fixed.

[JiaLiangC]

@sekikn
I apologize for not thoroughly testing and causing you the additional work of reviewing and retesting. I haven't completely set up and compiled the test environment for different operating systems for the big data components locally. Can we use Jenkins to compile a specific component and trigger its CI process?

[guyuqi]

@sekikn I apologize for not thoroughly testing and causing you the additional work of reviewing and retesting. I haven't completely set up and compiled the test environment for different operating systems for the big data components locally. Can we use Jenkins to compile a specific component and trigger its CI process?

Hi @JiaLiangC
Thanks for working on it.
You don't have to set up and compile the test environment for different distributions yourself.
Bigtop offers various Docker images for different distributions:
https://hub.docker.com/r/bigtop/slaves/tags?page=1&name=3.2.1
You can test the PR against these docker OS containers.
Just for your information. :-)

[JiaLiangC]

tested success on rocky8
./docker-hadoop.sh -d -dcp -C config_rockylinux-8.yaml -F docker-compose-cgroupv2.yml -G -L -k hdfs,ranger -k ranger -c 1
@sekikn

[sekikn]

Thank you for the update and advice @JiaLiangC @guyuqi.

As Yuqi said, you can use the images published on DockerHub for running build (use bigtop/slaves for this purpose, in which both puppet and toolchain are installed) and smoke test (use bigtop/puppet, in which only puppet is installed).
But I've just found that the current trunk images are obsolete. They were built over a year ago and the "trunk-rockylinux-8" image is recognized as "RedHat" by facter.

$ docker run --rm -it bigtop/puppet:trunk-rockylinux-8 cat /etc/redhat-release
Unable to find image 'bigtop/puppet:trunk-rockylinux-8' locally
trunk-rockylinux-8: Pulling from bigtop/puppet

...

Rocky Linux release 8.5 (Green Obsidian)
$ docker run --rm -it bigtop/puppet:trunk-rockylinux-8 facter operatingsystem
RedHat
$ docker images bigtop/puppet:trunk-rockylinux-8
REPOSITORY      TAG                  IMAGE ID       CREATED         SIZE
bigtop/puppet   trunk-rockylinux-8   26e196c3c91c   13 months ago   570MB

But if we rebuild it based on the latest Rocky image, facter recognizes it as "Rocky".

$ cd docker/bigtop-puppet
$ ./build.sh trunk-rockylinux-8

...

$ docker images bigtop/puppet:trunk-rockylinux-8
REPOSITORY      TAG                  IMAGE ID       CREATED          SIZE
bigtop/puppet   trunk-rockylinux-8   428a4b9f5c8c   12 minutes ago   606MB
$ docker run --rm -it bigtop/puppet:trunk-rockylinux-8 cat /etc/redhat-release
Rocky Linux release 8.8 (Green Obsidian)
$ docker run --rm -it bigtop/puppet:trunk-rockylinux-8 facter operatingsystem
Rocky

So the following code

    if ( $operatingsystem =~ /^(?i:(redhat))$/ and 0 <= versioncmp($operatingsystemmajrelease, '8')) {

should be as follows. Would you fix it?

    if ($operatingsystem =~ /^(?i:(redhat|rocky))$/ and 0 <= versioncmp($operatingsystemmajrelease, '8')) {

Regarding the trunk images, they should be periodically built and pushed into DockerHub so that developers can use the latest version. I'll configure the CI for doing so later.

[JiaLiangC]

@sekikn already fixed

[JiaLiangC]

@guyuqi @sekikn
Thank you very much for your detailed explanation. I have learned a lot.

[sekikn]

+1, thank you so much for your contribution @JiaLiangC! I ensured the latest PR works with CentOS 7, Rocky 8 and Ubuntu 22.04.

Only with Debian 11, running smoke test failed with the following error. But the current PR works with the most of the distros we support, so let's merge it first and then address the problem as another issue. I'll file it later.

$ ./docker-hadoop.sh -d -dcp -C config_debian-11.yaml -F docker-compose-cgroupv2.yml -G -L -k hdfs,hbase,ranger -s ranger -c 1

...

org.apache.bigtop.itest.ranger.TestRangerSmoke > testServiceDefs FAILED
    java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:6080/service/public/v2/api/servicedef
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1900)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at org.codehaus.groovy.runtime.ResourceGroovyMethods.configuredInputStream(ResourceGroovyMethods.java:2218)
        at org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:2319)
        at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:692)
        at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:660)
        at org.codehaus.groovy.runtime.dgm$1017.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:246)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.bigtop.itest.ranger.TestRangerSmoke.testServiceDefs(TestRangerSimple.groovy:39)

1 test completed, 1 failed
Finished generating test XML results (0.014 secs) into: /bigtop-home/bigtop-tests/smoke-tests/ranger/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.032 secs) into: /bigtop-home/bigtop-tests/smoke-tests/ranger/build/reports/tests/test
:bigtop-tests:smoke-tests:ranger:test (Thread[Daemon worker,5,main]) completed. Took 1.637 secs.

FAILURE: Build failed with an exception.