Skip to content

BIGTOP-3552. Add missing files to ambari-agent RPM.#782

Merged
sekikn merged 1 commit intoapache:masterfrom
iwasakims:BIGTOP-3552
Jun 17, 2021
Merged

BIGTOP-3552. Add missing files to ambari-agent RPM.#782
sekikn merged 1 commit intoapache:masterfrom
iwasakims:BIGTOP-3552

Conversation

@iwasakims
Copy link
Member

@iwasakims iwasakims commented Jun 8, 2021

https://issues.apache.org/jira/browse/BIGTOP-3552

smoke-tests of ambari succeeded on CentOS 7:

$ ./docker-hadoop.sh \
   --create 1 \
   --image bigtop/puppet:trunk-centos-7 \
   --memory 16g \
   --repo file:///bigtop-home/output \
   --disable-gpg-check \
   --stack bigtop-utils,ambari \
   --smoke-tests ambari
...
Now testing...
:bigtop-tests:smoke-tests:ambari:test (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 1.745 secs.

BUILD SUCCESSFUL in 52s
35 actionable tasks: 7 executed, 28 up-to-date
Stopped 1 worker daemon(s).
+ rm -rf buildSrc/build/test-results/binary
+ rm -rf /bigtop-home/.gradle

It seems not to work on CentOS 8. I filed BIGTOP-3553 for that.

@sekikn
Copy link
Contributor

sekikn commented Jun 17, 2021

+1, also tested on CentOS 8 and Fedora 33. Thanks @iwasakims!

@sekikn
Copy link
Contributor

sekikn commented Jun 17, 2021

It seems not to work on CentOS 8. I filed BIGTOP-3553 for that.

Oh, I missed that. I couldn't reproduce BIGTOP-3553 in my environment. Fixed within another issue?

$ ./docker-hadoop.sh -d -k bigtop-utils,ambari -s ambari -L -C config_centos-8.yaml -c 1

...

> Task :bigtop-tests:smoke-tests:ambari:test
Finished generating test XML results (0.017 secs) into: /bigtop-home/bigtop-tests/smoke-tests/ambari/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.048 secs) into: /bigtop-home/bigtop-tests/smoke-tests/ambari/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:ambari:test (Thread[Daemon worker,5,main]) completed. Took 2.919 secs.

BUILD SUCCESSFUL in 2m 0s

...

$ ./docker-hadoop.sh --exec 1 ambari-server --version
2.7.5.0-0
$ ./docker-hadoop.sh --exec 1 systemctl status ambari-server -l --no-pager
● ambari-server.service - LSB: ambari-server daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-server; generated)
   Active: active (running) since Thu 2021-06-17 02:02:45 UTC; 2h 33min ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 103 (limit: 50738)
   Memory: 494.8M
   CGroup: /docker/1fea5268b074629294d9dec3bd6171c675374063e02cb49ebc3accea91548a79/system.slice/ambari-server.service
           └─1214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dcom.sun.jndi.ldap.connect.pool.protocol=plain ssl -Dcom.sun.jndi.ldap.connect.pool.maxsize=20 -Dcom.sun.jndi.ldap.connect.pool.timeout=300000 -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer

Jun 17 02:02:23 1fea5268b074 ambari-server[1179]: Organizing resource files at /var/lib/ambari-server/resources...
Jun 17 02:02:23 1fea5268b074 ambari-server[1179]: Ambari database consistency check started...
Jun 17 02:02:23 1fea5268b074 ambari-server[1179]: Server PID at: /var/run/ambari-server/ambari-server.pid
Jun 17 02:02:23 1fea5268b074 ambari-server[1179]: Server out at: /var/log/ambari-server/ambari-server.out
Jun 17 02:02:23 1fea5268b074 ambari-server[1179]: Server log at: /var/log/ambari-server/ambari-server.log
Jun 17 02:02:45 1fea5268b074 ambari-server[1179]: Waiting for server start.........................
Jun 17 02:02:45 1fea5268b074 ambari-server[1179]: Server started listening on 8080
Jun 17 02:02:45 1fea5268b074 ambari-server[1179]: DB configs consistency check: no errors and warnings were found.
Jun 17 02:02:45 1fea5268b074 ambari-server[1179]: Ambari Server 'start' completed successfully.
Jun 17 02:02:45 1fea5268b074 systemd[1]: Started LSB: ambari-server daemon.
$ ./docker-hadoop.sh --exec 1 systemctl status ambari-agent -l --no-pager
● ambari-agent.service - LSB: ambari-agent daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-agent; generated)
   Active: active (running) since Thu 2021-06-17 02:02:55 UTC; 2h 33min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1373 ExecStart=/etc/rc.d/init.d/ambari-agent start (code=exited, status=0/SUCCESS)
    Tasks: 11 (limit: 50738)
   Memory: 34.8M
   CGroup: /docker/1fea5268b074629294d9dec3bd6171c675374063e02cb49ebc3accea91548a79/system.slice/ambari-agent.service
           ├─1432 /usr/bin/python2 /usr/lib/ambari-agent/lib/ambari_agent/AmbariAgent.py start
           └─1436 /usr/bin/python2 /usr/lib/ambari-agent/lib/ambari_agent/main.py start

Jun 17 02:02:53 1fea5268b074 ambari-agent[1373]: Checking ambari-common dir...
Jun 17 02:02:53 1fea5268b074 ambari-agent[1373]: Starting ambari-agent
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: Verifying ambari-agent process status...
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: /var/lib/ambari-agent/bin/ambari-agent: line 237: tput: command not found
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: Ambari Agent successfully started
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: /var/lib/ambari-agent/bin/ambari-agent: line 239: tput: command not found
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: Agent PID at: /run/ambari-agent/ambari-agent.pid
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: Agent out at: /var/log/ambari-agent/ambari-agent.out
Jun 17 02:02:55 1fea5268b074 ambari-agent[1373]: Agent log at: /var/log/ambari-agent/ambari-agent.log
Jun 17 02:02:55 1fea5268b074 systemd[1]: Started LSB: ambari-agent daemon.

@sekikn sekikn merged commit 0b089d1 into apache:master Jun 17, 2021
@iwasakims
Copy link
Member Author

iwasakims commented Jun 21, 2021

It seems not to work on CentOS 8. I filed BIGTOP-3553 for that.

Oh, I missed that. I couldn't reproduce BIGTOP-3553 in my environment. Fixed within another issue?

@sekikn The original description of the BIGTOP-3553 was based on my misunderstang. I updated the title and description. I still have issues described in the JIRA comments. Could you look into them?

@iwasakims
Copy link
Member Author

iwasakims commented Jun 21, 2021

I would like to know the version of systemd used in your successful case.

root@fe7b2a39b6a4 /]# rpm -q systemd
systemd-239-41.el8_3.x86_64

@sekikn
Copy link
Contributor

sekikn commented Jun 21, 2021

I used the same version as yours, but the test succeeded for some reason.

[vagrant@localhost docker]$ ./docker-hadoop.sh -d -k bigtop-utils,ambari -s ambari -L -C config_centos-8.yaml -c 1

(snip)

> Task :bigtop-tests:smoke-tests:ambari:test
Caching disabled for task ':bigtop-tests:smoke-tests:ambari:test' because:
  Build cache is disabled
Task ':bigtop-tests:smoke-tests:ambari:test' is not up-to-date because:
  No history is available.
Starting process 'Gradle Test Executor 2'. Working directory: /bigtop-home/bigtop-tests/smoke-tests/ambari Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/bin/java -Dawt.toolkit=sun.awt.X11.XToolkit -Dfile.encoding.pkg=sun.io -Dfile.separator=/ -Djava.awt.graphicsenv=sun.awt.X11GraphicsEnvironment -Djava.awt.printerjob=sun.print.PSPrinterJob -Djava.class.path=/root/.gradle/wrapper/dists/gradle-5.6.4-bin/c9880aa85176bf8c458862eb99f7e0a9/gradle-5.6.4/lib/gradle-launcher-5.6.4.jar -Djava.class.version=52.0 -Djava.endorsed.dirs=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/endorsed -Djava.ext.dirs=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/ext:/usr/java/packages/lib/ext -Djava.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib -Djava.runtime.name=OpenJDK Runtime Environment -Djava.runtime.version=1.8.0_292-b10 -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Djava.specification.name=Java Platform API Specification -Djava.specification.vendor=Oracle Corporation -Djava.specification.version=1.8 -Djava.vendor=Red Hat, Inc. -Djava.vendor.url=https://www.redhat.com/ -Djava.vendor.url.bug=https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-1.8.0-openjdk -Djava.version=1.8.0_292 -Djava.vm.info=mixed mode -Djava.vm.name=OpenJDK 64-Bit Server VM -Djava.vm.specification.name=Java Virtual Machine Specification -Djava.vm.specification.vendor=Oracle Corporation -Djava.vm.specification.version=1.8 -Djava.vm.vendor=Red Hat, Inc. -Djava.vm.version=25.292-b10 -Dlibrary.jansi.path=/root/.gradle/native/jansi/1.17.1/linux64 -Dline.separator=
 -Dorg.gradle.appname=gradlew -Dorg.gradle.native=false -Dos.arch=amd64 -Dos.name=Linux -Dos.version=4.18.0-240.22.1.el8_3.x86_64 -Dpath.separator=: -Dsun.arch.data.model=64 -Dsun.boot.class.path=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/classes -Dsun.boot.library.path=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/lib/amd64 -Dsun.cpu.endian=little -Dsun.cpu.isalist -Dsun.io.unicode.encoding=UnicodeLittle -Dsun.java.command=org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.6.4 -Dsun.java.launcher=SUN_STANDARD -Dsun.jnu.encoding=ANSI_X3.4-1968 -Dsun.management.compiler=HotSpot 64-Bit Tiered Compilers -Dsun.os.patch.level=unknown -Duser.dir=/bigtop-home -Duser.home=/root -Duser.name=root -Duser.timezone -Xmx512m -Dfile.encoding=ANSI_X3.4-1968 -Djava.io.tmpdir=/tmp -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /root/.gradle/caches/5.6.4/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 2'
Successfully started process 'Gradle Test Executor 2'

org.apache.bigtop.itest.ambari.TestAmbariSmoke > testBlueprints STANDARD_OUT
    [href:http://localhost:8080/api/v1/blueprints, items:[]]

org.apache.bigtop.itest.ambari.TestAmbariSmoke > testStackNameVersion STANDARD_OUT
    [versions:[[Versions:[stack_version:2.0, stack_name:ODPi], href:http://localhost:8080/api/v1/stacks/ODPi/versions/2.0]], Stacks:[stack_name:ODPi], href:http://localhost:8080/api/v1/stacks/ODPi]

Gradle Test Executor 2 finished executing tests.

> Task :bigtop-tests:smoke-tests:ambari:test
Finished generating test XML results (0.043 secs) into: /bigtop-home/bigtop-tests/smoke-tests/ambari/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.061 secs) into: /bigtop-home/bigtop-tests/smoke-tests/ambari/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:ambari:test (Thread[Daemon worker,5,main]) completed. Took 3.459 secs.

BUILD SUCCESSFUL in 2m 4s
35 actionable tasks: 7 executed, 28 up-to-date
Stopped 1 worker daemon(s).
+ rm -rf buildSrc/build/test-results/binary
+ rm -rf /bigtop-home/.gradle
[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 rpm -q systemd
systemd-239-41.el8_3.x86_64

The versions and statuses of other packages are as follows, just in case.

[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 rpm -q postgresql
postgresql-10.17-1.module_el8.4.0+823+f0dbe136.x86_64
[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-06-21 10:05:51 UTC; 6min ago
 Main PID: 1052 (postmaster)
    Tasks: 14 (limit: 50738)
   Memory: 45.3M
   CGroup: /docker/2358159baf2dc29f952292880e90eebe0b46dfefdc15c8d7cb1ae5ae7b9f54b1/system.slice/postgresql.service
           ├─1052 /usr/bin/postmaster -D /var/lib/pgsql/data
           ├─1053 postgres: logger process   
           ├─1055 postgres: checkpointer process   
           ├─1056 postgres: writer process   
           ├─1057 postgres: wal writer process   
           ├─1058 postgres: autovacuum launcher process   
           ├─1059 postgres: stats collector process   
           ├─1060 postgres: bgworker: logical replication launcher   
           ├─1220 postgres: ambari ambari 127.0.0.1(60254) idle
           ├─1221 postgres: ambari ambari 127.0.0.1(60258) idle
           ├─1232 postgres: ambari ambari 127.0.0.1(60268) idle
           ├─1997 postgres: ambari ambari 127.0.0.1(60356) idle
           ├─1998 postgres: ambari ambari 127.0.0.1(60358) idle
           └─1999 postgres: ambari ambari 127.0.0.1(60360) idle

Jun 21 10:05:51 2358159baf2d systemd[1]: postgresql.service: Succeeded.
Jun 21 10:05:51 2358159baf2d systemd[1]: Stopped PostgreSQL database server.
Jun 21 10:05:51 2358159baf2d systemd[1]: Starting PostgreSQL database server...
Jun 21 10:05:51 2358159baf2d postmaster[1052]: 2021-06-21 10:05:51.600 UTC [1052] LOG:  listening on IPv4 address "0.0.0.0", port 5432
Jun 21 10:05:51 2358159baf2d postmaster[1052]: 2021-06-21 10:05:51.600 UTC [1052] LOG:  listening on IPv6 address "::", port 5432
Jun 21 10:05:51 2358159baf2d postmaster[1052]: 2021-06-21 10:05:51.602 UTC [1052] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Jun 21 10:05:51 2358159baf2d postmaster[1052]: 2021-06-21 10:05:51.603 UTC [1052] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
Jun 21 10:05:51 2358159baf2d postmaster[1052]: 2021-06-21 10:05:51.608 UTC [1052] LOG:  redirecting log output to logging collector process
Jun 21 10:05:51 2358159baf2d postmaster[1052]: 2021-06-21 10:05:51.608 UTC [1052] HINT:  Future log output will appear in directory "log".
Jun 21 10:05:51 2358159baf2d systemd[1]: Started PostgreSQL database server.

[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 rpm -q ambari-server
ambari-server-2.7.5.0-1.el8.noarch
[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 systemctl status ambari-server --no-pager
● ambari-server.service - LSB: ambari-server daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-server; generated)
   Active: active (running) since Mon 2021-06-21 10:06:20 UTC; 7min ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 104 (limit: 50738)
   Memory: 463.6M
   CGroup: /docker/2358159baf2dc29f952292880e90eebe0b46dfefdc15c8d7cb1ae5ae7b9f54b1/system.slice/ambari-server.service
           └─1204 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/…

Jun 21 10:05:59 2358159baf2d ambari-server[1169]: Organizing resource files …...
Jun 21 10:05:59 2358159baf2d ambari-server[1169]: Ambari database consistenc…...
Jun 21 10:05:59 2358159baf2d ambari-server[1169]: Server PID at: /var/run/am…pid
Jun 21 10:05:59 2358159baf2d ambari-server[1169]: Server out at: /var/log/am…out
Jun 21 10:05:59 2358159baf2d ambari-server[1169]: Server log at: /var/log/am…log
Jun 21 10:06:20 2358159baf2d ambari-server[1169]: Waiting for server start..…...
Jun 21 10:06:20 2358159baf2d ambari-server[1169]: Server started listening o…080
Jun 21 10:06:20 2358159baf2d ambari-server[1169]: DB configs consistency che…nd.
Jun 21 10:06:20 2358159baf2d ambari-server[1169]: Ambari Server 'start' comp…ly.
Jun 21 10:06:20 2358159baf2d systemd[1]: Started LSB: ambari-server daemon.
Hint: Some lines were ellipsized, use -l to show in full.

[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 rpm -q ambari-agent
ambari-agent-2.7.5.0-1.el8.noarch
[vagrant@localhost docker]$ ./docker-hadoop.sh --exec 1 systemctl status ambari-agent --no-pager
● ambari-agent.service - LSB: ambari-agent daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-agent; generated)
   Active: active (running) since Mon 2021-06-21 10:06:31 UTC; 7min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1363 ExecStart=/etc/rc.d/init.d/ambari-agent start (code=exited, status=0/SUCCESS)
    Tasks: 11 (limit: 50738)
   Memory: 34.9M
   CGroup: /docker/2358159baf2dc29f952292880e90eebe0b46dfefdc15c8d7cb1ae5ae7b9f54b1/system.slice/ambari-agent.service
           ├─1422 /usr/bin/python2 /usr/lib/ambari-agent/lib/ambari_agent/AmbariAgent.py start
           └─1426 /usr/bin/python2 /usr/lib/ambari-agent/lib/ambari_agent/main.py start

Jun 21 10:06:29 2358159baf2d ambari-agent[1363]: Checking ambari-common dir...
Jun 21 10:06:29 2358159baf2d ambari-agent[1363]: Starting ambari-agent
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: Verifying ambari-agent process status...
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: tput: No value for $TERM and no -T specified
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: Ambari Agent successfully started
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: tput: No value for $TERM and no -T specified
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: Agent PID at: /run/ambari-agent/ambari-agent.pid
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: Agent out at: /var/log/ambari-agent/ambari-agent.out
Jun 21 10:06:31 2358159baf2d ambari-agent[1363]: Agent log at: /var/log/ambari-agent/ambari-agent.log
Jun 21 10:06:31 2358159baf2d systemd[1]: Started LSB: ambari-agent daemon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@iwasakims @sekikn