Skip to content

BIGTOP-3571. Ensure Hadoop and Flink are deployed in the correct order.#799

Merged
iwasakims merged 1 commit into
apache:masterfrom
sekikn:BIGTOP-3571
Jul 13, 2021
Merged

BIGTOP-3571. Ensure Hadoop and Flink are deployed in the correct order.#799
iwasakims merged 1 commit into
apache:masterfrom
sekikn:BIGTOP-3571

Conversation

@sekikn
Copy link
Copy Markdown
Contributor

@sekikn sekikn commented Jul 11, 2021

@sekikn
Copy link
Copy Markdown
Contributor Author

sekikn commented Jul 11, 2021

Tested both of the following commands worked with this PR on x86_64.

  1. Deploy HDFS and Flink simultanously and run Flink's smoke test, which requires HDFS installed:
$ ./docker-hadoop.sh -d -k hdfs,flink -s flink -L -C config_centos-8.yaml -c 1

...

org.apache.bigtop.itest.hadoop.flink.TestFlink STANDARD_ERROR
    21/07/11 02:08:02 INFO lang.Object: hadoop fs -rm -r /flink
    21/07/11 02:08:05 INFO lang.Object: hadoop fs -mkdir /flink
    21/07/11 02:08:07 INFO lang.Object: hadoop fs -put test.data /flink/

org.apache.bigtop.itest.hadoop.flink.TestFlink > testCheckRestfulAPI STANDARD_ERROR
    21/07/11 02:08:10 INFO lang.Object: awk '{if(/jobmanager.rpc.address:/) print $2}' < /etc/flink/conf/flink-conf.yaml
    21/07/11 02:08:10 INFO lang.Object: awk '{if(/rest.port:/) print $2}' < /etc/flink/conf/flink-conf.yaml
    21/07/11 02:08:10 INFO lang.Object: curl http://abebd8688324.bigtop.apache.org:8081/config

org.apache.bigtop.itest.hadoop.flink.TestFlink > testWordCountBatch STANDARD_ERROR
    21/07/11 02:08:11 INFO lang.Object: hdfs getconf -confKey fs.defaultFS
    21/07/11 02:08:12 INFO lang.Object: flink run $FLINK_HOME/examples/batch/WordCount.jar --input hdfs://abebd8688324.bigtop.apache.org:8020/flink/test.data --output hdfs://abebd8688324.bigtop.apache.org:8020/tmp/result.txt
    21/07/11 02:08:19 INFO lang.Object: hadoop fs -cat /tmp/result.txt

org.apache.bigtop.itest.hadoop.flink.TestFlink STANDARD_ERROR
    21/07/11 02:08:22 INFO lang.Object: hadoop fs -rm -r /flink

Gradle Test Executor 2 finished executing tests.

> Task :bigtop-tests:smoke-tests:flink:test
Finished generating test XML results (0.015 secs) into: /bigtop-home/bigtop-tests/smoke-tests/flink/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.038 secs) into: /bigtop-home/bigtop-tests/smoke-tests/flink/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:flink:test (Thread[Daemon worker,5,main]) completed. Took 24.509 secs.

BUILD SUCCESSFUL in 1m 45s
34 actionable tasks: 7 executed, 27 up-to-date
  1. Only deploy Flink, without HDFS:
$ ./docker-hadoop.sh -d -k flink -L -C config_centos-8.yaml -c 1

...

Notice: /Stage[main]/Flink::Jobmanager/Package[flink-jobmanager]/ensure: created
Notice: /Stage[main]/Flink::Jobmanager/Service[flink-jobmanager]/ensure: ensure changed 'stopped' to 'running'
Notice: /Stage[main]/Flink::Taskmanager/Package[flink-taskmanager]/ensure: created
Notice: /Stage[main]/Flink::Taskmanager/Service[flink-taskmanager]/ensure: ensure changed 'stopped' to 'running'
Notice: Applied catalog in 63.22 seconds

Copy link
Copy Markdown
Member

@iwasakims iwasakims left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@iwasakims iwasakims merged commit f686ada into apache:master Jul 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants