BIGTOP-2345 Create Flink packaging #101

rmetzger · 2016-04-21T19:46:12Z

This PR is based on the work done in #93.
I've picked only the parts regarding the packaging (ignoring the puppet deployment).

The changes have been tested on a Debian8 virtual machine and Fedora 23.
Packaging, installation, the init scripts, user creation, file removal, logging seem to work now.

I'm looking forward to any comments regarding the proposed changes.

mxm · 2016-04-22T14:39:57Z

bigtop-packages/src/rpm/flink/SPECS/flink.spec

+
+%dir %{_sysconfdir}/%{flink_name}
+%config(noreplace) %{initd_dir}/%{flink_name}-master
+%config(noreplace) %{initd_dir}/%{flink_name}-worker


I think those need to be

%config(noreplace) %{initd_dir}/%{flink_name}-jobmanager %config(noreplace) %{initd_dir}/%{flink_name}-taskmanager

It seems that the rebasing before the PR went horribly wrong :(

mxm · 2016-04-22T14:40:42Z

I'm getting the following error message when building using gradle flink-rpm:

RPM build errors:
File not found: /home/max/Desktop/bigtop/build/flink/rpm/BUILDROOT/flink-1.0.0-1.fc23.x86_64/etc/rc.d/init.d/flink-master
File not found: /home/max/Desktop/bigtop/build/flink/rpm/BUILDROOT/flink-1.0.0-1.fc23.x86_64/etc/rc.d/init.d/flink-worker
File not found: /home/max/Desktop/bigtop/build/flink/rpm/BUILDROOT/flink-1.0.0-1.fc23.x86_64/usr/share/doc/flink-1.0.0

bhupendrasingh · 2016-04-24T04:10:45Z

bigtop-packages/src/rpm/flink/SPECS/flink.spec

+%define etc_flink /etc/%{flink_name}
+%define config_flink %{etc_flink}/conf
+%define man_dir %{_mandir}
+%define flink_services master worker


I think this also needs to be changed to -
%define flink_services jobmanager taskmanager
try rebuilding the package again after running ./gradlew flink-clean

rmetzger · 2016-04-24T10:22:09Z

The gradle build of the RPM should be fixed. But there seems to be still an issue with the systemd start of the jobmanager.

rmetzger · 2016-04-25T09:49:29Z

I verified the RPM build again. It should be working now.
I'll now test the debian packaging again.

rmetzger · 2016-04-25T10:16:16Z

Debian is now fixed as well.

mxm · 2016-04-25T14:57:20Z

Tested the RPM package on Fedora 23. First, a bit of a disappointment:

● jobmanager.service - LSB: Flink jobmanager
   Loaded: loaded (/etc/rc.d/init.d/jobmanager)
   Active: failed (Result: exit-code) since Mon 2016-04-25 16:40:45 CEST; 16s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13344 ExecStart=/etc/rc.d/init.d/jobmanager start (code=exited, status=127)

Apr 25 16:40:41 localhost.localdomain jobmanager[13344]: /etc/rc.d/init.d/jobmanager: line 34: /lib/lsb/init-functions: No such file or directory
Apr 25 16:40:41 localhost.localdomain jobmanager[13344]: /etc/rc.d/init.d/jobmanager: line 74: log_success_msg: command not found
Apr 25 16:40:41 localhost.localdomain jobmanager[13344]: /etc/rc.d/init.d/jobmanager: line 120: pidofproc: command not found
Apr 25 16:40:41 localhost.localdomain su[13386]: (to flink) root on none
Apr 25 16:40:45 localhost.localdomain jobmanager[13344]: /etc/rc.d/init.d/jobmanager: line 120: pidofproc: command not found
Apr 25 16:40:45 localhost.localdomain jobmanager[13344]: /etc/rc.d/init.d/jobmanager: line 102: log_failure_msg: command not found
Apr 25 16:40:45 localhost.localdomain systemd[1]: jobmanager.service: Control process exited, code=exited status=127
Apr 25 16:40:45 localhost.localdomain systemd[1]: Failed to start LSB: Flink jobmanager.
Apr 25 16:40:45 localhost.localdomain systemd[1]: jobmanager.service: Unit entered failed state.
Apr 25 16:40:45 localhost.localdomain systemd[1]: jobmanager.service: Failed with result 'exit-code'.

Turns out, it works when you install lsb or more specifically redhat-lsb-core:

● jobmanager.service - LSB: Flink jobmanager
   Loaded: loaded (/etc/rc.d/init.d/jobmanager)
   Active: active (exited) since Mon 2016-04-25 16:42:56 CEST; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14071 ExecStart=/etc/rc.d/init.d/jobmanager start (code=exited, status=0/SUCCESS)

Apr 25 16:42:56 localhost.localdomain systemd[1]: Starting LSB: Flink jobmanager...
Apr 25 16:42:56 localhost.localdomain jobmanager[14071]: Starting Flink jobmanager (flink-jobmanager):[  OK  ]
Apr 25 16:42:56 localhost.localdomain jobmanager[14071]: Flink jobmanager is running[  OK  ]
Apr 25 16:42:56 localhost.localdomain systemd[1]: Started LSB: Flink jobmanager.

● taskmanager.service - LSB: Flink taskmanager
   Loaded: loaded (/etc/rc.d/init.d/taskmanager)
   Active: active (exited) since Mon 2016-04-25 16:44:28 CEST; 22s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 15428 ExecStart=/etc/rc.d/init.d/taskmanager start (code=exited, status=0/SUCCESS)

Apr 25 16:44:25 localhost.localdomain systemd[1]: Starting LSB: Flink taskmanager...
Apr 25 16:44:25 localhost.localdomain taskmanager[15428]: Starting Flink taskmanager (flink-taskmanager):[  OK  ]
Apr 25 16:44:25 localhost.localdomain su[15483]: (to flink) root on none
Apr 25 16:44:28 localhost.localdomain taskmanager[15428]: Started Flink taskmanager (flink-taskmanager):[  OK  ]
Apr 25 16:44:28 localhost.localdomain systemd[1]: Started LSB: Flink taskmanager.

So we need to add lsb as a dependency for Flink. Or do we assume people have it installed?

The jobmanager/taskmanager came up fine. I wonder, should we rename the service to flink-master and flink-worker? Or flink-jobmanager and flink-taskmanager? Simply TaskManager and JobManager could confuse people.

Another thing, the logs are named "flink-flink-jobmanager-0-localhost.localdomain.log". Probably could remove the flink-flink prefix.

Other than that, great work 👍

oflebbe · 2016-04-25T20:19:52Z

Hi. My Nitty picky comment:

Please see https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging on how to patch sources. (you may have a look at the hive or zookeeper packages for examples)

Short : rename 1837.patch to patch1-1837.patch . Insert the magic patterns into the spec file. remove the patch command from the do-component-build.

In order to get committed the patch on the branch has to be collapsed to only one patch with the exact message as the JIRA.

oflebbe · 2016-04-26T11:21:42Z

Hi the build looks very promising! You should not package /var/run, since this is only a temporary filesystem. You should create it with your init script (This will fix one of the lintian errors for debian/ubuntu).

rmetzger · 2016-04-26T12:14:17Z

Thank you for the feedback!
I'm trying to address them soon.

rmetzger · 2016-04-26T15:33:54Z

Thank you for the review @mxm!

Regarding the lsb dependency: I saw that the Hadoop spec for example has the following entry:

Requires: sh-utils, /lib/lsb/init-functions

I guess we can add that as well.

I wonder, should we rename the service to flink-master and flink-worker? Or flink-jobmanager and flink-taskmanager? Simply TaskManager and JobManager could confuse people.

I'm definitively against flink-master, flink-worker, because that's not an "official" terminology (at least in the Flink code we call them job- and taskmanager).
I also thought about flink-jobmanager and flink-taskmanager. Any other thoughts on this?

Another thing, the logs are named "flink-flink-jobmanager-0-localhost.localdomain.log". Probably could remove the flink-flink prefix.

The naming schema for logs is flink-<username>-<component>-<instance>-<host>.<log|out>. By starting the flink services with the flink user, we end up with that ugliness.
I'll look into it and try to see if I can change the behavior easily (the problem is that I need to create patches if I want to change anything on the flink source, and I want to avoid patching too much for bigtop).

@oflebbe: I'll check the guide again and adopt the patch integration accordingly.

rmetzger · 2016-04-26T16:01:04Z

@oflebbe quick question:
The spec file contains already the following section:

%if  %{?suse_version:1}0
# Required for init scripts
Requires: insserv
%global initd_dir %{_sysconfdir}/rc.d

%else
# Required for init scripts
Requires: /lib/lsb/init-functions
%global initd_dir %{_sysconfdir}/rc.d/init.d
%endif

I assume the if condition is true for suse, so the else branch applied for Max' build, requiring the init-functions?

rmetzger · 2016-04-26T16:16:38Z

Another follow up Q:

You should not package /var/run, since this is only a temporary filesystem. You should create it with your init script (This will fix one of the lintian errors for debian/ubuntu).

Okay, makes sense. I was looking a bit into other scripts and I saw that others are setting the $WORKING_DIR variable.
I wonder why /bigtop-packages/src/templates/init.d.tmpl is not taking care of the creation and deletion of the $WORKING_DIR for that?

mxm · 2016-04-26T16:28:17Z

I'm definitively against flink-master, flink-worker, because that's not an "official" terminology (at least in the Flink code we call them job- and taskmanager).

Then let's call them flink-jobmanager and flink-taskmanager. The flink- prefix will help people to discover the service names.

rmetzger · 2016-04-27T08:28:48Z

Okay, I'll rename the services

rmetzger · 2016-04-28T14:13:42Z

I addressed all remaining issues with the pull request.

Please let me know if there's anything else blocking this from being merged.

mbalassi · 2016-05-05T13:38:41Z

Hey guys, any update on this? I am looking forward to having the Flink integration in. :)

ghost · 2016-05-08T20:23:26Z

I have committed this to the master. Feel free to close the PR and thanks for your contribution!

rmetzger · 2016-05-09T12:38:09Z

Thanks a lot for merging.
I'm closing the PR now.

With ODPi 2.0 release branch cut, the main is now at version 2.1-SNAPSHOT

rmetzger mentioned this pull request Apr 22, 2016

WIP flink hacking --rpm build complete #93

Open

mxm reviewed Apr 22, 2016
View reviewed changes

bhupendrasingh reviewed Apr 24, 2016
View reviewed changes

rmetzger force-pushed the bigtop2345-pr branch from 1e67d97 to f4fb7ed Compare April 24, 2016 09:31

rmetzger force-pushed the bigtop2345-pr branch from 58bdc2b to 23447da Compare April 25, 2016 09:48

rmetzger force-pushed the bigtop2345-pr branch 3 times, most recently from f0dc70e to 069cfea Compare April 28, 2016 14:11

rmetzger changed the title ~~[BIGTOP-2345] Add deb and rpm packaging for Apache Flink~~ BIGTOP-2345 Create Flink packaging Apr 28, 2016

rmetzger force-pushed the bigtop2345-pr branch 2 times, most recently from f893e66 to 0dd8d18 Compare May 2, 2016 11:35

BIGTOP-2345 Create Flink packaging

72f005a

rmetzger force-pushed the bigtop2345-pr branch from 0dd8d18 to 72f005a Compare May 2, 2016 14:00

rmetzger closed this May 9, 2016

bharatm363 pushed a commit to bharatm363/bigtop that referenced this pull request Mar 9, 2018

Merge pull request apache#101 from rvs/odpi-master

1bf22e5

With ODPi 2.0 release branch cut, the main is now at version 2.1-SNAPSHOT

BIGTOP-2345 Create Flink packaging #101

BIGTOP-2345 Create Flink packaging #101

Uh oh!

Conversation

rmetzger commented Apr 21, 2016

Uh oh!

mxm Apr 22, 2016

Choose a reason for hiding this comment

Uh oh!

rmetzger Apr 24, 2016

Choose a reason for hiding this comment

Uh oh!

mxm commented Apr 22, 2016

Uh oh!

bhupendrasingh Apr 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rmetzger commented Apr 24, 2016

Uh oh!

rmetzger commented Apr 25, 2016

Uh oh!

rmetzger commented Apr 25, 2016

Uh oh!

mxm commented Apr 25, 2016

Uh oh!

oflebbe commented Apr 25, 2016

Uh oh!

oflebbe commented Apr 26, 2016

Uh oh!

rmetzger commented Apr 26, 2016

Uh oh!

rmetzger commented Apr 26, 2016

Uh oh!

rmetzger commented Apr 26, 2016

Uh oh!

rmetzger commented Apr 26, 2016

Uh oh!

mxm commented Apr 26, 2016

Uh oh!

rmetzger commented Apr 27, 2016

Uh oh!

rmetzger commented Apr 28, 2016

Uh oh!

mbalassi commented May 5, 2016

Uh oh!

ghost commented May 8, 2016

Uh oh!

rmetzger commented May 9, 2016

Uh oh!

Uh oh!

bhupendrasingh Apr 24, 2016 •

edited

Loading